Grey matter segmentation challenge: what's there and what's next?


Participating teams will apply their automatic or semi-automatic segmentation algorithms to anatomical MR images of the healthy spinal cord acquired at 4 different sites (University College London (UCL), Ecole Polytechnique de Montreal (EPM), Vanderbilt University (VDB), University Hospital Zurich (UHZ)). Algorithms will be evaluated against manual segmentations from four trained raters (one from each site) in terms of segmentation accuracy and precision.

DATA:
A multi-centre, multi-vendor data set of spinal cord anatomical images of healthy subjects is provided. It consists of:
  • UCL: 20 subjects with WM/GM segmentation (Philips 3T): Acquisition was performed using a 3T Philips Achieva MRI system with dual-transmit technology (Philips Healthcare, Best, Netherlands) and the manufacturer's product 16-channel neurovascular (NV) coil, the cervical cord was imaged in the axial-oblique plane (i.e. slices perpendicular to the cord) with the center of the imaging volume positioned at the level of C2-3 intervertebral disc. The MRI acquisition parameters were: Fat-suppressed 3D slab-selective fast field echo (3D-FFE) with TR=23 ms; TE=5 ms, flip angle alpha=7 degrees, FOV=240 x 180 mm, voxel size=0.5 x 0.5 x 5 mm^3, NEX=8, 10 axial contiguous slices, scanning time=13:34 min. A 15 mm section of the high-resolution 3D-FFE volumetric scan (i.e. 3 slices) was extracted, with the middle slice passing through the C2-3 intervertebral disc.
  • EPM: 20 subjects with WM/GM segmentation (Siemens 3T): Acquisition was performed using a 3T Siemens TIM Trio, 12ch head + 4ch neck coil, axial 2D spoiled gradient echo, TR=539ms, TE=5.41,12.56,19.16ms (averaged offline), FA=35°, BW=200Hz/pixel, resolution=0.5x0.5x7.5mm^3, 10 slices, matrix=320x320, R=2 acceleration with GRAPPA reconstruction, phase stabilization.
  • VDB: 20 subjects with WM/GM segmentation (Philips 3T): Imaging was performed on a 3T whole body Philips scanner (Philips Achieva, Best, Netherlands). A two-channel body coil was used in multi-transmit mode for excitation and a 16-channel SENSE neurovascular coil was used for reception. The mFFE consists of a multi-slice, multi-echo fast field echo (FFE) sequence in the axial plane with the following relevant parameters: TR=700 ms, TE/deltaTE=7.2/8.9 ms, number of echoes=3, flip angle=28 deg, FOV=160 x 160 mm, slice thickness=5 mm, voxel size=0.3 x 0.3 x 5 mm^3, NSA=2, slices=12, SENSE: RL=2. The resulting scan time is 5:45 minutes. Acquisition is centered at C3/C4.
  • UHZ: 20 subjects with WM/GM segmentation (Siemens 3T): Scanning was performed on a 3T Skyra MRI scanner (Siemens Healthcare, Erlangen, Germany) using a 16-channel radio-frequency (RF) receive head and neck coil and RF body transmit coil. All participants wore an MRI-compatible stiff neck (Laerdal Medicals, Stavanger, Norway) to minimize motion artefacts and were carefully positioned by the radiographers to acquire the data from the same position and to obtain high reproducibility between all participants. A 3D high-resolution optimized T2*-weighted multi-echo sequence (multiple echo data image combination; MEDIC) was applied to acquire five high-resolution 3D volumes of the cervical cord at C2/C3 level. Each volume consisted of twenty contiguous slices acquired in the axial-oblique plane and was obtained with a resolution of 0.25 x 0.25 x 2.50 mm3 within 2 minutes and 8 seconds for each of the five volumes. Following parameters were applied: field of view (FOV) of 162 x 192 mm2, matrix size of 648x768, time of repetition (TR) of 44 ms, time of echo (TE) of 19 ms, flip angle α=11°, and readout bandwidth of 260 Hz per pixel. After data acquisition the five 3D volumes were averaged in the spatial domain to create a single image with increased SNR.

Data have been split into two sets of 40 images each, with 10 images from each site. The first set represents the training data, and will include the manual segmentations of the grey matter by four raters. It will be released on the 1st of March. The second set represents the test data, and will be used to score the performance of the algorithms. It will be released on the 1st of April.

Initially, data will be send by email to the teams, then you have to register your interest to participate in the challenge using the 3rd Spinal Cord MRI workshop form and fill and accept the data license.

METRICS:
A number of quantitative metrics will be employed to evaluate the quality of the submitted segmentations. The evaluation metrics will include:
  • Dice Coefficient: gives a measure of the spatial overlap between two masks.
  • Mean Surface Distance: is the mean of them sum of the Euclidean distance for each voxel in the mask's contours.
  • Hausdorff Distance: measures how far is the contour between the two segmentations.
  • Skelotonized Median distance: measures the median distance between the two skeletonized (Zhang et al. ACM 1984) grey matter segmentations. It's an indicator of global errors.
  • Skeletonized Hausdorff Distance: measures the maximum distance between the two skeletonized (or thinned) grey matter segmentations. It's an indicator of local errors.
  • Jaccard Coefficient: similarity index between two masks.
  • Conformity Coefficient: measures the number of mis-segmented voxels to the number of correctly segmented (Chang et al. Neuroimage 2009).
  • Precision (Positive Predictive Value): is a good compromise between true and false positive.
  • Sensitivity (True Positive Rate): represents a method’s ability to segment GM.
  • Specificity (True Negative Rate): measures the importance of the oversegmented voxels.

GUIDELINES:
  • Numerical input parameters may be used and should be constant for a particular test data set.
  • Output grey matter segmentations should be in the same space and resolution than the provided data.
  • Other publicly available data sets may be used within the algorithm.
  • Only one set of segmentations may be provided per team.
  • There are no restrictions on how the algorithm is implemented in regards to platform, programming language, or dependent software libraries.
  • Algorithms will be executed solely by the competing team with the segmentation results provided to the organizers.
  • Output segmentations should be saved in NIFTI format with a label of 1 assigned to spinal cord grey matter and 0 otherwise. Segmentations should be in 3-D. For 3D segmentations, filenames should be as follows: "subject_teamname.nii.gz" (example: for file "site3-sc09-image.nii.gz" and team name "fpc", the final filename has to be: "site3-sc09_fpc.nii.gz").
  • Results have to be submitted at: http://niftyweb.cs.ucl.ac.uk/
  • Teams will prepare a 5 min presentation explaining their method, that will be delivered on the 13th May during the 3rd Spinal Cord MRI meeting in Singapore.
  • It has to be specified if the method requires human interaction or not (fully-automatic), and if so, what are the required steps (e.g., cropping, normalization, centering, pre-segmentation, etc.).

RESULTS:

Comparison of each method segmentation versus each one of the four raters masks for the test dataset with the mean (std) Dice score coefficient (DSC), mean surface distance (MSD), Hausdorff distance (HD), skeletonized Hausdorff distance (SHD), skeletonized median distance (SMD), true positive rate (TPR), true negative rate (TNR), positive predictive value (PPV), Jaccard index (JI) and conformity coefficient (CC).



In bold face, the best obtained result. MSD, HD, SHD and SMD are in millimetres and lower values mean better, for all the other metrics higher values mean better score.

PUBLICATION:

Ferran Prados, John Ashburner, Claudia Blaiotta, Tom Brosch, Julio Carballido-Gamio, Manuel Jorge Cardoso, Benjamin N. Conrad, Esha Datta, Gergely Dávid, Benjamin De Leener, Sara M. Dupont, Patrick Freund, Claudia A.M. Gandini Wheeler-Kingshott, Francesco Grussu, Roland Henry, Bennett A. Landman, Emil Ljungberg, Bailey Lyttle, Sebastien Ourselin, Nico Papinutto, Salvatore Saporito, Regina Schlaeger, Seth A. Smith, Paul Summers, Roger Tam, Marios C. Yiannakas, Alyssa Zhu, Julien Cohen-Adad, Spinal cord grey matter segmentation challenge, NeuroImage, Volume 152, 15 May 2017, Pages 312-329, ISSN 1053-8119.



The results will be sent to the following e-mail address:


Dataset:


Is it automatic?:


Select a file in zip format (Maximum size 50 Mb):
[Select files - zip containing the 40 grey matter segmentation masks of one dataset]
Your browser doesn't have Flash, Silverlight or HTML5 support.

 By filling in this form I agree to have the provided email address stored in order to use it as email address to send the requested results. I allow use of this email address for contacting me about the project. I understand that NiftyWeb will not disclose this information to any other parties.

Upload your segmentations and obtain the different metric results



The queue is empty and CHALLENGE is ready to process your files!
Total files processed so far: 303/303

IMPORTANT DATES:
1st March 2016 - Trainning data release
1st April 2016 - Test data release
1st May 2016 - Submission of test results
13th May 2016 - Challenge day

REGISTRATION:
It is free, but please register to the 3rd Spinal Cord MRI workshop using this form and state that you are going to participate to the challenge.

ORGANIZERS
Julien Cohen-Adad, Ecole Polytechnique de Montreal (Canada)
Ferran Prados, University College London (United Kingdom)
Bennet A. Landman, Vanderbilt University (United States)
Patrick Freund, University of Zurich (Switzerland)
Claudia A.M. Gandini Wheeler-Kingshott, University College London (United Kingdom)
Paul Summers, University of Modena (Italy)
Sara Dupont, Ecole Polytechnique de Montreal (Canada)
Marios Yiannakas, University College London (United Kingdom)
Seth Smith, Vanderbilt University (United States)
David Gergely, University of Zurich (Switzerland)
Benjamin De Leener, Ecole Polytechnique de Montreal (Canada)
Francesco Grussu, University College London (United Kingdom)

DISCLAIMER: THIS ONLINE TOOL IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.