Clinical Research Papers:

Chemical exchange saturation transfer MRI serves as predictor of early progression in glioblastoma patients

PDF |  HTML  |  Supplementary Files  |  How to cite

Oncotarget. 2018; 9:28772-28783. https://doi.org/10.18632/oncotarget.25594

Metrics: PDF 1653 views  |   HTML 2751 views  |   ?  

Sebastian Regnery, Sebastian Adeberg, Constantin Dreher, Johanna Oberhollenzer, Jan-Eric Meissner, Steffen Goerke, Johannes Windschuh, Katerina Deike-Hofmann, Sebastian Bickelhaupt, Moritz Zaiss, Alexander Radbruch, Martin Bendszus, Wolfgang Wick, Andreas Unterberg, Stefan Rieken, Jürgen Debus, Peter Bachert, Mark Ladd, Heinz-Peter Schlemmer and Daniel Paech _


Sebastian Regnery1,2, Sebastian Adeberg3, Constantin Dreher2, Johanna Oberhollenzer2, Jan-Eric Meissner4, Steffen Goerke4, Johannes Windschuh4, Katerina Deike-Hofmann2, Sebastian Bickelhaupt2, Moritz Zaiss5, Alexander Radbruch2, Martin Bendszus6, Wolfgang Wick7, Andreas Unterberg8, Stefan Rieken1, Jürgen Debus1, Peter Bachert4, Mark Ladd4,9,10, Heinz-Peter Schlemmer2 and Daniel Paech2

1Department of Radiation Oncology, University Hospital Heidelberg, Heidelberg, Germany

2German Cancer Research Center (DKFZ), Division of Radiology, Heidelberg, Germany

3German Cancer Research Center (DKFZ), HIRO (Heidelberg Institute for Radiation Oncology), Heidelberg, Germany

4German Cancer Research Center (DKFZ), Division of Medical Physics in Radiology, Heidelberg, Germany

5Max-Planck-Institute, Tübingen, Germany

6Department of Neuroradiology, University Hospital Heidelberg, Heidelberg, Germany

7Department of Neurology, University Hospital Heidelberg, Heidelberg, Germany

8Department of Neurosurgery, University Hospital Heidelberg, Heidelberg, Germany

9Faculty of Physics and Astronomy, University of Heidelberg, Heidelberg, Germany

10Faculty of Medicine, University of Heidelberg, Heidelberg, Germany

Correspondence to:

Daniel Paech, email: [email protected]

Keywords: magnetic resonance imaging; amide-proton-transfer-imaging; nuclear overhauser imaging; glioblastoma; predictive biomarker

Received: March 29, 2018     Accepted: May 24, 2018     Published: June 19, 2018


Purpose: To prospectively investigate chemical exchange saturation transfer (CEST) MRI in glioblastoma patients as predictor of early tumor progression after first-line treatment.

Experimental Design: Twenty previously untreated glioblastoma patients underwent CEST MRI employing a 7T whole-body scanner. Nuclear Overhauser effect (NOE) as well as amide proton transfer (APT) CEST signals were isolated using Lorentzian difference (LD) analysis and relaxation compensated by the apparent exchange-dependent relaxation rate (AREX) evaluation. Additionally, NOE-weighted asymmetric magnetic transfer ratio (MTRasym) and downfield-NOE-suppressed APT (dns-APT) were calculated. Patient response to consecutive treatment was determined according to the RANO criteria. Mean signal intensities of each contrast in the whole tumor area were compared between early-progressive and stable disease.

Results: Pre-treatment tumor signal intensity differed significantly regarding responsiveness to first-line therapy in NOE-LD (p = 0.0001), NOE-weighted MTRasym (p = 0.0186) and dns-APT (p = 0.0328) contrasts. Hence, significant prediction of early progression was possible employing NOE-LD (AUC = 0.98, p = 0.0005), NOE-weighted MTRasym (AUC = 0.83, p = 0.0166) and dns-APT (AUC = 0.80, p = 0.0318). The NOE-LD provided the highest sensitivity (91%) and specificity (100%).

Conclusions: CEST derived contrasts, particularly NOE-weighted imaging and dns-APT, yielded significant predictors of early progression after fist-line therapy in glioblastoma. Therefore, CEST MRI might be considered as non-invasive tool for customization of treatment in the future.


Glioblastoma patients face a dismal prognosis despite optimal standard treatment consisting of resection followed by adjuvant chemoradiotherapy (CRT) [14]. However, a small group survives considerably longer [1, 3]. Hence, biomarkers predicting tumor response towards first-line treatment are highly desirable. Currently, histopathological analysis of O6-methylguanine DNA methyltransferase (MGMT) promotor methylation status yields a potential tool to assess chemosensitivity [47], but its role in clinical decision making remains controversial [4, 810]. Apart from that, present clinical imaging methods do not yield non-invasive biomarkers of response yet.

Chemical exchange saturation transfer (CEST) MRI might have the potential to fill this gap. CEST MRI is based on magnetic saturation of exchangeable protons in solute metabolites such as proteins, amino acids and lipids which serve as endogenous contrast agents [1114]. A protein-weighted CEST-spectrum consists of distinct signals, mainly originating from the nuclear Overhauser effect (NOE) [1315] and the amide proton transfer (APT) [12, 16]. Many CEST studies primarily extract APT signals using the asymmetric magnetic transfer ratio (MTRasym) [12, 1720], which faces limitations due to overlap of different CEST signals and contributions of non-CEST effects [14, 21, 22]. Ultra-high magnetic fields enable high resolution CEST spectra, which in turn support a multi Lorentzian fit analysis to isolate NOE and APT signals [14, 22]. CEST effects are sensitive to biochemical [12, 14, 21, 2325] as well as histopathological [19, 20, 2628] tumor properties, therefore yielding complementary information to current MRI methods [12, 14, 22, 29]. Recently, even an accurate discrimination of distinct treatment-related changes in glioblastoma could be observed [18, 30].

In this work, we obtained high resolution protein-weighted CEST spectra at a 7 Tesla MRI scanner in untreated glioblastoma patients. We hypothesized that the resulting CEST contrasts can serve as predictors of treatment response in newly-diagnosed glioblastoma patients.


To summarize the study design, all patients underwent protein-weighted CEST imaging on a 7T MRI prior to first-line treatment. The resulting CEST spectra were fitted using a five-pool multi-Lorentzian analysis which yielded the Lorentzian difference (LD) of NOE (NOE-LD) and APT (APT-LD) contrasts [14, 21]. A NOE-weighted MTRasym contrast was calculated as well [15]. In addition, the apparent exchange-dependent relaxation rate (AREX) was applied to correct for non-CEST effects (spillover, T1- and T2-relaxation, semi-solid magnetization transfer) [22]. Thus, NOE-AREX and APT-AREX contrasts were available, too. Finally, the downfield-NOE-suppressed (dns) APT was calculated to yield a further isolated APT contrast [29]. After the end of treatment, all patient cases were classified as early progression or stable disease based on the updated RANO criteria [31]. Subsequently, each CEST contrast was compared between the two groups of early progressors and stable disease based on the RANO criteria.

Patient characteristics are summarized in Table 1. In the investigated study cohort, eight patients presented early progression following first-line standard treatment, whereas the disease remained stable in twelve patients (Figure 1).

Table 1: Patient characteristics




 Median age


 Interquartile range

















 No IDH1-mutation



MGMT-promotor status

Stable disease (N)

Stable disease [%]

Early progression (N)

Early progression [%]






 Not methylated





 Not assessed











 Subtotal resection



 Gross total resection




Stable disease (N)

Stable disease [%]

Early progression (N)

Early progression [%]






 No resection








 60 Gy/2 Gy + TMZ



 40.05 Gy/2.67 Gy ± TMZ



Initial response (RANO)



 Stable disease



 Early progression



(IDH = isocitrate-dehydrogenase 1, MGMT = O6-methylguanine DNA methyltransferase, Gy = Gray, TMZ = temozolomide).

Enrollment of patients and consecutive response assessment.

Figure 1: Enrollment of patients and consecutive response assessment. (n/a = data not available, GBM = glioblastoma multiforme, SD = stable disease, PD = progressive disease).

Pre-treatment mean signal intensities of early-progressive tumors showed generally lower values on NOE-weighted contrasts (NOE-LD: median stable disease = 11.66, median early progression = 10.37, p = 0.0001 and NOE-AREX: median stable disease = 9.91, median early progression = 8.95, p = 0.1288). Accordingly, the NOE-weighted MTRasym (median stable disease = 5.71, median early progression = 4.52, p = 0.0186) presented higher values in case of early progression (Figure 2, NOE-weighted imaging). The difference between early progression and stable disease reached statistical significance for NOE-LD and NOE-weighted MTRasym. For APT-weighted measures, there was a trend towards higher pre-treatment mean tumor signal intensities in early-progressive disease compared to stable disease (APT-LD: median stable disease = 5.28, median early progression = 5.36, p = 0.7725; APT-AREX: median stable disease = 4.22, median early progression = 4.73, p = 0.3218; dns-APT: median stable disease = 2.14, median early progression = 2.71, p = 0.0328) (Figure 2, APT-weighted imaging). This trend increased with isolation of the APT signal, reaching a significant difference between the two groups only in case of the most isolated dns-APT contrast. Correspondingly, ROC AUC analysis showed that a statistically significant prediction of early progression after first-line standard treatment was possible employing NOE-LD (AUC = 0.98, p = 0.0005), NOE-weighted MTRasym (AUC = 0.83, p = 0.0166) and dns-APT (AUC = 0.80, p = 0.0318) measures (Figure 3A and 3B). The highest predictive accuracy was achieved utilizing the NOE-LD with a sensitivity of 91% and a specificity of 100%. MTRasym yielded a sensitivity of 73% and specificity of 100%, whereas dns-APT reached sensitivity and specificity values of 82% and 88%.

Description of pre-treatment differences for all parameters.

Figure 2: Description of pre-treatment differences for all parameters. Pre-treatment differences of tumor mean signal intensity for all MRI contrasts as well as patient age regarding therapy response. NOE-weighted imaging: NOE-LD (a1) and -AREX (a2) show a lower signal in the patients with early progresses, whereas the opposite holds true for NOE-weighted MTRasym (a3). APT-weighted imaging: APT-weighted contrasts show an increasing tendency towards higher values in the early-progressive group from left (APT-LD, b1) to right (dns-APT, b3). It seems that the more isolated the APT-contribution, the clearer this tendency gets. Clinical parameters: No clear intergroup difference in mean signal intensities of T2w-TSE at 7T (c1) and patient age can (c3) be observed. There seems to be a slight trend to lower values in early progression concerning mean ADC signals (c2). (*** = statistically significant according to Mann–Whitney U test with α ≤ 0.05, n.s. = not statistically significant).

Receiver Operating Characteristics (ROC) graphs.

Figure 3: Receiver Operating Characteristics (ROC) graphs. (A) NOE-weighted CEST including NOE-LD (red, AUC = 0.98, p = 0.0005) and MTRasym (yellow, AUC = 0.83, p = 0.0166) which provide accurate prediction of early progression and reached statistical significance as opposed to NOE-AREX (orange, AUC = 0.72, p = 0.1167). (B) APT-weighted CEST with dns-APT (green, AUC = 0.80, p = 0.0318) being the only accurate and significant predictor of early progression compared to APT-AREX (blue, AUC = 0.64, p = 0.3218) and APT-LD (cyan, AUC = 0.50, p = 1). (C) Clinical parameters T2w-TSE at 7T (black, AUC = 0.56, p = 0.6434), ADC at 3T (dashed purple, AUC = 0.56, p = 0.6797) and patient age (grey, AUC = 0.56, p = 0.6434) do not show good predictive accuracy with AUC values close to 0.5 and did not reach statistical significance either.

None of the investigated clinical parameters (T2-weighted turbo spin echo sequence (T2w-TSE) at 7T: median stable disease = 554, median early progression = 563, p = 0.6784; apparent diffusion coefficient (ADC) at 3T: median stable disease = 10.72, median early progression = 10.35, p = 0.7168; Age: median stable disease = 60 years, median early progression = 60 years, p = 0.6710) presented any significant difference between early-progressive and stable disease, even though a slight tendency towards lower values in early progression was observed for the ADC contrast at 3T (Figure 2, Clinical parameters). Consequently, clinical parameters were not able to predict early progression according to ROC AUC analysis (Figure 3C). Furthermore, the descriptive analysis of MGMT-promotor methylation status and extent of surgery showed merely slight differences between patients experiencing early progression and stable disease for both parameters (Table 1). There was no significant interdependence of the MGMT-status and response to treatment (p = 1.00) or the extent of surgery and response to treatment (p = 1.00).

All statistical findings are summarized in Table 2. Figure 4 shows exemplary patient images comparing an early progression with a stable disease for all investigated contrasts. Close-up views of tumor region are additionally provided (Supplementary Figure 1).

Table 2: Results of statistical analysis


Stable disease (median & iqr)

Early progression (median & iqr)

p-val. (U-test)

AUC (95% CI)

p-val. (AUC)

Best cut-off

Sensitivity (95% CI)

Specificity (95% CI)


11.66 (11.18–12.31)

10.37 (10.31 – 10.48)


0.98 (0.92 – 1.00)



0.91 (0.82 – 1.00)

1.00 (0.88 – 1.00)


9.91 (9.25 – 11.90)

8.95 (8.25 – 9.87)


0.72 (0.48 – 0.95)



0.64 (0.27 – 1.00)

0.75 (0.38 – 1.00)

MTR-asym (NOEw)

−5.71 (−6.41 – − 4.82)

-4.52 (−4.90 – 3.39)


0.83 (0.64 – 1.00)



0.73 (0.45 – 1.00)

1.00 (0.50 – 1.00)


5.28 (5.12 – 5.39)

5.36 (4.90 – 6.06)


0.50 (0.18 – 0.82)



0.91 (0.18 – 1.00)

0.38 (0.13 – 1.00)


4.22 (3.85 – 4.76)

4.73 (4.27 – 4.80)


0.64 (0.37 – 0.90)



0.64 (0.18 – 0.91)

0.75 (0.50 -1.00)


2.14 (1.92 – 2.28)

2.71 (2.56 – 3.09)


0.80 (0.57 – 1.00)



0.82 (0.54 – 1.00)

0.88 (0.63 – 1.00)


554 (485 – 666)

563 (444 – 632)


0.56 (0.17 – 0.71)



0.58 (0.33 – 0.83)

0.50 (0.13 – 0.88)

ADC (3 Tesla)

10.72 (9.92 – 11.63)

10.35 (10.17 – 10.86)


0.56 (0.27 – 0.85)



0.64 (0.09 – 0.91)

0.75 (0.38 – 1.00)


60 years (51 – 70 years)

60 years (54 – 70 years)


0.56 (0.29 – 0.83)



0.88 (0.63 – 1.00)

0.42 (0.17 – 0.67)

(iqr = interquartile range, AUC = area under curve, CI = confidence interval).

Exemplary CEST MR-images.

Figure 4: Exemplary CEST MR-images. Top: Early progression. Bottom: Stable disease. a1 and a2: T1-weighting with gadolinium enhancement (3 Tesla), b1 and b2: T2-weighted TSE (7 Tesla), c1 and c2: ADC-map (3 Tesla), d1 and d2: NOE-LD, e1 and e2: NOE-AREX, f1 and f2: NOE-weighted MTRasym, g1 and g2: APT-LD, h1 and h2: APT-AREX, i1 and i2: dns-APT. NOE-LD and -AREX-mediated images show a decreased tumor signal, with much lower values in the early progression case (arrows in d1e1 compared to d2e2). The opposite is true for the MTRasym contrasts (arrow in f1 compared to f2). APT-mediated images show a considerably higher signal intensity within the early-progressive tumor (arrows in g1i1 compared to g2i2).


In this study, we investigated high resolution CEST spectra obtained on a 7T MRI scanner allowing for simultaneous quantification of isolated APT- and NOE-weighted contrasts in glioblastoma patients prior to treatment. We showed that NOE-mediated contrasts, namely the NOE-LD and NOE-weighted MTRasym, as well as the dns-APT contrast enabled accurate prediction of early progression following first-line treatment.

At present, early evaluation of therapy response in glioblastoma is a huge clinical challenge. The state of the art is based on repeated post-treatment MRI examinations which face major limitations early after treatment due to possible pseudo progression [3133]. Thus, a reliable identification of patients who did not respond to adjuvant CRT is substantially delayed, which in turn results in delayed start of salvage therapies indicated in these cases. This is particularly problematic in view of the fact that many patients suffer from early tumor progression [1, 3]. Recently, CEST MRI already showed the potential to enhance post-treatment response evaluation in glioblastoma [18, 30]. A pre-treatment assessment of tumor sensitivity towards standard therapy would allow the assignment of patients to the most promising therapeutic approach right from the beginning. MGMT promotor methylation status seems to be a helpful tool to assess chemosensitivity especially in the elderly population [47]. However, its clinical value concerning younger glioblastoma patients remains controversial due to several methodological and normative limitations [4, 810]. At this point, broadening the application of CEST MRI in the diagnostic work-up to obtain pre-treatment response predictors might be highly valuable. The resulting non-invasive imaging biomarkers could provide a convenient way to repeatedly evaluate the tumor throughout disease management. Moreover, CEST MRI might guide biopsies to relevant tumor sites, thus also enhancing the efficacy of histopathological tools. This could be an important step towards individualized therapy.

To our knowledge, this is the first approach employing CEST MRI as predictor of glioblastoma response to standard therapy in previously untreated patients. So far, APT imaging has shown promising results in post-treatment evaluation of tumor response. A drop in APT-signal after chemotherapy [34] or radiotherapy [35] correlated with decreasing cellularity and might serve as biomarker of response in glioblastoma. Furthermore, APT-weighted imaging could significantly distinguish between tumor progression and treatment related changes in glioblastoma with high APT values being characteristic for progression [18, 30]. Consequently, APT signal seems to be correlated with proliferation [18, 20, 34], thus possibly reflecting tumor aggressiveness and response to therapy. This is in accordance with higher pre-treatment APT signal values in early-progressing tumors found in our work. However, only the most isolated APT-mediated contrast showed a statistically significant difference regarding response to therapy. This result is in agreement with a recent study by Zhang et al. reporting increased correlation with tumor proliferation for isolated APT signals compared to conventional approaches based on MTRasym [36].

Concerning NOE-mediated contrasts, it was previously shown that post-treatment NOE-weighted signals can significantly differentiate between tumor progression and radiation necrosis in brain metastases [37] whereas no significant correlation of any pre-treatment CEST signals in tumor tissue with consecutive therapy response could be found [38]. This is partly contrary to our study results, where NOE-weighted CEST provided highest diagnostic accuracy. However, it is important to note that Desmond et al. [38] worked with different imaging parameters because the particularly low B1-amplitudes (0.6–1.0 μT) which we applied at a static field of 7T especially enhance NOE-mediated CEST effects [14, 15, 23, 28, 39]. In addition, we investigated glioblastoma patients instead of brain metastasis, with numerous studies describing a drop of NOE signals within glioma [14, 15, 22, 28, 29]. This drop seems to be more pronounced in higher-grade tumors [40], which corresponds with our finding of predominantly low NOE values in early-progressing tumors. Better isolation of different CEST signals also led to a better predictive accuracy when comparing the NOE-LD with the NOE-weighted MTRasym. However, the additionally relaxation-compensated NOE-AREX metric yielded lower AUC values than NOE-LD. This is most likely due to a higher standard-deviation of NOE-AREX values resulting from the inverse metric approach [21]. The higher statistical errors of NOE-AREX values may in turn have decreased the effect size in our patient sample.

The origins of the different CEST signals are subject to ongoing research, but previously demonstrated contributions of protein concentration [14, 25], pH [12, 21, 25], protein conformation [23, 24, 41] and cellular proliferation [18, 20, 28, 34] imply that CEST MRI might yield non-invasive biomarkers for essential tumor characteristics.

This study found an essential difference in NOE- and APT-weighted CEST contrasts as predictors of early progression in glioblastomas. NOE-weighted contrasts showed a decrease in early-progressive tumors, whereas APT-weighted measures showed an opposite trend with increased APT signals in early progressors. As mentioned above, both APT and NOE mediated signals are primarily associated with protein/ peptide content and cellularity. Since APT and NOE show opposite signal alteration in tumor tissue, this result cannot only be caused by variations in protein concentration. A possible explanation is the decomposition of proteins/ peptides in tumors yielding an increased proportion of smaller protein fragments and peptides [42]. This would, in turn, result in increased APT effects (better accessibility of amide protons to bulk water exchange) and simultaneously decreased NOE signal, since the NOE has been shown to be strongly affected by protein size and conformation [23, 24, 41]. These effects were even more pronounced in the group of early progressors, possibly reflecting particularly aggressive tumor tissue.

A frequently reported pH-dependence of NOE effects, with low values being associated to low pH, has most likely only minor contributions to the observed signal change [14].

Moreover, CEST contrasts have been shown to be sensitive to histopathological features of gliomas such as tumor grade, isocitrate-dehydrogenase (IDH) -mutations and MGMT promotor status [19, 20, 26, 27, 43]. This supports the approach of employing CEST MRI as early response predictor in glioblastoma. Furthermore, all evaluated CEST signals provide complementary information to clinical MRI and methodological isolation clearly separates different CEST effects from each other [21, 28, 29]. Consequently, future approaches should investigate multiparametric prediction models based on various isolated CEST contrasts, traditional MRI parameters, and clinical information in order to further increase diagnostic performance of MRI.

Our study has some limitations. First, validation in a bigger patient trial is mandatory. However, this is the first study examining the value of multi-pool CEST MRI as response predictor, yielding statistically significant results in a prospective setting. Secondly, 7T MRI scanners are only available at few centers today which limits immediate implementation. Nevertheless, successful isolation of CEST-effects at clinical MRI scanners (3.0 T) in various diagnostic studies [3638, 44] showed that translation of our findings into standard protocols is feasible. Still, our results are only valid for low B1-amplitudes at the high B0-field of 7T and reproducibility at clinical scanners needs future verification. Another possible limitation is the assessment of general standard first-line therapy involving different surgical approaches and age-adapted radiotherapy protocols in elderly glioblastoma patients. However, our study cohort represents a common clinical spectrum, which seemingly did not impede the significance of CEST MRI, underlining the robustness of this method. Furthermore, we did not find a significant difference between early progression and stable disease according to patient age or performed surgical approach. In contrast, CEST MRI yielded significant intergroup differences, thus providing the only considerable predictors of early tumor progression in this trial. Finally, the correlation of CEST effects with prognostic histopathological features [19, 20, 26, 27, 36] could have interfered with response prediction. However, we exclusively included grade IV glioblastomas and just one patient carried an IDH1-mutation in this study. Moreover, our findings showed only slight differences in MGMT-promotor status regarding therapy response. This strongly suggests that CEST based contrasts, which yielded significant prediction of early tumor progression, provide complementary information to histopathology. Ultimately, the response assessment serving as a reference in this study is based on the updated RANO criteria [31], which face limitations in the initial setting due to possible pseudo progression [32, 33]. Nevertheless, RANO criteria are increasingly employed in neuro-oncology trials as the current state of the art in objective response assessment. In our study, we followed the recommendations of the RANO working group to validate apparent tumor progression within 12 weeks post-treatment through a second follow-up examination [31]. This allowed us to detect early pseudo progression successfully in several patients.

In conclusion, multi-pool CEST MRI derived contrasts, particularly NOE-weighted imaging, yielded the potential to predict early tumor progression after first-line therapy in previously untreated glioblastoma patients and might therefore be a promising non-invasive tool for customization of treatment in the future.



From October 2015 to September 2017, 40 consecutive patients with recently-diagnosed intracranial tumors underwent a 7T MRI examination prior to treatment. MRI inclusion criteria were an age of 18 years or older, prior findings suspicious for glioma, no previous treatment and eligibility for 7T MRI. Moreover, for evaluation of the predictive capacities of CEST at 7T, histopathologically confirmed diagnosis of glioblastoma multiforme (GBM, WHO grade IV) and complete clinical records from follow-up examinations were required. Finally, 20 patients were enrolled in this study, whereas the rest had to be excluded due to histopathological diagnosis other than glioblastoma or missing clinical data (Figure 1). Eleven patients were previously reported as part of a different patient cohort [22, 29]. The CEST spectra obtained in one patient were strongly distorted due to motion artifacts, hence only clinical imaging of this patient was eligible for evaluation. Patient characteristics are summarized in Table 1. Resection was performed in twelve patients, whereas eight did not receive a therapeutic resection. Subsequently, all patients underwent adjuvant treatment consisting of radiotherapy (standard protocol: 60 Gray, 30 fractions) with concomitant (75 mg/m²) and adjuvant (150–200 mg/m²) administration of temozolomide. Therapy was adapted following suggestions for elderly [6, 7] in five patients (40.05 Gy, 15 fractions and/or temozolomide in the standard dose).

Response assessment

Patient response status to treatment was determined based on clinical 3T MRI and neurological evaluation derived from the first and second follow-up examinations (approx. one and three months after the end of radiotherapy). Radiological findings were routinely evaluated by the department of neuroradiology in accordance with the updated RANO criteria [31]. Subsequently, comprehensive evaluation of radiological and neurological status was done by a neurooncologist following the updated RANO criteria [31]. Results obtained in each examination were rated as complete response (CR), partial response (PR), stable disease (SD) or progressive disease (PD). For all patients classified as potential PD in the first follow up, the rating of the second follow up was considerable to account for possible pseudo progression within the first 12 weeks [3133]. Only if evaluation of both time steps led to the conclusion that PD was present, this case was rated as PD in accordance with the RANO criteria [31]. Otherwise, the classification of the second follow-up overruled the initial rating, thus defining a pseudo progression.

The whole workflow of this study including the response assessment is summarized in Figure 1. In total, twelve cases were classified as SD. Conversely, eight patients suffered from PD, thus early progression was present.

Clinical MRI at 3T

Clinical MRI exams were performed at 3T prior to therapy and as part of follow up examinations based on the following protocol parameters: T2w-TSE [echo time (TE) = 86 ms; repetition time (TR) = 5550 ms; field of view (FoV): 229 × 172 mm²; matrix: 384 × 230; slice thickness: 5 mm], T2-weighted fluid attenuated inversion recovery (T2w-FLAIR) [TE = 135 ms; TR = 8500 ms; FoV: 230 × 172 mm²; matrix: 256 × 192; slice thickness: 5 mm], T1-weighted gadolinium contrast-enhanced (T1w-gdce) [TE = 4.04 ms; TR = 1710 ms; FoV: 256 × 256 mm²; matrix: 512 × 512; slice thickness: 1 mm] and diffusion imaging yielding the ADC [echo planar readout, TE = 90 ms; TR = 5300 ms; b = 0 s/mm² and b = 1200 s/mm²; FoV: 229 × 229 mm²; matrix: 130 × 130; slice thickness: 5 mm]. This is in accordance with the consensus recommendations for standardized brain tumor imaging and response evaluation as proposed by Ellingson et al. [45].


The examinations were performed on a 7T MRI scanner (MAGNETOM 7.0 Tesla; Siemens Healthcare, Erlangen, Germany) with a single channel transmit/24 channel receive 1H head coil (Nova Medical, Wilmington, USA) prior to therapy. CEST imaging based on a custom-developed 2D gradient echo (GRE) sequence according to Zaiss et al. [22]. In detail, images were obtained after saturation [train of 150 Gaussian shaped radiofrequency (RF) pulses, tp = 15 ms, td = 10 ms, duty-cycle = 60%, tsat = 3.75 s] for two distinct B1 amplitudes [1.0 μT and 0.6 μT] and for three adjacent slices [slice thickness = 5 mm] [22]. MTRasym was calculated at 3.5 ppm [16], yet yielding a predominantly NOE-mediated contrast due to the low B1-amplitudes (0.6–1.0 μT) applied at 7 Tesla [14, 23, 39]. Separation of NOE and APT-mediated CEST effects was based on a multi-Lorentzian fitting approach as previously described [22], which was applied on the high-resolution CEST spectra at 7T. Subsequently, different NOE- and APT-weighted contrasts were evaluated at B1 = 0.6 μT and corrected for field inhomogeneities according to Windschuh et al. [46], the first being the Lorentzian difference (LD) [14]. Additional application of the relaxation-compensated metric according to Zaiss et al. [21] led to the apparent exchange-dependent relaxation rate (AREX) which is corrected for spillover, T1- and T2-relaxation and semi-solid magnetization transfer [22]. Furthermore, residual overlap of downfield-resonating NOE with the APT signal was removed using the downfield-NOE-suppressed (dns) APT contrast as reported by Zaiss et al. [29]. Mapping of B0 and B1 inhomogeneities was achieved following the simultaneous mapping of water shift and B1 (WASABI) approach [47]. In total, the CEST MRI scans required 22–25 min including shimming, B0/B1 correction and relaxation-compensation. Exemplary CEST spectra from the tumor region and contralateral normal appearing white matter are provided, both for a patient with early progression and stable disease (Supplementary Figure 2).

In addition, high-resolution T2w-TSE imaging was performed [TE = 54 ms; TR = 14130 ms; FoV: 220 × 178.8 mm²; matrix: 512 × 416.3; slice thickness: 2 mm] as previously reported [48].

Data analysis

Co-registration of all images was performed employing an automatic multi-modal rigid registration algorithm in MITK [49]. Manual segmentation of the tumor region including all areas of abnormal signal intensity on T1-gdce and T2-weighted images but excluding necrosis was done by an experienced neuroradiologist (A.R., 10 years of experience). This approach enables better reproducibility than manual selection of distinct regions of interest within the tumor. Subsequently, the mean signal intensities over all included voxels were calculated for the ADC obtained at 3T and for all multi-pool CEST contrasts and the T2w-TSE obtained at 7T.

Statistical analysis

The median and interquartile range of the tumor mean signal intensities were calculated for each contrast as well as patient age in the early-progression and stable disease group respectively. Mann–Whitney U tests were performed to compare the signal intensities and age between patients with stable disease and patients experiencing early progression. In addition, receiver operating characteristic (ROC) area under the curve (AUC) analyses were applied to assess prediction of early progression. Consecutively, best thresholds were determined for each contrast following Youden’s J statistics. Moreover, a possible interdependence of the MGMT-promotor status and therapy response as well as the extent of the surgical approach and therapy response was tested employing Fisher’s exact test. The data analysis employed R version 3.4.3 and the pROC package [50]. The level of significance was set to p < 0.05 for all performed tests.

Ethics statement

This study was conducted in accordance with the declaration of Helsinki and received approval by the local ethics committee. MRI examinations were performed after written informed consent was obtained from the patient.


ADC: apparent diffusion coefficient; APT: amide proton transfer; AREX: apparent exchange-dependent relaxation rate; AUC: area under the curve; CEST: chemical exchange saturation transfer; CR: complete response; CRT: chemoradiotherapy; dns-APT: downfield- nuclear Overhauser effect -suppressed amide proton transfer; FLAIR: fluid attenuated inversion recovery; FoV: field of view; GBM: glioblastoma multiforme; gdce: gadolinium contrast-enhanced; GRE: gradient echo; IDH1: isocitrate-dehydrogenase 1; LD: Lorentzian difference; MGMT: O6-methylguanine DNA methyltransferase; MITK: medical imaging interaction toolkit; MRI: magnetic resonance imagin; MTRasym: asymmetric magnetic transfer ratio; NOE: nuclear Overhauser effect; PD: progressive disease; PR: partial response; RANO: response assessment in neuro-oncology working group; RF: radiofrequency; ROC: receiver operating characteristic; SD: stable disease; T: Tesla; TE: echo time; TMZ: Temozolomide; TR: repetition time; TSE: turbo spin echo; WHO: world health organization.

Author contributions

Study concepts/study design (all authors), data analysis and interpretation (all authors), manuscript writing and final approval (all authors), implementation of methods and workflow (D.P., M.Z., A.R., J.W.), clinical/experimental studies (S.Re., D.P, S.A., J.O., C.D. J.E.M., J.W, M.Z.), statistical analysis (S.Re., D.P.), Guarantors of integrity of entire study (D.P., M.E.L., H.P.S.).


The authors declare no potential conflicts of interest.


No financial support to declare.


1. Ostrom QT, Gittleman H, Liao P, Vecchione-Koval T, Wolinsky Y, Kruchko C, Barnholtz-Sloan JS. CBTRUS Statistical Report: Primary brain and other central nervous system tumors diagnosed in the United States in 2010-2014. Neuro Oncol. 2017; 19:v1–v88. https://doi.org/10.1093/neuonc/nox158.

2. Stupp R, Mason WP, van den Bent MJ, Weller M, Fisher B, Taphoorn MJ, Belanger K, Brandes AA, Marosi C, Bogdahn U, Curschmann J, Janzer RC, Ludwin SK, et al. Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. N Engl J Med. 2005; 352:987–96. https://doi.org/10.1056/NEJMoa043330.

3. Stupp R, Hegi ME, Mason WP, van den Bent MJ, Taphoorn MJ, Janzer RC, Ludwin SK, Allgeier A, Fisher B, Belanger K, Hau P, Brandes AA, Gijtenbeek J, et al. Effects of radiotherapy with concomitant and adjuvant temozolomide versus radiotherapy alone on survival in glioblastoma in a randomised phase III study:5–year analysis of the EORTC-NCIC trial. Lancet Oncol. 2009; 10:459–66. https://doi.org/10.1016/s1470-2045(09)70025-7.

4. Stupp R, Brada M, van den Bent MJ, Tonn JC, Pentheroudakis G. High-grade glioma: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2014; 25:93–101. https://doi.org/10.1093/annonc/mdu050.

5. Hegi ME, Diserens AC, Gorlia T, Hamou MF, de Tribolet N, Weller M, Kros JM, Hainfellner JA, Mason W, Mariani L, Bromberg JE, Hau P, Mirimanoff RO, et al. MGMT gene silencing and benefit from temozolomide in glioblastoma. N Engl J Med. 2005; 352:997–1003. https://doi.org/10.1056/NEJMoa043331.

6. Malmstrom A, Gronberg BH, Marosi C, Stupp R, Frappaz D, Schultz H, Abacioglu U, Tavelin B, Lhermitte B, Hegi ME, Rosell J, Henriksson R. Temozolomide versus standard 6-week radiotherapy versus hypofractionated radiotherapy in patients older than 60 years with glioblastoma: the Nordic randomised, phase 3 trial. Lancet Oncol. 2012; 13:916–26. https://doi.org/10.1016/s1470-2045(12)70265-6.

7. Wick W, Platten M, Meisner C, Felsberg J, Tabatabai G, Simon M, Nikkhah G, Papsdorf K, Steinbach JP, Sabel M, Combs SE, Vesper J, Braun C, et al. Temozolomide chemotherapy alone versus radiotherapy alone for malignant astrocytoma in the elderly: the NOA-08 randomised, phase 3 trial. Lancet Oncol. 2012; 13:707–15. https://doi.org/10.1016/s1470-2045(12)70164-x.

8. Weller M, Stupp R, Reifenberger G, Brandes AA, van den Bent MJ, Wick W, Hegi ME. MGMT promoter methylation in malignant gliomas: ready for personalized medicine? Nat Rev Neurol. 2010; 6:39–51. https://doi.org/10.1038/nrneurol.2009.197.

9. Wick W, Weller M, van den Bent M, Sanson M, Weiler M, von Deimling A, Plass C, Hegi M, Platten M, Reifenberger G. MGMT testing—the challenges for biomarker-based glioma treatment. Nat Rev Neurol. 2014; 10:372–85. https://doi.org/10.1038/nrneurol.2014.100.

10. Taylor JW, Schiff D. Treatment considerations for MGMT-unmethylated glioblastoma. Curr Neurol Neurosci Rep. 2015; 15:507. https://doi.org/10.1007/s11910-014-0507-z.

11. Ward KM, Aletras AH, Balaban RS. A new class of contrast agents for MRI based on proton chemical exchange dependent saturation transfer (CEST). J Magn Reson. 2000; 143:79–87. https://doi.org/10.1006/jmre.1999.1956.

12. Zhou J, Payen JF, Wilson DA, Traystman RJ, van Zijl PC. Using the amide proton signals of intracellular proteins and peptides to detect pH effects in MRI. Nat Med. 2003; 9:1085–90. https://doi.org/10.1038/nm907.

13. van Zijl PC, Yadav NN. Chemical exchange saturation transfer (CEST): what is in a name and what isn‘t? Magn Reson Med. 2011; 65:927–48. https://doi.org/10.1002/mrm.22761.

14. Jones CK, Huang A, Xu J, Edden RA, Schar M, Hua J, Oskolkov N, Zaca D, Zhou J, McMahon MT, Pillai JJ, van Zijl PC. Nuclear Overhauser enhancement (NOE) imaging in the human brain at 7T. Neuroimage. 2013; 77:114–24. https://doi.org/10.1016/j.neuroimage.2013.03.047.

15. Paech D, Zaiss M, Meissner JE, Windschuh J, Wiestler B, Bachert P, Neumann JO, Kickingereder P, Schlemmer HP, Wick W, Nagel AM, Heiland S, Ladd ME, et al. Nuclear Overhauser Enhancement Mediated Chemical Exchange Saturation Transfer Imaging at 7 Tesla in Glioblastoma Patients. PLoS One. 2014; 9:e104181. https://doi.org/10.1371/journal.pone.0104181.

16. Zhou J, Lal B, Wilson DA, Laterra J, van Zijl PCM. Amide proton transfer (APT) contrast for imaging of brain tumors. Magnetic Resonance in Medicine. 2003; 50:1120–6. https://doi.org/10.1002/mrm.10651.

17. Guivel-Scharen V, Sinnwell T, Wolff SD, Balaban RS. Detection of proton chemical exchange between metabolites and water in biological tissues. J Magn Reson. 1998; 133:36–45. https://doi.org/10.1006/jmre.1998.1440.

18. Park JE, Kim HS, Park KJ, Kim SJ, Kim JH, Smith SA. Pre- and Posttreatment Glioma: Comparison of Amide Proton Transfer Imaging with MR Spectroscopy for Biomarkers of Tumor Proliferation. Radiology. 2016; 278:514–23. https://doi.org/10.1148/radiol.2015142979.

19. Togao O, Hiwatashi A, Yamashita K, Kikuchi K, Keupp J, Yoshimoto K, Kuga D, Yoneyama M, Suzuki SO, Iwaki T, Takahashi M, Iihara K, Honda H. Grading diffuse gliomas without intense contrast enhancement by amide proton transfer MR imaging: comparisons with diffusion- and perfusion-weighted imaging. Eur Radiol. 2017; 27:578–88. https://doi.org/10.1007/s00330-016-4328-0.

20. Togao O, Yoshiura T, Keupp J, Hiwatashi A, Yamashita K, Kikuchi K, Suzuki Y, Suzuki SO, Iwaki T, Hata N, Mizoguchi M, Yoshimoto K, Sagiyama K, et al. Amide proton transfer imaging of adult diffuse gliomas: correlation with histopathological grades. Neuro Oncol. 2014; 16: 441–8. https://doi.org/10.1093/neuonc/not158.

21. Zaiss M, Xu J, Goerke S, Khan IS, Singer RJ, Gore JC, Gochberg DF, Bachert P. Inverse Z-spectrum analysis for spillover-, MT-, and T1 -corrected steady-state pulsed CEST-MRI—application to pH-weighted MRI of acute stroke. NMR Biomed. 2014; 27:240–52. https://doi.org/10.1002/nbm.3054.

22. Zaiss M, Windschuh J, Paech D, Meissner JE, Burth S, Schmitt B, Kickingereder P, Wiestler B, Wick W, Bendszus M, Schlemmer HP, Ladd ME, Bachert P, et al. Relaxation-compensated CEST-MRI of the human brain at 7T: Unbiased insight into NOE and amide signal changes in human glioblastoma. Neuroimage. 2015; 112:180–8. https://doi.org/10.1016/j.neuroimage.2015.02.040.

23. Zaiss M, Kunz P, Goerke S, Radbruch A, Bachert P. MR imaging of protein folding in vitro employing nuclear-Overhauser-mediated saturation transfer. NMR Biomed. 2013; 26:1815–22. https://doi.org/10.1002/nbm.3021.

24. Goerke S, Zaiss M, Kunz P, Klika KD, Windschuh JD, Mogk A, Bukau B, Ladd ME, Bachert P. Signature of protein unfolding in chemical exchange saturation transfer imaging. NMR Biomed. 2015; 28:906–13. https://doi.org/10.1002/nbm.3317.

25. Jin T, Wang P, Zong X, Kim SG. MR imaging of the amide-proton transfer effect and the pH-insensitive nuclear overhauser effect at 9.4 T. Magn Reson Med. 2013; 69: 760–70. https://doi.org/10.1002/mrm.24315.

26. Jiang S, Rui Q, Wang Y, Heo HY, Zou T, Yu H, Zhang Y, Wang X, Du Y, Wen X, Chen F, Wang J, Eberhart CG, et al. Discriminating MGMT promoter methylation status in patients with glioblastoma employing amide proton transfer-weighted MRI metrics. Eur Radiol. 2018; 28:2115–2123. https://doi.org/10.1007/s00330-017-5182-4.

27. Jiang S, Zou T, Eberhart CG, Villalobos MAV, Heo HY, Zhang Y, Wang Y, Wang X, Yu H, Du Y, van Zijl PCM, Wen Z, Zhou J. Predicting IDH mutation status in grade II gliomas using amide proton transfer-weighted (APTw) MRI. Magn Reson Med. 2017; 78:1100–9. https://doi.org/10.1002/mrm.26820.

28. Paech D, Burth S, Windschuh J, Meissner JE, Zaiss M, Eidel O, Kickingereder P, Nowosielski M, Wiestler B, Sahm F, Floca RO, Neumann JO, Wick W, et al. Nuclear Overhauser Enhancement imaging of glioblastoma at 7 Tesla: region specific correlation with apparent diffusion coefficient and histology. PLoS One. 2015; 10:e0121220. https://doi.org/10.1371/journal.pone.0121220.

29. Zaiss M, Windschuh J, Goerke S, Paech D, Meissner JE, Burth S, Kickingereder P, Wick W, Bendszus M, Schlemmer HP, Ladd ME, Bachert P, Radbruch A. Downfield-NOE-suppressed amide-CEST-MRI at 7 Tesla provides a unique contrast in human glioblastoma. Magn Reson Med. 2017; 77:196–208. https://doi.org/10.1002/mrm.26100.

30. Park KJ, Kim HS, Park JE, Shim WH, Kim SJ, Smith SA. Added value of amide proton transfer imaging to conventional and perfusion MR imaging for evaluating the treatment response of newly diagnosed glioblastoma. Eur Radiol. 2016; 26:4390–403. https://doi.org/10.1007/s00330-016-4261-2.

31. Wen PY, Macdonald DR, Reardon DA, Cloughesy TF, Sorensen AG, Galanis E, Degroot J, Wick W, Gilbert MR, Lassman AB, Tsien C, Mikkelsen T, Wong ET, et al. Updated response assessment criteria for high-grade gliomas: response assessment in neuro-oncology working group. J Clin Oncol. 2010; 28:1963–72. https://doi.org/10.1200/jco.2009.26.3541.

32. Brandsma D, Stalpers L, Taal W, Sminia P, van den Bent MJ. Clinical features, mechanisms, and management of pseudoprogression in malignant gliomas. Lancet Oncol. 2008; 9:453–61. https://doi.org/10.1016/s1470-2045(08)70125-6.

33. de Wit MC, de Bruin HG, Eijkenboom W, Sillevis Smitt PA, van den Bent MJ. Immediate post-radiotherapy changes in malignant glioma can mimic tumor progression. Neurology. 2004; 63:535–7.

34. Sagiyama K, Mashimo T, Togao O, Vemireddy V, Hatanpaa KJ, Maher EA, Mickey BE, Pan E, Sherry AD, Bachoo RM, Takahashi M. In vivo chemical exchange saturation transfer imaging allows early detection of a therapeutic response in glioblastoma. Proc Natl Acad Sci U S A. 2014; 111:4542–7. https://doi.org/10.1073/pnas.1323855111.

35. Hong X, Liu L, Wang M, Ding K, Fan Y, Ma B, Lal B, Tyler B, Mangraviti A, Wang S, Wong J, Laterra J, Zhou J. Quantitative multiparametric MRI assessment of glioma response to radiotherapy in a rat model. Neuro Oncol. 2014; 16:856–67. https://doi.org/10.1093/neuonc/not245.

36. Zhang J, Zhu W, Tain R, Zhou XJ, Cai K. Improved Differentiation of Low-Grade and High-Grade Gliomas and Detection of Tumor Proliferation Using APT Contrast Fitted from Z-Spectrum. Mol Imaging Biol. 2018 Jan 8. https://doi.org/10.1007/s11307-017-1154-y. [Epub ahead of print].

37. Mehrabian H, Desmond KL, Soliman H, Sahgal A, Stanisz GJ. Differentiation between Radiation Necrosis and Tumor Progression Using Chemical Exchange Saturation Transfer. Clin Cancer Res. 2017; 23:3667–75. https://doi.org/10.1158/1078-0432.ccr-16-2265.

38. Desmond KL, Mehrabian H, Chavez S, Sahgal A, Soliman H, Rola R, Stanisz GJ. Chemical exchange saturation transfer for predicting response to stereotactic radiosurgery in human brain metastasis. Magn Reson Med. 2017; 78:1110–20. https://doi.org/10.1002/mrm.26470.

39. Zhou J, Hong X, Zhao X, Gao JH, Yuan J. APT-weighted and NOE-weighted image contrasts in glioma with different RF saturation powers based on magnetization transfer ratio asymmetry analyses. Magn Reson Med. 2013; 70:320–7. https://doi.org/10.1002/mrm.24784.

40. Heo HY, Jones CK, Hua J, Yadav N, Agarwal S, Zhou J, van Zijl PC, Pillai JJ. Whole-brain amide proton transfer (APT) and nuclear overhauser enhancement (NOE) imaging in glioma patients using low-power steady-state pulsed chemical exchange saturation transfer (CEST) imaging at 7T. J Magn Reson Imaging. 2016; 44:41–50. https://doi.org/10.1002/jmri.25108.

41. Goerke S, Milde KS, Bukowiecki R, Kunz P, Klika KD, Wiglenda T, Mogk A, Wanker EE, Bukau B, Ladd ME, Bachert P, Zaiss M. Aggregation-induced changes in the chemical exchange saturation transfer (CEST) signals of proteins. NMR Biomed. 2017; 30. https://doi.org/10.1002/nbm.3665.

42. Hobbs SK, Shi G, Homer R, Harsh G, Atlas SW, Bednarski MD. Magnetic resonance image-guided proteomics of human glioblastoma multiforme. J Magn Reson Imaging. 2003; 18:530–6. https://doi.org/10.1002/jmri.10395.

43. Paech D, Windschuh J, Oberhollenzer J, Dreher C, Sahm F, Meissner JE, Goerke S, Schuenke P, Zaiss M, Regnery S, Bickelhaupt S, Bäumer P, Bendszus M, et al. Assessing the predictability of IDH mutation and MGMT methylation status in glioma patients using relaxation-compensated multi-pool CEST MRI at 7.0 Tesla. Neuro-Oncology. 2018 May 4. https://doi.org/10.1093/neuonc/noy073. [Epub ahead of print].

44. Chen L, Xu X, Zeng H, Chan KWY, Yadav N, Cai S, Schunke KJ, Faraday N, van Zijl PCM, Xu J. Separating fast and slow exchange transfer and magnetization transfer using off-resonance variable-delay multiple-pulse (VDMP) MRI. Magn Reson Med. 2018 Feb 5. https://doi.org/10.1002/mrm.27111. [Epub ahead of print].

45. Ellingson BM, Bendszus M, Boxerman J, Barboriak D, Erickson BJ, Smits M, Nelson SJ, Gerstner E, Alexander B, Goldmacher G, Wick W, Vogelbaum M, Weller M, et al. Consensus recommendations for a standardized Brain Tumor Imaging Protocol in clinical trials. Neuro Oncol. 2015; 17:1188–98. https://doi.org/10.1093/neuonc/nov095.

46. Windschuh J, Zaiss M, Meissner JE, Paech D, Radbruch A, Ladd ME, Bachert P. Correction of B1-inhomogeneities for relaxation-compensated CEST imaging at 7 T. NMR Biomed. 2015; 28:529–37. https://doi.org/10.1002/nbm.3283.

47. Schuenke P, Windschuh J, Roeloffs V, Ladd ME, Bachert P, Zaiss M. Simultaneous mapping of water shift and B1 (WASABI)-Application to field-Inhomogeneity correction of CEST MRI data. Magn Reson Med. 2017; 77:571–80. https://doi.org/10.1002/mrm.26133.

48. Paech D, Schuenke P, Koehler C, Windschuh J, Mundiyanapurath S, Bickelhaupt S, Bonekamp D, Bäumer P, Bachert P, Ladd ME, Bendszus M, Wick W, Unterberg A, et al. T1ρ-weighted Dynamic Glucose-enhanced MR Imaging in the Human Brain. Radiology. 2017; 285:914–922. https://doi.org/10.1148/radiol.2017162351.

49. Nolden M, Zelzer S, Seitel A, Wald D, Muller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I. The Medical Imaging Interaction Toolkit: challenges and advances : 10 years of open-source development. Int J Comput Assist Radiol Surg. 2013; 8:607–20. https://doi.org/10.1007/s11548-013-0840-8.

50. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Muller M. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011; 12:77. https://doi.org/10.1186/1471-2105-12-77.

Creative Commons License All site content, except where otherwise noted, is licensed under a Creative Commons Attribution 4.0 License.
PII: 25594