The value of arterial spin labelling in adults glioma grading: systematic review and meta-analysis

Metrics: PDF 644 views  |   Full Text 1163 views  |   ?  

Amirah Alsaedi _, Fabio Doniselli, Hans Rolf Jäger, Jasmina Panovska-Griffiths, Antonio Rojas-Garcia, Xavier Golay and Sotirios Bisdas


Amirah Alsaedi1,2, Fabio Doniselli3,4, Hans Rolf Jäger2,5, Jasmina Panovska-Griffiths6, Antonio Rojas-Garcia6, Xavier Golay2 and Sotirios Bisdas2,5

1Department of Radiology Technology, Taibah University, Medina, KSA

2Department of Brain Repair & Rehabilitation, Queen Square Institute of Neurology, University College London, London, UK

3Postgraduate School in Radiodiagnostics, Università degli Studi di Milano, Milan, Italy

4PhD Course in Clinical Research, Università degli Studi di Milano, Milan, Italy

5Lysholm Department of Neuroradiology, The National Hospital for Neurology and Neurosurgery, University College Hospitals NHS Trust, London, UK

6Department of Applied Health Research, University College London, London, UK

Correspondence to:

Amirah Alsaedi, email: amirah.alsaedi.13@ucl.ac.uk

Keywords: glioma; arterial spin labeling; grading

Received: October 30, 2018     Accepted: February 01, 2019     Published: February 22, 2019


This study aimed to evaluate the diagnostic performance of arterial spin labelling (ASL) in grading of adult gliomas. Eighteen studies matched the inclusion criteria and were included after systematic searches through EMBASE and MEDLINE databases. The quality of the included studies was assessed utilizing Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). The quantitative values were extracted and a meta-analysis was subsequently based on a random-effect model with forest plot and joint sensitivity and specificity modelling. Hierarchical summary receiver operating characteristic (HROC) curve analysis was also conducted. The absolute tumour blood flow (TBF) values can differentiate high-grade gliomas (HGGs) from low-grade gliomas (LGGs) and grade II from grade IV tumours. However, it lacked the capacity to differentiate grade II from grade III tumours and grade III from grade IV tumours. In contrast, the relative TBF (rTBF) is effective in differentiating HGG from LGG and in glioma grading. The maximum rTBF (rTBFmax) demonstrated the best results in glioma grading. These results were also reflected in the sensitivity/ specificity analysis in which the rTBFmax showed the highest discrimination performance in glioma grading. The estimated effect size for the rTBF was approximately similar between HGGs and LGGs, and grade II and grade III tumours, (–1.46 (–2.00, –0.91), p- value < 0.001), (–1.39 (–1.89, –0.89), p- value < 0.001), respectively; while it exhibited smaller effect size between grade III and grade IV (–1.05 (–1.82, –0.27)), p < 0.05). Sensitivity and specificity analysis replicate these results as well. This meta-analysis suggests that ASL is useful for glioma grading, especially when considering the rTBFmax parameter.


WHO grade staging of gliomas has implications for prognosis and choice of therapy and MRI plays a leading role in all phases of tumour management, including diagnosis, therapy, and follow-up. T1-weighted post contrast MRI allows identification regions of blood brain barrier (BBB) disruption [1, 2] which are usually associated with higher WHO grades but presence of contrast enhancement can be misleading as some low-grade gliomas (LGGs) demonstrate contrast uptake, with lack of enhancement being observed in some high-grade gliomas (HGGs) [3]. Arterial spin labelling (ASL), which uses magnetically labelled blood water as an inherently diffusible tracer, is now performed in clinical settings as, unlike contrast-enhanced perfusion MRI techniques, it can provide absolute cerebral blood flow (CBF) quantification, eliminates the need for contrast agent, and can be repeated for therapy monitoring. Additionally, ASL is less sensitive to vessels permeability changes, which suggests that it provides tumour perfusion information that reflects vascular density [4].

Quantitative ASL measurements have introduced as output relative (or normalised) tumour blood flow (rTBF) and/or absolute tumour blood flow (TBF) values, which have been reported as useful in distinguishing between HGGs and LGGs [57] and in glioma grading [8] with some studies reporting nevertheless negative results [911]. The purpose of this systematic review and meta-analysis was to address this ambiguity and provide evidence for the diagnostic accuracy of ASL in preoperative glioma grading.


Literature search and selection

This meta-analysis followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines with the research question being ‘What is the diagnostic value of arterial spin labelling (ASL) in the differentiation of glioma grades in adult patients?’ [12]. The search terms were identified according to the Population/Intervention/Comparator/Outcomes (PICO category) recommendations and were linked by Boolean operators (‘OR’ within each PICO category; and ‘AND’ between PICO categories). The identified search terms were framed in concepts. Concept 1 (P): glioma OR neuroglia OR glioma; Concept 2 (I): arterial spin OR artery spin; Concept 3 (O): diagnosis OR grading OR differentiate. Finally, the search was conducted without the third concept in order to cast a wider net. As a result, steps 1 and 2 were combined [(glioma OR neuroglia OR glioma) AND (arterial spin OR artery spin)]. The systematic search was performed in June 2018 through EMBASE ‘(1974 to 8 June 2018)’ and Ovid MEDLINE (R) ‘In-process and other non-indexed citations’ databases to find relevant articles that met the defined search terms; this resulted in the identification of 111 and 48 items respectively. A total of 159 items were found, which were rendered to 122 after duplicates were removed. We included records focusing on pre-treatment glioma grading in adults using ASL. Relevant articles were selected according to PRISMA (see relevant flow chart in Figure 1). 104 of them were excluded as they did not met the research question; for example: they applied ASL in animal models [13] use ASL in order to asses the treatment [14, 15] or applied in pediatric population [16]. Eventually, 18 studies were deemed eligible in terms of the selected inclusion criteria.

Preferred Reporting Items of Systematic Reviews and Meta-Analyses-PRISMA flow chart for the study selection process.

Figure 1: Preferred Reporting Items of Systematic Reviews and Meta-Analyses-PRISMA flow chart for the study selection process.

Eight out of the 18 studies reported the TBF values as a mean and standard deviation, whereas three reported the cut-off values and the corresponding diagnostic sensitivity and specificity rates. The remaining 7 studies provided all the aforementioned information. Due to high variability in expressing the TBF amongst the studies, we decided to rename the studies output as follows: when the region of interest (ROI) included the entire tumour outlined on conventional images, the absolute or relative to the healthy appearing white matter TBF was referred to as TBFmean/rTBFmean. When the tumour ROI was placed on the highest signal in the perfusion map, the TBF is referred as maximum TBFmax/rTBFmax. In some studies, both mean and maximum TBF/rTBF values were reported.

The considerable variability in the study populations, ASL labelling methods, and acquisition parameters in addition to the diversity of post-processing analyses process among the included studies could potentially limit the power of this study evidence thus the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) [17] tool was used by 2 independent researchers to assess the risk of bias of the included studies and their applicability.

Statistical methods

Analyses were performed for a number of tumour WHO grading combinations: HGGs vs. LGGs; grade II vs. grade III; grade II vs. grade IV; grade III vs. grade IV. All outcomes were measured on a continuous scale. The mean, standard deviation (SD) and number of subjects were extracted from each individual study. For studies where the data range was only reported, the SD was assumed to be a quarter of the whole range. Studies with no reported measure of variability (e.g. SD, inter-quartile range, range) were excluded from this analysis (study 14 and 17 for rTBFmax, still they provided the ROC analysis and TBFmax, respectively). Some studies reported the same parameter from both the whole sample and sub-group (e.g. study 8, Supplementary Table 2); whereas some studies reported the same outcome from the same sample using either different ASL techniques (e.g. study 16, Supplementary Table 2) or different image analysis methodology (e.g. study 18, Supplementary Table 1). In such instances, the different data sets from the same study were treated as being from ‘different’ studies to avoid doubling the study weight. The Chi-square test for heterogeneity was used to determine if the results from different studies varied significantly. Additionally, heterogeneity was quantified using the I2 statistic, which gives the percentage of the variability in effect estimates that is due to heterogeneity. An I2 value of over 50% was regarded as indicating substantial heterogeneity.

Subsequently, the results from the different studies were pooled. The measurement scales of some outcomes varied, and thus the standardised mean difference (SMD) between groups was calculated in preference to the raw mean difference. A random-effects model was used for all meta-analyses, regardless of the degree of heterogeneity between studies. The examination of publication bias was investigated using graphical methods (Funnel plots) examining the association between the effect size (SMD) and the uncertainty (standard error) in the calculating effect size.

The diagnostic performance (sensitivity and specificity) of the ASL in differentiating between HGGs and LGGs and in glioma grading were also investigated. The data analysis used an approach outlined by Li et al. [18, 19] that jointly models the two outcomes (sensitivity and specificity), due to the known inverse relationship between these two measures. The approach taken here was to fit a two-level mixed logistic regression model, with independent binomial distributions for the true positives and true negatives conditional on the sensitivity and specificity in each study, and a bivariate normal model for the logit transforms of sensitivity and specificity between studies. This approach gives pooled estimates of sensitivity and specificity, along with corresponding confidence intervals (CI) for each. A hierarchical summary receiver operating characteristic (HROC) curve was also generated. This analysis was performed using the “metandi” command in STATA 15 (StataCorp LLC, College Station, TX, USA).


Eligible ASL studies

The studies are grouped according to the applied ASL acquisition technique - pseudo-continuous-ASL, continuous -ASL and pulsed-ASL (PCASL, CASL, PASL, respectively) - and summarised on Supplementary Tables 1–3. The examined gliomas histology, the ASL technical parameters, the selected TBF metrics, and any statistical significant difference between HGGs and LGGs are also presented on Supplementary Tables 1–3.

QUADAS-2 assessment

Five of the included studies expressed low risk of bias and concerns regarding applicability following the QUADAS-2 assessment. The summary graph of QUADAS-2 assessment is demonstrated in Figure 2. The risk of bias graph shows the four domains; the flow and timing and the reference standard domains possess the lowest risk of bias. The index test domain, could introduce bias as in the majority of the included studies the reviewers were not blinded to the standard reference (high) or the authors did not provide this information. In the patient selection domain, about 25% of the eligible for meta-analysis studies did not use a consecutive or random selection (high) while ~ 25% did not explain the process of patient selection (unclear). The concerns regarding applicability graph involved three domains; the reference standard and the index test domains showed low concerns as all the studies included ASL as index test and histopathological examination as a reference standard. However, the patient selection domain expressed ~10% (high) as one of the studies included both adult and pediatric patients and another one include residual gliomas in their analysis.

The Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) results for the included studies.

Figure 2: The Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) results for the included studies.

Differentiation of HGGs from LGGs

With the exception of TBFmean, there was considerable heterogeneity among the studies (Table 1). The Chi-square test for heterogeneity was statistically significant, and additionally the I2 values were high. All ASL-derived biomarkers were found to be significantly lower in LGGs than in HGGs. The larger effects were observed for the rTBF, where the sizes of group differences were typically higher than for the TBF. The mean differences between LGGs and HGGs for the rTBF was approximately 1.5 SDs. The rTBF funnel plots typically showed values outside the confidence limit in both directions, reflecting the large study heterogeneity. However, there was no clear picture that the effect size was associated with the standard error of the SMD to suggest publication bias (Egger test, P- value = 0.17, 0.72, 0.24 for rTBF, rTBFmean and rTBFmax, respectively). The TBF funnel-plots suggested that the majority of points were within the confidence limits, corroborating the lack of obviouas publication bias (Egger test, P- value = 0.05, 0.43, 0.10 for TBF, TBFmean and TBFmax, respectively). A graphical illustration of the results is shown in the Supplementary Figures 1–6.

Table 1: Comparison of the differences in ASL-related biomarkers between HGGs and LGGs and between different glioma grades


Number of studies

Total sample size


Effect size

Egger test to evaluate publication bias





SMD (95% CI) (*)









–1.46 (–2.00, –0.91)



rTBF mean






–1.53 (–2.26, –0.79)



rTBF max






–1.36 (–2.23, –0.49)









–0.82 (–1.20, –0.45)



TBF mean






–0.61 (–0.99, –0.23)



TBF max






–0.96 (–1.53, –0.39)











–1.39 (–1.89, –0.89)









–0.90 (–1.85, 0.04)











–2.07 (–3.38, –0.76)









–1.44 (–2.76, –0.12)











–1.05 (–1.82, –0.27)









–0.45 (–0.95, 0.05)



(*) SMD calculated as the difference between LGGs ASL-parameters and HGGs ASL-parameters (usually higher than the LGG counterparts).

(-) No pooled results due to low number of included studies.

Differentiation of grade II from grade III gliomas

A summary of the analysis results is presented on Table 1. rTBF values were characterised by relatively small degree of heterogeneity between studies and were found to significantly vary between the two investigated WHO grades, with substantially lower values in grade II patients. The mean rTBF value was 1.4 SDs lower in grade II than in grade III patients. There were only two studies reporting absolute TBF values, and there was a relatively large amount of heterogeneity in them (I2:66%). The absolute TBF showed a trend for lower in grade II patients by an average of 0.9 SDs. Concerning publication bias, all points were within the confidence limits and there was no clear evidence of asymmetry of the funnel plot (Egger test, P- value = 0.46 for rTBF). Graphical illustrations of these results are shown in Supplementary Figures 7 and 8.

Differentiation of grade II from grade IV gliomas

The ASL-derived parameters between grade II and grade IV patients are summarised on Table 1. The results showed significant heterogeneity among the included studies with ASL biomarkers showing significantly lower values in grade II patients. The larger effect was observed for rTBF, which was 2 SDs lower in grade II patients than in grade IV gliomas. The effect size for TBF was substantial (1.4 SDs). The funnel-plot for the rTBF was fairly symmetrical suggesting no definite evidence of publication bias (Egger test, P- value = 0.25 for rTBF). The results are illustrated graphically in Supplementary Figures 9 and 10.

Differentiation of grade III from grade IV gliomas

The differences in ASL-derived tumour perfusion between grade III and grade IV gliomas are summarised on Table 1 and suggest a significant degree of heterogeneity (I2:69%) through the 6 studies that provided rTBF as ASL output. The pooled results suggest significantly lower rTBF values in grade III patients compared to their grade IV counterparts. The SMD between grades was 1 SD, which is slightly lower than the difference between grade II and III gliomas. The results for TBF showed little heterogeneity between studies and a trend for lower TBF in the grade III patients (p = 0.08). The funnel plot appeared fairly asymmetrical for TBF, with obviously larger SMD values being found in the smaller studies (owing to larger standard errors) and smaller effects in the larger patient cohorts (with smaller SE) but not for the rTBF (Egger test, P- value = 0.04, 0.19 for TBF and rTBF, respectively). For both ASL-derived parameters, there was some suggestion of publication bias (see Supplementary Figures 11 and 12).

HROC curve analysis of ASL-based histological grading

The diagnosic performance of ASL for determining the individual glioma grades (Table 2) was characterised by high sensitivity (>90%) for diffrentiation between grade II and III. The sensitivity was slightly lower for the classification between grade III and IV, but still relatively high. This is in agreement with the aforementioned results as the effect size between grade II and grade III gliomas was higher than that between the grade III and grade IV tumours. The method specificity for the individual tumour grading was below 70%. Both the 95% confidence and prediction regions indicate wide variability of the true sensitivity and specificity. The ASL sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) for various cut-off values in glioma grading from the involved studies illustrated on Supplementary Table 4. The HROC plots are showed on Figure 3.

Table 2: Diagnostic performance of the ASL in discrimination between glioma grades

Glioma grading

Number of studies included

Total sample size

Sensitivity (95% CI)

Specificity (95% CI)

AUC (95% CI)

II vs. III




94% (75%, 99%)

61% (48%, 73%)

0.76, (0.72, 0.79)

II vs. IV











86% (75%, 93%)

69% (57%, 79%)

0.75, (0.71, 0.79)

(*) No pooled results due to low number of included studies.

HROC plot of the summary point of the sensitivity and the specificity (square) and its 95% CI (the green curve) of rTBF from ASL to differentiate between.

Figure 3: HROC plot of the summary point of the sensitivity and the specificity (square) and its 95% CI (the green curve) of rTBF from ASL to differentiate between. (A) grade II and grade III (94%, CI (75%, 99%)) and (61%, CI (48%, 73%)), respectively; (B) grade III and grade IV (86%, CI (75%, 93%)) and (69%, CI (57%, 79%)), respectively.

HROC curve analysis of ASL-based differentiation between HGGs and LGGs

The overall results suggested relatively high sensitivity and specificity, both approx. 85%, by using ASL biomarkers to stratify the gliomas in HGGs and LGGs. There was a similar level of sensitivity for each of the individual ASL parameters under analysis. Specificity was slightly more variable ranging from 79% for rTBF mean up to 92% for rTBF max. Both the 95% confidence and prediction regions indicate wide variability of the true sensitivity and specificity. A summary of the results is presented on Table 3. The sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) of various cut-off ASL values among glioma grades from the involved studies are shown in Supplementary Table 5 with the HROC plots presented in Figure 4.

Table 3: Diagnostic performance of the ASL imaging biomarkers in stratifying the tumours between HGGs and LGGs


Number of studies

Total sample size

Sensitivity (95% CI)

Specificity (95% CI)

AUC (95% CI)







86% (78%, 91%)

84% (76%, 90%)

0.91, (0.89, 0.93)












86% (77%, 91%)

84% (76%, 90%)

0.91, (0.89, 0.94)

rTBF max




85% (69%, 94%)

92% (80%, 97%)

0.95, (0.93, 0.97)

rTBF mean




84% (71%, 92%)

79% (66%, 88%)

0.87, (0.84, 0.90)

(*) No pooled results due to low number of included studies.

HROC plot shows the summary point of the sensitivity and the specificity (square) and its 95% CI (the green curve).

Figure 4: HROC plot shows the summary point of the sensitivity and the specificity (square) and its 95% CI (the green curve). (A) from all of the analysed ASL parameters to differentiate between HGGs and LGGs (86%, CI (78%, 91%)) and (84%, CI (76%, 90%)), respectively; (B) rTBF to differentiate between HGGs and LGGs (86%, CI (77%, 91%)) and (84%, CI (76%, 90%)), respectively; (C) rTBFmean to differentiate between HGGs and LGGs (84%, CI (71%, 92%)) and (79%, CI (66%, 88%), respectively; (D) rTBFmax to differentiate between HGGs and LGGs (85%, CI (69%, 94%)) and (92%, CI (80%, 97%)).


The results of this meta-analysis indicate that the absolute TBF can be used to differentiate HGGs from LGGs as well as grade II from grade IV gliomas. However, absolute TBF could not discriminate grade II from grade III and grade III from grade IV glial tumours. In contrast, rTBF was more effective than absolute TBF in differentiating HGGs from LGGs and presented satisfactory accuracy in glioma grading. In addition, rTBFmax parameter demonstrated the best performance in glioma grading overall. The same result was observed in the sensitivity and specificity analysis where the rTBFmax provided the highest sensitivity and specificity values. The estimated effect size for rTBF was approximately similar between HGGs and LGGs (-1.46 (-2.00, -0.91), p- value < 0.001), and between grade II and grade III (-1.39 (-1.89, -0.89), p- value < 0.001), while it expressed smaller effect size between grade III and grade IV (-1.05 (-1.82, -0.27)), p < 0.05). The same result was also noticed in the sensitivity and specificity analyses. Fudaba et al. also reported that rTBFmax provided higher sensitivity and specificity than rTBFmean [20].

The systematic literature review and the heterogeneity analysis highlighted the variations across the included studies, which could be due to inclusion of mixed glioma types, the applied ASL approaches and its parameters (e.g post-labelling-delays (PLD)), and the method of image processing and analysis. Notably, all studies that reported the inability of ASL to distinguish HGGs from LGGs [911] were conducted in mixed gliomas patient cohorts. This sounds rational as even low-grade oligodendrogliomas are associated with elevated perfusion and thus introduce diagnostic bias [7, 21]. A study that included only oligodendrogliomas found also impossible to determine the WHO grade using ASL biomarkers [22]. On contrary, a handful of studies in astrocytomas demonstrated that ASL can identify HGGs from LGGs [2325] and grade them accordingly with remarkable sensitivity and specificifity [8, 20]. In spite of that, other studies with mixed glioma types reported the efficiency of ASL in differentiating between HGGs and LGGs [6, 7] and glioma grading [5]. The 2016 WHO classification of brain tumours puts more emphasis on the genetic and molecular subtyping of gliomas by stratifying them according to the isocitrate-dehydrogenase-(IDH) and 1p/19q mutation status or co-deletion, respectively [26]. The ASL studies reviewed in this meta-analysis lacked information about molecular and genetic subtypes of gliomas, which is likely to represent a source of variation as the IDH-wild gliomas has been reported to have higher perfusion values than those with IDH-mutation [9, 27].

The different ASL labelling approaches and acquisition parameters across the included studies obviously result in quantitative ASL metrics variation. The three common ASL labelling methods used in these studies were the pulsed-ASL (PASL); the continuous-ASL (CASL); and the pseudo-continuous-ASL (PCASL). PASL is the most widely used technique due to its broad availability, the low specific absorption rate (SAR), and the robust labelling efficiency [28] across a wide range of blood velocity. However, this method suffers from the lowest signal-to-noise-ratio (SNR) in comparison to the other labelling methods. CASL is subject to higher SAR and lower labelling efficiency than PASL. PCASL has the advantages of both CASL (high SNR) and PASL (low SAR) however, its labelling efficiency is lower than that of PASL. Thus, PCASL has been reported to demonstrate the best reproducibility among all aforementioned ASL labelling methods, at least in healthy volunteers [29].

All but two studies with negative results [9, 30] that applied PASL in gliomas showed its efficiency in differentiating between HGGs and LGGs [7, 20, 2325, 31, 32] and in glioma grading [8]. All studies that utilised CASL in gliomas [21, 22, 30] were promising in differentiation between HGGs and LGGs. Nonetheless, recent studies drew conflicting conclusions regarding the usefulness of PCASL acquisitions as it helped to identify HGGs from LGGs [6] and to perform grading [5], whilst a couple of publications didn’t confirm the findings [911]. This discrepancy could be due to the low labelling efficiency of PCASL, which varies between scanners and patients. Another important acquisition parameter in ASL is the inversion time (TI), also called post labelling delay (PLD). Selection of the most suitable TI in tumours tends to be challenging as ASL acquisition at a single TI reduces the sensitivity to the blood transit time but does not eliminate it. This sensitivity can be considered by the use of multiple PLDs [33] in CASL/PCASL or by quantitative-image of perfusion using a single-subtraction-(QUIPSS-II) [34]/ QUIPSS-II with thin slice TI1 periodic saturation (Q2TIPS) [35] in PASL. Generally, the delay time has to be long enough to enable all the labelled blood bolus to transfer from the capillary bed to the target tissue in the labelling plane, but short enough to preserve the signals from T1-decay. Furtner et al. identified the most effective TI for determining the HGGs from the LGGs astrocytomas at 370 ms [23]. Other studies employed variable single delay times within a range of 1200–1900 ms, and reported similar or lower sensitivity and specificity [6, 8, 20, 24, 30, 31]. Unlike using a single TI, the use of multiple TIs makes CBF quantification more reliable and less sensitive to bolus arrival time. Cebeci et al. used 8 TIs to distinguish HGG from LGG [7] and Yang et al. acquired ASL at 16 TIs in astrocytomas, which not only differentiated HGGs from LGGs, but also enabled glioma grading [8].

The reported tumour perfusion was heavily skewed by the non-standardised ROI analysis on the calculated CBF maps. Normalised TBF values (also known as rTBF) have been widely used as they reduce the values scatterring within a group by mitigating age and hemodynamic variations related risks [25]. Also, patients with brain tumours usually present elevated intracranial pressure, which in turn reduces the global CBF and the accuracy of the estimation of regional TBF. On the other hand, several authors argue that the use of rTBF leads to variation between observers due to random error caused by the internal reference on the normal tissue [36, 37]. This could partially explain the high variation in the rTBF of malignant tumours. Nevertheless, the relative values are considered more reliable than the absolute TBF values when distinguishing between HGGs and LGGs [7, 25] which has also been confirmed by our meta-analysis. Previous ASL tumour studies have used various normal brain regions for normalisation, including the GM [38] the WM [31], the mean of both [32], or the contralateral normal tissue mirrored to tumour [25]. Interestingly, the mirror ROI yielded better results than GM and WM as an internal standard [39], which is most likely due to the approximately similar distance from the labelling plane. Nevertheless, the majority of studies have used the contralateral normal appearing white matter for normalisation purposes [5, 8, 20, 22, 24, 31, 40]. Furthermore, it is important to note that the image employed to delineate the tumour mask may affect the reliability and validity of results. A number of studies generated tumour mask from the MRI-conventional image whilst others selected the ROI through visual inspection of the maximum signal intensity visually on the ASL-subtracted image or the M0 image. Regarding the choice of contrast-enhanced T1-weighted images as reference for tumour mask, we should bear in mind that the enhanced tissue represents blood brain barrier disruption (BBB) rather than increased perfusion [41].

In line with our meta-analysis results, TBFmax has been reported to be more precise than TBFmean for tumour characterisation [42, 43]. This is plausible, as HGGs tend to be heterogeneous, and hence the TBFmax will be representative of the most anaplastic tumour part. In addition, TBFmean estimation is affected by the partial volume effect making the TBFmax more suitable as a biomarker. Nevertheless, histogram analysis [44, 45] that also captures the tumour heterogeneity is probably the method with the highest diagnostic accuracy and reproducibility.

Several studies have examined ASL feasibility and its complementary role in routine brain tumour examinations by comparing it with more well-established MRI methods; to other MRI-perfusion methods, dynamic susceptibility contrast-enhanced (DSC) [7, 25, 43] and dynamic contrast-enhanced (DCE) [9, 10] to MRI-diffusion methods (MRI diffusion) [5, 6, 31] and to MR-spectroscopy (MRS) [20, 22, 32]. Currently, dynamic susceptibility contrast (DSC) is the clinically most utilised MR perfusion technique in brain tumour examinations. However, arterial spin labelling (ASL), is useful for those who could not tolerate high-rate contrast injection or (relatively) contraindicated to the usage of contrast agent (impaired renal function, allergies, paediatric population) along with raising issues over the permanent gadolinium depositions in brain [46]. The studies that involved both indexed examinations in glioma grading, focused on the use of ASL as an alternative or surrogate of DSC [7, 25, 37] e.g. by examining the correlation or interchangeability of the estimated perfusion metrics from each examination, rather than suggesting which method is superior. As a matter of fact, these studies demonstrate the non-inferiority of ASL in the specific patient cohorts. Warmuth et al. reported a strong positive correlation between the rTBF measurement from ASL and DSC [25]. Another study measured rTBFmax and rTBFmean reporting excellent correlation between ASL and DSC [37]. Cebeci et al. used PASL to demonstrate moderate but still significant correlation between the rTBF values from the ASL and DSC [7]. These studies provide preliminary evidence that ASL can be used as a non-invasive alternative to DSC addressing the shortcoming of non-diffusible tracer (gadolinium) in DSC that leaks out of dysfunctional BBB and may lead to underestimation of the rTBF measurement [47]. In addition, there is scarce evidence on the prognostic role of each perfusion techniques [4853], where rCBV measurements seem to provide the best sensitivity and specificity to predict tumor recurrence and survival time in gliomas patients [54].

Two of the included studies in this meta-analysis examined the correlation between ASL and DCE in gliomas [9, 10]. Both studies (utilising PASL and PCASL) suggested that ASL was not an effective method for glioma grading and reported poor to moderate correlation between ASL and DCE. This might be attributed to a variation in ROI selection and number of high-grade gliomas in each study. In order to consider ASL as a viable alternative to other MRI perfusion methods (DSC and DCE), its reproducibility and inter-observer variability in patients with brain tumours have to be tested. A number of studies have reported good inter-observer variability in tumour patients using PASL [8, 24] quantitative STAR labeling of arterial regions (QUASAR) [42] and PCASL [5, 11]. Hirai et al. reported excellent reproducibility in glioma patients using both maximum and mean TBF [42].

Regarding the future directions of ASL imaging in tumours, Yoo et al. recenty used PCASL to investigate the connection between the HGGs perfusion values and genetic biomarkers [55] and found the epidermal growth factor receptor (EGFR) to be significantly correlated with rTBF and absolute TBF. Furthermore, Yamashita et al. demonstrated that TBF and rTBF values were significantly greater in GBM patients with IDH-wild type status than those with IDH-mutation [27] and Brendle et al. reported that ASL, unlike DCE, could stratify astrocytomas accortding to the IDH-mutation status [9].

There are some limitations in our study. First, approximately all the evaluated perfusion metrics founde to be heterogenous among the included studies in this meta-analysis. This is expected, because heterogeneity among MRI measurments are unavoidable as they differ from center to center and even between platformas in the same center. However, this heterogeneity has been taken into account in the analysis stage via random effect model utilization. Second, the small sample size of the included studies (18 studies). Still, this reflect the strict methodological standard in order to be faithfull to the assigned research question.


This meta-analysis aimed to shed light into the diagnostic performance of ASL in glioma grading and demonstrated the suitability of ASL-derived perfusion metrics in glioma grading. rTBFmax showed the best diagnostic and staging performance. Hence, ASL metrics capacity as imaging biomarkers can be routinely useful for the characterisation and staging of gliomas at baseline, with possible implications for treatment selection and surveillance imaging. However, further research with larger numbers of patients and well defined tumour subtypes, including molecular information, is needed to refine any TBF-relate threshold values that allow higher diagnostic and prognostic accuracy and are essential for the wide dissmination of the technique.


The views expressed in this article are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.


This research was supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. SB is funded by the National Institute for Health Research (NIHR) grant number BRC345/NS/SB/101410. JPG and ARG's research is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Bart’s Health NHS Trust (NIHR CLAHRC North Thames).


1. Dean BL, Drayer BP, Bird CR, Flom RA, Hodak JA, Coons SW, Carey RG. Gliomas: classification with MR imaging. Radiology. 1990; 174:411–15. https://doi.org/10.1148/radiology.174.2.2153310.

2. Felix R, Schörner W, Laniado M, Niendorf HP, Claussen C, Fiegler W, Speck U. Brain tumors: MR imaging with gadolinium-DTPA. Radiology. 1985; 156:681–88. https://doi.org/10.1148/radiology.156.3.4040643.

3. Runge VM, Clanton JA, Price AC, Wehr CJ, Herzer WA, Partain CL, James AE Jr. The use of Gd DTPA as a perfusion agent and marker of blood-brain barrier disruption. Magn Reson Imaging. 1985; 3:43–55. https://doi.org/10.1016/0730-725X(85)90008-6.

4. Noguchi T, Yoshiura T, Hiwatashi A, Togao O, Yamashita K, Nagao E, Shono T, Mizoguchi M, Nagata S, Sasaki T, Suzuki SO, Iwaki T, Kobayashi K, et al. Perfusion imaging of brain tumors using arterial spin-labeling: correlation with histopathologic vascular density. AJNR Am J Neuroradiol. 2008; 29:688–93. https://doi.org/10.3174/ajnr.A0903.

5. Shen N, Zhao L, Jiang J, Jiang R, Su C, Zhang S, Tang X, Zhu W. Intravoxel incoherent motion diffusion-weighted imaging analysis of diffusion and microperfusion in grading gliomas and comparison with arterial spin labeling for evaluation of tumor perfusion. J Magn Reson Imaging. 2016; 44:620–32. https://doi.org/10.1002/jmri.25191.

6. Lin Y, Li J, Zhang Z, Xu Q, Zhou Z, Zhang Z, Zhang Y, Zhang Z. Comparison of intravoxel incoherent motion diffusion-weighted MR imaging and arterial spin labeling MR imaging in gliomas. Biomed Res Int. 2015; 2015:234245. https://doi.org/10.1155/2015/234245.

7. Cebeci H, Aydin O, Ozturk-Isik E, Gumus C, Inecikli F, Bekar A, Kocaeli H, Hakyemez B. Assesment of perfusion in glial tumors with arterial spin labeling; comparison with dynamic susceptibility contrast method. Eur J Radiol. 2014; 83:1914–19. https://doi.org/10.1016/j.ejrad.2014.07.002.

8. Yang S, Zhao B, Wang G, Xiang J, Xu S, Liu Y, Zhao P, Pfeuffer J, Qian T. Improving the grading accuracy of astrocytic neoplasms noninvasively by combining timing information with cerebral blood flow: A multi-Ti arterial spin-labeling MR imaging study. AJNR Am J Neuroradiol. 2016; 37:2209–2216. https://doi.org/10.3174/ajnr.A4907.

9. Brendle C, Hempel JM, Schittenhelm J, Skardelly M, Tabatabai G, Bender B, Ernemann U, Klose U. Glioma Grading and Determination of IDH Mutation Status and ATRX loss by DCE and ASL Perfusion. Clin Neuroradiol. 2018; 28:421–428. https://doi.org/10.1007/s00062-017-0590-z.

10. Roy B, Awasthi R, Bindal A, Sahoo P, Kumar R, Behari S, Ojha BK, Husain N, Pandey CM, Rathore RK, Gupta RK. Comparative evaluation of 3-dimensional pseudocontinuous arterial spin labeling with dynamic contrast-enhanced perfusion magnetic resonance imaging in grading of human glioma. J Comput Assist Tomogr. 2013; 37:321–26. https://doi.org/10.1097/RCT.0b013e318282d7e2.

11. Bai Y, Lin Y, Zhang W, Kong L, Wang L, Zuo P, Vallines I, Schmitt B, Tian J, Song X, Zhou J, Wang M. Noninvasive amide proton transfer magnetic resonance imaging in evaluating the grading and cellularity of gliomas. Oncotarget. 2017; 8:5834–42. https://doi.org/10.18632/oncotarget.13970.

12. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009; 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097.

13. Sun Y, Schmidt NO, Schmidt K, Doshi S, Rubin JB, Mulkern RV, Carroll R, Ziu M, Erkmen K, Poussaint TY, Black P, Albert M, Burstein D, Kieran MW. Perfusion MRI of U87 brain tumors in a mouse model. Magn Reson Med. 2004; 51:893–99. https://doi.org/10.1002/mrm.20029.

14. Wong ET, Timmons J, Callahan A, O’Loughlin L, Giarusso B, Alsop DC. Phase I study of low-dose metronomic temozolomide for recurrent malignant gliomas. BMC Cancer. 2016; 16:914. https://doi.org/10.1186/s12885-016-2945-2.

15. Lyu Y, Liu S, You H, Hou B, Wang Y, Ma W, Feng F. Evaluation of recurrent high-grade gliomas treated with bevacizumab: A preliminary report of 3D pseudocontinuous artery spin labeling. J Magn Reson Imaging. 2017; 46:565–573. https://doi.org/10.1002/jmri.25558.

16. Morana G, Piccardo A, Tortora D, Puntoni M, Severino M, Nozza P, Ravegnani M, Consales A, Mascelli S, Raso A, Cabria M, Verrico A, Milanaccio C, Rossi A. Grading and outcome prediction of pediatric diffuse astrocytic tumors with diffusion and arterial spin labeling perfusion MRI in comparison with 18F-DOPA PET. Eur J Nucl Med Mol Imaging. 2017; 44:2084–2093. https://doi.org/10.1007/s00259-017-3777-2.

17. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011; 155:529–36. https://doi.org/10.7326/0003-4819-155-8-201110180-00009.

18. Li PM, Linos E, Gurgel RK, Fischbein NJ, Blevins NH. Evaluating the utility of non-echo-planar diffusion-weighted imaging in the preoperative evaluation of cholesteatoma: a meta-analysis. Laryngoscope. 2013; 123:1247–50. https://doi.org/10.1002/lary.23759.

19. Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001; 20:2865–84. https://doi.org/10.1002/sim.942.

20. Fudaba H, Shimomura T, Abe T, Matsuta H, Momii Y, Sugita K, Ooba H, Kamida T, Hikawa T, Fujiki M. Comparison of multiple parameters obtained on 3T pulsed arterial spin-labeling, diffusion tensor imaging, and MRS and the Ki-67 labeling index in evaluating glioma grading. AJNR Am J Neuroradiol. 2014; 35:2091–8. https://doi.org/10.3174/ajnr.A4018.

21. Wolf RL, Wang J, Wang S, Melhem ER, O’Rourke DM, Judy KD, Detre JA. Grading of CNS neoplasms using continuous arterial spin labeled perfusion MR imaging at 3 Tesla. J Magn Reson Imaging. 2005; 22:475–82. https://doi.org/10.1002/jmri.20415.

22. Chawla S, Wang S, Wolf RL, Woo JH, Wang J, O’Rourke DM, Judy KD, Grady MS, Melhem ER, Poptani H. Arterial spin-labeling and MR spectroscopy in the differentiation of gliomas. AJNR Am J Neuroradiol. 2007; 28:1683–89. https://doi.org/10.3174/ajnr.A0673.

23. Furtner J, Schöpf V, Schewzow K, Kasprian G, Weber M, Woitek R, Asenbaum U, Preusser M, Marosi C, Hainfellner JA, Widhalm G, Wolfsberger S, Prayer D. Arterial spin-labeling assessment of normalized vascular intratumoral signal intensity as a predictor of histologic grade of astrocytic neoplasms. AJNR Am J Neuroradiol. 2014; 35:482–89. https://doi.org/10.3174/ajnr.A3705.

24. Kim MJ, Kim HS, Kim JH, Cho KG, Kim SY. Diagnostic accuracy and interobserver variability of pulsed arterial spin labeling for glioma grading. Acta Radiol. 2008; 49:450–7. https://doi.org/10.1080/02841850701881820.

25. Warmuth C, Günther M, Zimmer C. Quantification of blood flow in brain tumors: comparison of arterial spin labeling and dynamic susceptibility-weighted contrast-enhanced MR imaging. Radiology. 2003; 228:523–32. https://doi.org/10.1148/radiol.2282020409.

26. Louis DN, Perry A, Reifenberger G, von Deimling A, Figarella-Branger D, Cavenee WK, Ohgaki H, Wiestler OD, Kleihues P, Ellison DW. The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary. Acta Neuropathol. 2016; 131:803–20. https://doi.org/10.1007/s00401-016-1545-1.

27. Yamashita K, Hiwatashi A, Togao O, Kikuchi K, Hatae R, Yoshimoto K, Mizoguchi M, Suzuki SO, Yoshiura T, Honda H. MR Imaging-Based Analysis of Glioblastoma Multiforme: Estimation of IDH1 Mutation Status. AJNR Am J Neuroradiol. 2016; 37:58–65. https://doi.org/10.3174/ajnr.A4491.

28. Wong EC, Buxton RB, Frank LR. A theoretical and experimental comparison of continuous and pulsed arterial spin labeling techniques for quantitative perfusion imaging. Magn Reson Med. 1998; 40:348–55. https://doi.org/10.1002/mrm.1910400303.

29. Chen Y, Wang DJ, Detre JA. Test-retest reliability of arterial spin labeling with common labeling strategies. J Magn Reson Imaging. 2011; 33:940–49. https://doi.org/10.1002/jmri.22345.

30. Canale S, Rodrigo S, Tourdias T, Mellerio C, Perrin M, Souillard R, Oppenheim C, Meder JF. 32. [Grading of adults primitive glial neoplasms using arterial spin-labeled perfusion MR imaging]. [Article in French]. J Neuroradiol. 2011; 38:207–13. https://doi.org/10.1016/j.neurad.2010.12.003.

31. Kim HS, Kim SY. A prospective study on the added value of pulsed arterial spin-labeling and apparent diffusion coefficients in the grading of gliomas. AJNR Am J Neuroradiol. 2007; 28:1693–99. https://doi.org/10.3174/ajnr.A0674.

32. Weber MA, Zoubaa S, Schlieter M, Jüttler E, Huttner HB, Geletneky K, Ittrich C, Lichy MP, Kroll A, Debus J, Giesel FL, Hartmann M, Essig M. Diagnostic performance of spectroscopic and perfusion MRI for distinction of brain tumors. Neurology. 2006; 66:1899–906. https://doi.org/10.1212/01.wnl.0000219767.49705.9c.

33. Alsop DC, Detre JA. Reduced transit-time sensitivity in noninvasive magnetic resonance imaging of human cerebral blood flow. J Cereb Blood Flow Metab. 1996; 16:1236–49. https://doi.org/10.1097/00004647-199611000-00019.

34. Wong EC, Buxton RB, Frank LR. Quantitative imaging of perfusion using a single subtraction (QUIPSS and QUIPSS II). Magn Reson Med. 1998; 39:702–08. https://doi.org/10.1002/mrm.1910390506.

35. Luh WM, Wong EC, Bandettini PA, Hyde JS. QUIPSS II with thin-slice TI1 periodic saturation: a method for improving accuracy of quantitative perfusion imaging using pulsed arterial spin labeling. Magn Reson Med. 1999; 41:1246–54. https://doi.org/10.1002/(SICI)1522-2594(199906)41:6<1246::AID-MRM22>3.0.CO;2-N.

36. Wetzel SG, Cha S, Johnson G, Lee P, Law M, Kasow DL, Pierce SD, Xue X. Relative cerebral blood volume measurements in intracranial mass lesions: interobserver and intraobserver reproducibility study. Radiology. 2002; 224:797–803. https://doi.org/10.1148/radiol.2243011014.

37. Abrigo JM, Fountain DM, Provenzale JM, Law EK, Kwong JS, Hart MG, Tam WWS. Magnetic resonance perfusion for differentiating low-grade from high-grade gliomas at first presentation. Cochrane Database Syst Rev. 2018; 1:CD011551. https://doi.org/10.1002/14651858.CD011551.pub2.

38. Nabavizadeh SA, Assadsangabi R, Hajmomenian M, Santi M, Vossough A. High accuracy of arterial spin labeling perfusion imaging in differentiation of pilomyxoid from pilocytic astrocytoma. Neuroradiology. 2015; 57:527–33. https://doi.org/10.1007/s00234-015-1497-5.

39. Jiang J, Zhao L, Zhang Y, Zhang S, Yao Y, Qin Y, Wang CY, Zhu W. Comparative analysis of arterial spin labeling and dynamic susceptibility contrast perfusion imaging for quantitative perfusion measurements of brain tumors. Int J Clin Exp Pathol. 2014; 7:2790–99.

40. Gao F, Guo R, Hu XJ, Li CJ, Li M. Noninvasive Tumor Grading of Glioblastomas Before Surgery Using Arterial Spin Labeling. A Cohort Study. Anal Quant Cytopathol Histpathol. 2015; 37:339–46.

41. White CM, Pope WB, Zaw T, Qiao J, Naeini KM, Lai A, Nghiemphu PL, Wang JJ, Cloughesy TF, Ellingson BM. Regional and voxel-wise comparisons of blood flow measurements between dynamic susceptibility contrast magnetic resonance imaging (DSC-MRI) and arterial spin labeling (ASL) in brain tumors. J Neuroimaging. 2014; 24:23–30. https://doi.org/10.1111/j.1552-6569.2012.00703.x.

42. Hirai T, Kitajima M, Nakamura H, Okuda T, Sasao A, Shigematsu Y, Utsunomiya D, Oda S, Uetani H, Morioka M, Yamashita Y. Quantitative blood flow measurements in gliomas using arterial spin-labeling at 3T: intermodality agreement and inter- and intraobserver reproducibility study. AJNR Am J Neuroradiol. 2011; 32:2073–79. https://doi.org/10.3174/ajnr.A2725.

43. van Westen D, Petersen ET, Wirestam R, Siemund R, Bloch KM, Ståhlberg F, Björkman-Burtscher IM, Knutsson L. Correlation between arterial blood volume obtained by arterial spin labelling and cerebral blood volume in intracranial tumours. MAGMA. 2011; 24:211–23. https://doi.org/10.1007/s10334-011-0255-x.

44. Just N. Improving tumour heterogeneity MRI assessment with histograms. Br J Cancer. 2014; 111:2205–13. https://doi.org/10.1038/bjc.2014.512.

45. Lee S, Choi SH, Ryoo I, Yoon TJ, Kim TM, Lee SH, Park CK, Kim JH, Sohn CH, Park SH, Kim IH. Evaluation of the microenvironmental heterogeneity in high-grade gliomas with IDH1/2 gene mutation using histogram analysis of diffusion-weighted imaging and dynamic-susceptibility contrast perfusion imaging. J Neurooncol. 2015; 121:141–50. https://doi.org/10.1007/s11060-014-1614-z.

46. Guo BJ, Yang ZL, Zhang LJ. Gadolinium Deposition in Brain: Current Scientific Evidence and Future Perspectives. Front Mol Neurosci. 2018; 11:335. https://doi.org/10.3389/fnmol.2018.00335.

47. Paulson ES, Schmainda KM. Comparison of Dynamic Susceptibility-weighted Contrast-enhanced MR Methods: Recommendations for Measuring Relative Cerebral Blood Volume in Brain Tumors. Radiology. 2008; 249:601–13. https://doi.org/10.1148/radiol.2492071659.

48. Khashbat D, Abe T, Ganbold M, Iwamoto S, Uyama N, Irahara S, Otomi Y, Harada M, Kageji T, Nagahiro S. Correlation of 3D Arterial Spin Labeling and Multi-Parametric Dynamic Susceptibility Contrast Perfusion MRI in Brain Tumors. J Med Invest. 2016; 63:175–81. https://doi.org/10.2152/jmi.63.175.

49. Vidyasagar R, Abernethy L, Pizer B, Avula S, Parkes LM. Quantitative measurement of blood flow in paediatric brain tumours-a comparative study of dynamic susceptibility contrast and multi time-point arterial spin labelled MRI. Br J Radiol. 2016; 89:20150624. https://doi.org/10.1259/bjr.20150624.

50. Soni N, Dhanota DP, Kumar S, Jaiswal AK, Srivastava AK. Perfusion MR imaging of enhancing brain tumors: comparison of arterial spin labeling technique with dynamic susceptibility contrast technique. Neurol India. 2017; 65:1046–52. https://doi.org/10.4103/neuroindia.NI_871_16.

51. Morana G, Tortora D, Staglianò S, Nozza P, Mascelli S, Severino M, Piatelli G, Consales A, Lequin M, Garrè ML, Rossi A. Pediatric astrocytic tumor grading: comparison between arterial spin labeling and dynamic susceptibility contrast MRI perfusion. Neuroradiology. 2018; 60:437–46. https://doi.org/10.1007/s00234-018-1992-6.

52. Lindner T, Ahmeti H, Juhasz J, Helle M, Jansen O, Synowitz M, Ulmer S. A comparison of arterial spin labeling and dynamic susceptibility perfusion imaging for resection control in glioblastoma surgery. Oncotarget. 2018; 9:18570–18577. https://doi.org/10.18632/oncotarget.24970.

53. Wang YL, Chen S, Xiao HF, Li Y, Wang Y, Liu G, Lou X, Ma L. Differentiation between radiation-induced brain injury and glioma recurrence using 3D pCASL and dynamic susceptibility contrast-enhanced perfusion-weighted imaging. Radiother Oncol. 2018; 129:68–74. https://doi.org/10.1016/j.radonc.2018.01.009.

54. Rau MK, Braun C, Skardelly M, Schittenhelm J, Paulsen F, Bender B, Ernemann U, Bisdas S. Prognostic value of blood flow estimated by arterial spin labeling and dynamic susceptibility contrast-enhanced MR imaging in high-grade gliomas. J Neurooncol. 2014; 120:557–66. https://doi.org/10.1007/s11060-014-1586-z.

55. Yoo RE, Choi SH, Cho HR, Kim TM, Lee SH, Park CK, Park SH, Kim IH, Yun TJ, Kim JH, Sohn CH, Han MH, Chang KH. Tumor blood flow from arterial spin labeling perfusion MRI: a key parameter in distinguishing high-grade gliomas from primary cerebral lymphomas, and in predicting genetic biomarkers in high-grade gliomas. J Magn Reson Imaging. 2013; 38:852–60. https://doi.org/10.1002/jmri.24026.

Creative Commons License All site content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License.
PII: 26674