Automated vs manual delineations of regions of interest- a comparison in commercially available perfusion MRI software
© Galinovic et al.; licensee BioMed Central Ltd. 2012
Received: 24 August 2011
Accepted: 21 May 2012
Published: 18 July 2012
In perfusion magnetic resonance imaging a manual approach to delineation of regions of interest is, due to rater bias and time intensive operator input, clinically less favorable than an automated approach would be. The goal of our study was to compare the performances of these approaches.
Using Stroketool, PMA and Perfscape/Neuroscape perfusion maps of cerebral blood flow, mean transit time and Tmax were created for 145 patients with acute ischemic stroke. Volumes of hypoperfused tissue were calculated using both a manual and an automated protocol, and the results compared between methods.
The median difference between the automatically and manually derived volumes was up to 210 ml in Perfscape/Neuroscape, 123 ml in PMA and 135 ml in Stroketool. Correlation coefficients between perfusion volumes and radiological and clinical outcome were much lower for the automatic volumes than for the manually derived ones.
The agreement of the two methods was very poor, with the automated use producing falsely exaggerated volumes of hypoperfused tissue. Software improvements are necessary to enable highly automated protocols to credibly assess perfusion deficits.
KeywordsMagnetic resonance imaging Perfusion MRI Acute ischemic stroke
The increased use of the perfusion imaging (PI) – diffusion-weighted imaging (DWI) mismatch hypothesis  in studies of acute ischemic stroke (AIS) and clinical practice [1, 2] is raising demands from software packages developed for volumetric calculations of hypoperfusion. Typically these programs offer the possibility to delineate a region of interest (ROI) manually but also, to a varying extent, automatically. Manual approaches to delineation are biased and require time intensive operator input. Therefore a mostly automated procedure, if accurate, would be preferred in clinical practice. The goal of our study was comparing the automated approach to the manual approach while using a number of different software packages.
The study design and cohort characteristics have been previously reported . Briefly, using Stroketool (Digital Image Solutions, Germany, http://www.digitalimagesolutions.de), PMA (v184.108.40.206, ASIST, Japan, http://asist.umin.jp/index-e.htm) and Perfscape/Neuroscape (Olea Medical SAS, France, http://www.olea-medical.com) perfusion maps of cerebral blood flow (CBF), mean transit time (MTT) and Tmax were calculated for 145 patients imaged within 24 hours of AIS. The inclusion criteria were: clinically and radiologically confirmed AIS, hypoperfusion on the initial PI examination (assessed by the attending neuroradiologist) and availability of a follow-up MRI scan. For each parameter map, three thresholds were applied. The Tmax thresholds were 4, 6 and 8 seconds of delay  and the MTT thresholds were 5, 6 and 8 seconds. As no uniform CBF scale was available, the three CBF thresholds were different across software. Both MTT and CBF thresholds were chosen empirically using a random sample of acute stroke patients. ROI volumes were calculated using a manual and an automated protocol. In the automated protocol, once the thresholds have been applied, no further post-processing was done. Maps created in PMA and Stroketool also underwent a second post-processing step in SPM8 (Wellcome Trust Centre for Neuroimaging, UK) to cut away scalp and spaces filled with cerebrospinal fluid (CSF). This was not necessary for Perfscape/Neuroscape due to the program’s implemented filtering. In the manual protocol a human rater excluded, from the thresholded maps, areas unlikely to reflect credible hypoperfusion.
Radiological outcome was defined as the final lesion volume on follow-up FLAIR images. Clinical outcome was defined as the National Institute of Health Stroke Scale (NIHSS) score at the time of hospital discharge. All statistics were done in PASW Statistics 18. Analyses of correlations were performed using the Spearman signed-ranks correlation test.
Difference of ROI volumes between the automated and the manual protocol
PMA with filtering *
Stroketool with filtering *
Tmax 4 s
Tmax 6 s
Tmax 8 s
2.6 - 28.1
13.5 – 55.4
0.4 – 15.8
16.1 – 50.1
0.2 – 6.5
MTT 5 s
MTT 6 s
MTT 8 s
CBF highest threshold
CBF medium threshold
CBF lowest threshold
The median volume of the perfusion deficit varied greatly based on the map and threshold and even across software for the same map and threshold. This discrepancy could in part be explained through the different choice of AIFs, the use of different deconvolution techniques, different implementations of the same calculation algorithm as well as differences in motion correction across different programs [5–7]. Our group had already conducted a study on a group of patients with no ischemia, using the same three software packages, and observed the presence of numerous artifacts; typically the cortex proximal to the skull and infratentorial cerebral and cerebellar tissue . Programs without implemented CSF filtering also suffered from artifacts such as ventricles, eyeballs and scalp . As expected, the same artifacts were present in our current patient cohort. This explains the weaker correlation coefficients and overshoot of the automated delineations as compared to the manual ones. Although most of the values on the Bland-Altman difference plot fall within the limits of agreement (Figure 1), these are much too broad with regards to the cohort’s median lesion volumes and the fact that, based on location, even a lesion of a few ml can be clinically significant. Additional CSF filtering greatly reduced the differences between the automatic and the manual volumes, pointing to a need for implemented filtering.
In conclusion, current automated use of the here evaluated programs would lead to falsely exaggerated volumes of hypoperfused tissue in patients with AIS. However a number of improvements, such as algorithms for judging perfusion asymmetry between hemispheres and allowing selection of the vessel territory of expected hypoperfusion, could aid automated protocols in credibly assessing perfusion deficits.
Sources of funding
The research leading to these results has received funding from the Federal Ministry of Education and Research via the grant Center for Stroke Research Berlin (01 EO 0801).
- Hacke W, Albers G, Al-Rawi Y, Bogousslavsky J, Davalos A, Eliasziw M, et al: The Desmoteplase in Acute Ischemic Stroke Trial (DIAS): a phase II MRI-based 9-hour window acute stroke thrombolysis trial with intravenous desmoteplase. Stroke. 2005, 36: 66-73. 10.1161/01.STR.0000149938.08731.2c.View ArticlePubMedGoogle Scholar
- Kohrmann M, Juttler E, Fiebach JB, Huttner HB, Siebert S, Schwark C, et al: MRI versus CT-based thrombolysis treatment within and beyond the 3 h time window after stroke onset: a cohort study. Lancet Neurol. 2006, 5: 661-667. 10.1016/S1474-4422(06)70499-9.View ArticlePubMedGoogle Scholar
- Galinovic I, Ostwaldt AC, Soemmer C, Bros H, Hotter B, Brunecker P, et al: Search for a map and threshold in perfusion MRI to accurately predict tissue fate: a protocol for assessing lesion growth in patients with a persistent vessel occlusion. Cerebrovasc Dis. 2011, 32: 186-193. 10.1159/000328663.View ArticlePubMedGoogle Scholar
- Olivot JM, Mlynash M, Thijs VN, Kemp S, Lansberg MG, Wechsler L, et al: Optimal Tmax threshold for predicting penumbral tissue in acute stroke. Stroke. 2009, 40: 469-475. 10.1161/STROKEAHA.108.526954.View ArticlePubMedGoogle Scholar
- Kosior RK, Kosior JC, Frayne R: Improved dynamic susceptibility contrast (DSC)-MR perfusion estimates by motion correction. J Magn Reson Imaging. 2007, 26: 1167-72. 10.1002/jmri.21128.View ArticlePubMedGoogle Scholar
- Kim J, Leirab EC, Callison RC, Ludwiga B, Moritani T, Magnotta VA, Madsen MT: Toward fully automated processing of dynamic susceptibility contrast perfusion MRI for acute ischemic cerebral stroke. Comput Methods Programs Biomed. 2010, 98: 204-213. 10.1016/j.cmpb.2009.12.005.View ArticlePubMedGoogle Scholar
- Yamada K, Wu O, Gonzalez RG, Bakker D, Copen WA, Weisskoff RM, Rosen BR, Yagi K, Nishimura T, Sorensen AG: Magnetic resonance perfusion-weighted imaging of acute cerebral infarction: effect of the calculation methods and underlying vasculopathy. Stroke. 2002, 33 (1): 87-94. 10.1161/hs0102.101893.View ArticlePubMedGoogle Scholar
- Galinovic I, Brunecker P, Ostwaldt AC, Soemmer C, Hotter B, Fiebach JB: Fully automated postprocessing carries a risk of substantial overestimation of perfusion deficits in acute stroke magnetic resonance imaging. Cerebrovasc Dis. 2011, 31: 408-413. 10.1159/000323212.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2342/12/16/prepub