Comparison of measurement methods with a mixed effects procedure accounting for replicated evaluations (COM_{3}PARE): method comparison algorithm implementation for head and neck IGRT positional verification
 Anuradha Roy†^{1},
 Clifton D. Fuller†^{2}Email author,
 David I. Rosenthal^{2} and
 Charles R. Thomas Jr.^{3}
Received: 17 July 2014
Accepted: 24 July 2015
Published: 28 August 2015
Abstract
Purpose
Comparison of imaging measurement devices in the absence of a goldstandard comparator remains a vexing problem; especially in scenarios where multiple, nonpaired, replicated measurements occur, as in imageguided radiotherapy (IGRT). As the number of commercially available IGRT presents a challenge to determine whether different IGRT methods may be used interchangeably, an unmet need conceptually parsimonious and statistically robust method to evaluate the agreement between two methods with replicated observations. Consequently, we sought to determine, using an previously reported head and neck positional verification dataset, the feasibility and utility of a Comparison of Measurement Methods with the Mixed Effects Procedure Accounting for Replicated Evaluations (COM_{3}PARE), a unified conceptual schema and analytic algorithm based upon Roy’s linear mixed effects (LME) model with Kronecker product covariance structure in a doubly multivariate setup, for IGRT method comparison.
Methods
An anonymized dataset consisting of 100 paired coordinate (X/ measurements from a sequential series of head and neck cancer patients imaged nearsimultaneously with cone beam CT (CBCT) and kilovoltage Xray (KVX) imaging was used for model implementation. Softwaresuggested CBCT and KVX shifts for the lateral (X), vertical (Y) and longitudinal (Z) dimensions were evaluated for bias, intermethod (betweensubject variation), intramethod (withinsubject variation), and overall agreement using with a script implementing COM_{3}PARE with the MIXED procedure of the statistical software package SAS (SAS Institute, Cary, NC, USA).
Results
COM_{3}PARE showed statistically significant bias agreement and difference in intermethod between CBCT and KVX was observed in the Zaxis (both p − value<0.01). Intramethod and overall agreement differences were noted as statistically significant for both the X and Zaxes (all p − value<0.01). Using prespecified criteria, based on intramethod agreement, CBCT was deemed preferable for Xaxis positional verification, with KVX preferred for superoinferior alignment.
Conclusions
The COM_{3}PARE methodology was validated as feasible and useful in this pilot head and neck cancer positional verification dataset. COM_{3}PARE represents a flexible and robust standardized analytic methodology for IGRT comparison. The implemented SAS script is included to encourage other groups to implement COM_{3}PARE in other anatomic sites or IGRT platforms.
Background

First known application of LMEbased COM_{3}PARE hypothesis testing protocol for method comparison using imaging data.

Demonstration of feasibility and utility of COM_{3}PARE using an established head and neck positional verification dataset, previously presented with standard method comparison approaches.
Methods
Datasets
 1.
No significant bias (i.e., no difference between the means of the two methods under a prespecified threshold nor a statistically significant difference between said means).
 2.
No statistically significant difference in the intersubject (betweensubject) variability of the two methods.
 3.
No statistically significant difference in the intrasubject (withinsubject) variability (i.e., repeatability) of the two methods.
For this study, we prespecified a bias threshold of an absolute value of <0.1 cm, with a statistically significant difference designated by α<0.05. To assess the aforementioned criteria, we implemented the LME methodology proposed by Roy^{48}, referred to as COM_{3}PARE (see Appendix A).
Statistical analysis with COM_{3}PARE
As mentioned in the introduction the number of replicated measurements on each patient or subject may not be equal, and also the number of replications of the two methods on the same subject may not be equal. Let \(p^{KVX}_{i}\) and \(p^{CBCT}_{i}\) be the number of replications on subject i by the established method (KVX), and a new method (CBCT) respectively. Let \(p_{i}= \max \left ({p^{KVX}_{i}, p^{CBCT}_{i}}\right)\), and n_{ i }=2p_{ i }. Therefore, the number of observations on the ith subject is n_{ i }, under the assumption that the ith subject has \(\left p^{KVX}_{i} p^{CBCT}_{i}\right \) missing values.
where b_{1},b_{2},…,b_{ N },ε_{1},ε_{2},…,ε_{ N } are independent, and y_{1},y_{2},…,y_{ N } are also all independent. LME model allows for the explicit analysis of betweensubject (D) and withinsubject (R_{ i }) sources of variation of the two methods. We define the two methods by a vector variable Mvar; Mvar=1 for the KVX method and Mvar=2 for the CBCT method. We choose the intercept and the vector variable Mvar as fixed effects, thus the design matrix X_{ i } has three columns, and consequently β=(β_{ o },β_{1},β_{2})^{′} is a 3dimensional vector containing the fixed effects. We also choose the vector variable Mvar as random effects, i.e., Mvar is random across individual subjects; thus the design matrix Z_{ i } has two columns. Therefore, b_{ i }= (b_{1i},b_{2i})^{′} is a 2dimensional vector containing the random effects.
Thus, the covariance matrix has the same structure for each subject, except that of the dimension. The 2×2 block diagonals Block Ω_{ i } in the overall variancecovariance matrix Ω_{ i } represent the overall variancecovariance matrix between the two methods. Similarly, the 2×2 block diagonals in the overall correlation matrix Ω_{ i }_Correlation represent the overall correlation matrix between the two methods. Thus, the offdiagonal element in this 2×2 overall correlation matrix gives the overall correlation between the two methods. It can be easily seen that the overall variability is the sum of betweensubject variability and withinsubject variability (see Roy^{48} for detail). Thus, we see that if there is a disagreement in overall variabilities, then it may be due to the disagreement in either betweensubject variabilities or withinsubject variabilities or both.
MIXED procedure of SAS
We use MIXED procedure (PROC MIXED) of SAS to get the maximum likelihood estimates (MLEs) of β,D, R_{ i } and Ω_{ i }. METHOD=ML specifies MIXED procedure to calculate the maximum likelihood estimates of the parameters. The COVTEST option requests hypothesis tests for the random effects. CLASS statement specifies the categorical variables. DDFM=KR specifies the KenwardRoger^{51} correction for computing the denominator degrees of freedom for the fixed effects. KenwardRoger correction is suggested whenever one has replicated or repeated measures data; also for missing data. The SOLUTION (S) option in the MODEL statement provides the estimate of the difference between the two mean readings (bias) of the two methods. RANDOM and REPEATED statements specify the structure of the covariance matrices D and R_{ i }. See the sample program in Appendix A that demonstrates the use of RANDOM and REPEATED statements. PROC MIXED calculates the (n_{ i }×n_{ i })dimensional submatrix R_{ i } of the ith subject from the (2p×2p)dimensional matrix (V⊗Σ), and eventually calculates (n_{ i }×n_{ i })dimensional submatrix Ω_{ i }. When the number of replications on each subject by respective methods is unequal, PROC MIXED considers the case as missing value situation. Options V=3 and VCORR=3 in the RANDOM statement give the estimate of the overall variancecovariance matrix Ω_{3} and the corresponding Ω_{3}_Correlation matrix, i.e., for the third subject. The option G in the RANDOM statement gives the estimate of the betweensubject variancecovariance matrix D. Option R in the REPEATED statement gives the estimate of the variancecovariance matrix R_{1} for the first subject. One can get the Ω_{ i } variancecovariance matrix and the corresponding Ω_{ i }_Correlation matrix for all subjects by specifying V= 1 to N, and VCORR=1 to N in the RANDOM statement. When the correlation matrix V on the replicated measurements assumes equicorrelated structure and Σ as unstructured, we use the option TYPE=UN along with SUBJECT=REPLICATE(PATIENT) in the REPEATED statement. This gives the 2× 2 withinsubject variancecovariance matrix Σ. See Appendix A.
Related hypotheses testings to test the disagreement between KVX and CBCT
If there is a disagreement between the two methods, it is important to know whether it is due to the bias, due to the difference in betweensubject variabilities or due to the difference in withinsubject variabilities of the two methods. If it is due to the bias between the two methods, it is easy to correct. The output of PROC MIXED always gives the bias, its t − value and its p − value. Nonetheless, it is not straightforward to check the agreement or disagreement in betweensubject variabilities and in withinsubject variabilities of the two methods. We will accomplish these by the indirect use of PROC MIXED in two steps (described below) by using likelihood ratio tests.
Testing of hypothesis of difference between the means of KVX and CBCT
Output of PROC MIXED (Solution for Fixed Effects) gives the bias and the corresponding t − value and p − value.
Testing of hypothesis of difference in betweensubject variabilities of KVX and CBCT
The log likelihood function under both null hypothesis and alternating hypothesis must be maximized separately. We do this by setting the option METHOD=ML in PROC MIXED statement. The option TYPE=UN in the RANDOM statement, along with the option TYPE=UN in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under H_{ d }. Similarly, the option TYPE=CS in the RANDOM statement, along with the option TYPE=UN in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under K_{ d }.
PROC MIXED calculates “LRT df" under the heading of “Null Model Likelihood Ratio Test", see Appendix B.
Testing of hypothesis of difference in withinsubject variabilities of KVX and CBCT
The option TYPE=UN in the RANDOM statement, along with TYPE=UN in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under H_{ σ }. TYPE=UN in the RANDOM statement, along with TYPE=CS in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under K_{ σ }. The test statistic −2 lnΛ_{ σ } under K_{ σ } follows a chisquare distribution with d.f. ν_{ σ }= LRT df (underH_{ σ })−LRT df (underK_{ σ }).
Testing of hypothesis of difference in overall variabilities of KVX and CBCT
The option TYPE=UN in the RANDOM statement, along with TYPE=UN in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under H_{ ω }. The option TYPE=CS in the RANDOM statement, along with TYPE=CS in the REPEATED statement, is used to calculate the “2 Log Likelihood" for the covariance structure under K_{ ω }. The test statistic −2 lnΛ_{ ω } under K_{ ω } follows a chisquare distribution with d.f. ν_{ ω }= LRT df (underH_{ ω })−LRT df (underK_{ ω }).
Results
The pvalue for testing the withinsubject variabilities of the two methods by using IML procedure of SAS is calculated at the third stage (see Appendix A). The p − value= 5.5112E − 9 (see Appendix B).
Betweenmethod bias
Bias (cm)  p − value  

X  0.0335  0.6077 
Y  0.0428  0.2836 
Z  0.0942  0.0253 
Intermethod agreement
KVX (cm)  CBCT (cm)  p − value  

X  0.0413  0.0670  0.4795 
Y  0.0511  0.0464  0.7518 
Z  0.0273  0.0848  0.0010 
Intramethod agreement
KVX (cm)  CBCT (cm)  p − value  

X  0.3396  0.1047  5.5 ×10^{−9} 
Y  0.1687  0.1693  1.0 
Z  0.0485  0.0825  0.0034 
Overall agreement
KVX (cm)  CBCT (cm)  p − value  

X  0.3809  0.1717  3.7 ×10^{−8} 
Y  0.2198  0.2157  0.9512 
Z  0.0758  0.1673  1.3 ×10^{−5} 
Mixed effects estimated correlation coefficient
Correlation  

coefficient  
X  0.5329 
Y  0.8038 
Z  0.7336 
Using the aforementioned criteria, automated shifts from CBCT and kV Xray, acquired and processed in the manner detailed are interchangeable only for measurements of the Yaxis (anteroposterior), and for example, should not be used on alternating days in facilities with both systems in either X or Zaxis. Additionally, our method suggests that, with lower intramethod variability in the Xaxis (lateral), CBCT is the preferred measurement method, while in the Zaxis (superoinferior) kV Xray measurement is preferable.
Discussion
 1.
The bias and overall agreement must fall within a prespecified range (e.g., bias/agreement of <0.1 cm between IGRT devices).
 2.
There should be no statistically significant, using a prespecified threshold (e.g., <0.05) difference in the intersubject variability of the two methods.
 3.
There should be no statistically significant difference in the intrasubject variability (i.e., repeatability) of the two methods.
 4.
In cases where criteria 2 and 3 are NOT met, the preferred IGRT technique is the one exhibiting the lower intrasubject variability (i.e., greater repeatability).
Conclusion
COM_{3}PARE represents an attempt at a unified conceptual schema and analytic algorithm for method compassion of IGRT platforms. Initial application in a head and neck positional verification dataset shows feasibility and utility.
Appendix A
SAS code
Below we provide the sample SAS code to test withinsubject variabilities by fitting the linear mixed effects model to our KVX and CBCT shifts for the lateral (X). We first fit the linear mixed effects model for the null hypothesis, then we fit the linear mixed effects model for the alternating hypothesis, and then find the p − value for the test. Appropriate changes can be made to test betweensubject variabilities and overall variabilities using the SAS commands as described in Sections Testing of hypothesis of difference in betweensubject variabilities of KVX and CBCT and Testing of hypothesis of difference in overall variabilities of KVX and CBCT. Appropriate changes can be also made for vertical (Y) and longitudinal (Z) dimensions and for any other data sets.
Appendix B
Declarations
Acknowledgements
Special thanks to Joseph Ting, PhD for dataset utilization permission.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986; 1:307–10.View ArticlePubMedGoogle Scholar
 Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet. 1995; 346:1085–7.View ArticlePubMedGoogle Scholar
 Bland JM, Altman DG. Comparing two methods of clinical measurement: a personal history. Int J Epidemiol. 1995; 24(Suppl 1):S7–14.View ArticlePubMedGoogle Scholar
 Bland JM, Altman DG. Measurement error. BMJ (Clinical research ed.) 1996; 312:1654.View ArticleGoogle Scholar
 Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999; 8:135–60.View ArticlePubMedGoogle Scholar
 Bland JM, Altman DG. Applying the right statistics: analyses of measurement studies. Ultrasound Obstet Gynecol. 2003; 22:85–93.View ArticlePubMedGoogle Scholar
 Bland JM, Altman DG. Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat. 2007; 17:571–82.View ArticlePubMedGoogle Scholar
 Bisdas S, Konstantinou G, SurlanPopovic K, Khoshneviszadeh A, Baghi M, Vogl TJ, et al. Dynamic contrastenhanced CT of head and neck tumors: comparison of firstpass and permeability perfusion measurements using two different commercially available tracer kinetics models. Acad Radiol. 2008; 15:1580–9.View ArticlePubMedGoogle Scholar
 Cronin P, Saab A, Kelly AM, Gross BH, Patel S, Kazerooni EA, Carlos RC. Measurements of pulmonary vein ostial diameter and distance to first bifurcation: A comparison of different measurement methods. Eur J Radiol. 2009; 71(1):61–8.View ArticlePubMedGoogle Scholar
 Kanza RE, Higashino H, Kido T, Kurata A, Saito M, Sugawara Y, Mochizuki T. Quantitative assessment of regional left ventricular wall thickness and thickening using 16 multidetectorrow computed tomography: comparison with cine magnetic resonance imaging. Radiat Med. 2007; 25:119–26.View ArticlePubMedGoogle Scholar
 Kwee TC, Takahara T, Koh DM, Nievelstein RA, Luijten PR. Comparison and reproducibility of ADC measurements in breathhold, respiratory triggered, and freebreathing diffusionweighted MR imaging of the liver. J Magn Reson Imaging. 2008; 28:1141–8.View ArticlePubMedGoogle Scholar
 Mahnken AH, Spuentrup E, Niethammer M, Buecker A, Boese J, Wildberger JE Flohr T, et al. Quantitative and qualitative assessment of left ventricular volume with ECGgated multislice spiral CT: value of different image reconstruction algorithms in comparison to MRI. Acta Radiol. 2003; 44:604–11.View ArticlePubMedGoogle Scholar
 Martin KE, Helvie MA, Zhou C, Roubidoux MA, Bailey JE, Paramagul C, et al. Mammographic density measured with quantitative computeraided method: comparison with radiologists’ estimates and BIRADS categories. Radiology. 2006; 240:656–65.View ArticlePubMedGoogle Scholar
 Murakami R, Uozumi H, Hirai T, Nishimura R, Katsuragawa S, Shiraishi S, Toya R, Tashiro K, Kawanaka K, Oya N, Tomiguchi S, Yamashita Y. Impact of FDGPET/CT fused imaging on tumor volume assessment of headandneck squamous cell carcinoma: intermethod and interobserver variations. Acta Radiol. 2008; 49:693–9.View ArticlePubMedGoogle Scholar
 Nieman K, Shapiro MD, Ferencik M, Nomura CH, Abbara S, Hoffmann U, et al. Reperfused myocardial infarction: contrastenhanced 64Section CT in comparison to MR imaging. Radiology. 2008; 247:49–56.View ArticlePubMedGoogle Scholar
 Agazaryan N, Tenn SE, Desalles AA, Selch MT. Imageguided radiosurgery for spinal tumors: methods, accuracy and patient intrafraction motion. Phys Med Biol. 2008; 53:1715–27.View ArticlePubMedGoogle Scholar
 Dawson LA, Jaffray DA. Advances in imageguided radiation therapy. J Clin Oncol. 2007; 25:938–946.View ArticlePubMedGoogle Scholar
 Grills IS, Hugo G, Kestin LL, Galerani AP, Chao KK, Wloch J, et al. Imageguided radiotherapy via daily online conebeam CT substantially reduces margin requirements for stereotactic lung radiotherapy. Int J Radiat Oncol Biol Phys. 2008; 70:1045–56.View ArticlePubMedGoogle Scholar
 Ippolito E, Mertens I, Haustermans K, Gambacorta MA, Pasini D, Valentini V. IGRT in rectal cancer. Acta oncologica (Stockholm, Sweden). 2008; 47:1317–24.View ArticleGoogle Scholar
 Kupelian PA, Langen KM, Willoughby TR, Zeidan OA, Meeks SL. Imageguided radiotherapy for localized prostate cancer: treating a moving target. Semin Radiat Oncol. 2008; 18:58–66.View ArticlePubMedGoogle Scholar
 Wertz H, Lohr F, Dobler B, Mai S, Wenz F. Dosimetric impact of imageguided translational isocenter correction for 3D conformal radiotherapy of the prostate. Strahlenther Onkol. 2007; 183:203–10.View ArticlePubMedGoogle Scholar
 Wiersma RD, Mao W, Xing L. Combined kV and MV imaging for realtime tracking of implanted fiducial markers. Med Phys. 2008; 35:1191–8.View ArticlePubMedPubMed CentralGoogle Scholar
 Yorke ED, Keall P, Verhaegen F. Anniversary paper: Role of medical physicists and the AAPM in improving geometric aspects of treatment accuracy and precision. Med Phys. 2008; 35:828–39.View ArticlePubMedGoogle Scholar
 Fuller CD, Thomas CR, Schwartz S, Golden N, Ting J, Wong A, et al. Method comparison of ultrasound and kilovoltage xray fiducial marker imaging for prostate radiotherapy targeting. Phys Med Biol. 2006; 51:4981–93.View ArticlePubMedGoogle Scholar
 Moseley DJ, White EA, Wiltshire KL, Rosewall T, Sharpe MB, Siewerdsen JH, et al. Comparison of localization performance with implanted fiducial markers and conebeam computed tomography for online imageguided radiotherapy of the prostate. Int J Radiat Oncol Biol Phys. 2007; 67:942–53.View ArticlePubMedPubMed CentralGoogle Scholar
 Artignan X, Smitsmans MH, Lebesque JV, Jaffray DA, van Her M, Bartelink H. Online ultrasound image guidance for radiotherapy of prostate cancer: impact of image acquisition on prostate displacement. Int J Radiat Oncol Biol Phys. 2004; 59:595–601.View ArticlePubMedGoogle Scholar
 Cury FL, Shenouda G, Souhami L, Duclos M, Faria SL, David M, et al. Ultrasoundbased image guided radiotherapy for prostate cancer: comparison of crossmodality and intramodality methods for daily localization during external beam radiotherapy. Int J. Radiat Oncol Biol Phys. 2006; 66:1562–7.View ArticlePubMedGoogle Scholar
 Fung AY, Ayyangar KM, Djajaputra D, Nehru RM, Enke CA. Ultrasoundbased guidance of intensitymodulated radiation therapy. NMed Dosim. 2006; 31:20–9.View ArticleGoogle Scholar
 Fuss M, Salter BJ, Cavanaugh SX, Fuss C, Sadeghi A, Fuller CD, et al. Daily ultrasoundbased imageguided targeting for radiotherapy of upper abdominal malignancies. Int J Radiat Oncol Biol Phys. 2004; 59:1245–56.View ArticlePubMedGoogle Scholar
 Langen KM, Pouliot J, Anezinos C, Aubin M, Gottschalk AR, Hsu IC, et al. Evaluation of ultrasoundbased prostate localization for imageguided radiotherapy. Int J. Radiat Oncol Biol Phys. 2003; 57:635–44.View ArticlePubMedGoogle Scholar
 Patel RR, Orton N, Tome WA, Chappell R, Ritter MA. Rectal dose sparing with a balloon catheter and ultrasound localization in conformal radiation therapy for prostate cancer. Radiother Oncol. 2003; 67:285–94.View ArticlePubMedGoogle Scholar
 Choi M, Fuller CD, Wang SJ, Siddiqi A, Wong A, Thomas Jr CR, Fuss M. Effect of body mass index on shifts in ultrasoundbased imageguided intensitymodulated radiation therapy for abdominal malignancies. Radiother Oncol. 2009; 91(1):114–9.View ArticlePubMedGoogle Scholar
 Gayou O, Miften M. Comparison of megavoltage conebeam computed tomography prostate localization with online ultrasound and fiducial markers methods. Med Phys. 2008; 35:531–8.View ArticlePubMedGoogle Scholar
 Johnston H, Hilts M, Beckham W, Berthelet E. 3D ultrasound for prostate localization in radiation therapy: a comparison with implanted fiducial markers. Med Phys. 2008; 35:2403–13.View ArticlePubMedGoogle Scholar
 Aubry JF, Beaulieu L, Girouard LM, Aubin S, Tremblay D, Laverdiere J, et al. Measurements of intrafraction motion and interfraction and intrafraction rotation of prostate by threedimensional analysis of daily portal imaging with radiopaque markers. Int J Radiat Oncol Biol Phys. 2004; 60:30–9.View ArticlePubMedGoogle Scholar
 Balter JM, Sandler HM, Lam K, Bree RL, Lichter AS, Haken TENRK. Measurement of prostate movement over the course of routine radiotherapy using implanted markers. Int J Radiat Oncol Biol Phys. 1995; 31:113–8.View ArticlePubMedGoogle Scholar
 Evans PM. Anatomical imaging for radiotherapy. Phys Med Biol. 2008; 53:R151–191.View ArticlePubMedGoogle Scholar
 Lometti MW, Thurston D, Aubin M, Bock A, Verhey L, Lockhart JM, Bland R, Pouliot J, Roach 3rd M. Are lateral electronic portal images adequate for accurate online daily targeting of the prostate? Results of a prospective study. Med Dosim. 2008; 33:22–9.Google Scholar
 Sornsen de Kostevan JR, Cuijpers JP, Geest de FG, Lagerwaard FJ, Slotman BJ, Senan S. Verifying 4D gated radiotherapy using timeintegrated electronic portal imaging: a phantom and clinical study. adiat Oncol (London, England). 2007; 2:32.View ArticleGoogle Scholar
 Morin O, Gillis A, Chen J, Aubin M, Bucci MK, Roach 3rd M, et al. Megavoltage conebeam CT: system description and clinical applications. Med Dosim. 2006; 31:51–61.Google Scholar
 Pang G, BaniHashemi A, Au P, O’Brien PF, Rowlands JA, Morton G, et al. Megavoltage cone beam digital tomosynthesis (MVCBDT) for imageguided radiotherapy: a clinical investigational system. Phys Med Biol. 2008; 53:999–1013.View ArticlePubMedGoogle Scholar
 Pisani L, Lockman D, Jaffray D, Yan D, Martinez A, Wong J. Setup error in radiotherapy: online correction using electronic kilovoltage and megavoltage radiographs. Int J Radiat Oncol Biol Phys. 2000; 47:825–39.View ArticlePubMedGoogle Scholar
 Pouliot J, BaniHashemi A, Chen J, Svatos M, Ghelmansarai F, Mitschke M, et al. Lowdose megavoltage conebeam CT for radiation therapy. Int J Radiat Oncol Biol Phys. 2005; 61:552–60.View ArticlePubMedGoogle Scholar
 Serago CF, Buskirk SJ, Igel TC, Gale AA, Serago NE, Earle JD. Comparison of daily megavoltage electronic portal imaging or kilovoltage imaging with marker seeds to ultrasound imaging or skin marks for prostate localization and treatment positioning in patients with prostate cancer. Int J Radiat Oncol Biol Phys. 2006; 65:1585–92.View ArticlePubMedGoogle Scholar
 Wu QJ, Godfrey DJ, Wang Z, Zhang J, Zhou S, Yoo S, Brizel DM, Yin FF. Onboard patient positioning for headandneck IMRT: comparing digital tomosynthesis to kilovoltage radiography and conebeam computed tomography. Int J Radiat Oncol Biol Phys. 2007; 69:598–606.View ArticlePubMedGoogle Scholar
 Scarbrough TJ, Golden NM, Ting JY, Fuller CD, Wong A, Kupelian PA, Thomas Jr CR. Comparison of ultrasound and implanted seed marker prostate localization methods: Implications for imageguided radiotherapy. Int J Radiat Oncol Biol Phys. 2006; 65:378–87.View ArticlePubMedGoogle Scholar
 Stutzel J, Oelfke U, Nill S. A quantitative image quality comparison of four different image guided radiotherapy devices. Radiother Oncol. 2008; 86:20–4.View ArticlePubMedGoogle Scholar
 Roy A. An application of linear mixed effects model to assess the agreement between two methods with replicated observations. J Biopharm Stat. 2009; 19:150–73.View ArticlePubMedGoogle Scholar
 Barnhart HX, Haber MJ, Lin LI. An overview on assessing agreement with continuous measurements. J Biopharm Stat. 2007; 17:529–69.View ArticlePubMedGoogle Scholar
 Fuller CD, Scarbrough TJ, Sonke JJ, Rasch CR, Choi M, Ting JY, et al. Method comparison of automated matching softwareassisted conebeam CT and stereoscopic kilovoltage xray positional verification imageguided radiation therapy for head and neck cancer: a prospective analysis. Phys Med Biol. 2009; 54:7401–15.View ArticlePubMedGoogle Scholar
 Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics. 1997; 53:983–997.View ArticlePubMedGoogle Scholar
 Roy A. Estimating correlation coefficient between two variables with repeated observations using mixed effects model. Biometrical J. 2006; 48:286–301.View ArticleGoogle Scholar
 Cui Y, Galvin JM, Straube WL, Bosch WR, Purdy JA, Li XA, Xiao Y. MultiSystem Verification of Registrations for ImageGuided Radiotherapy in Clinical Trials. Int J Radiat Oncol Biol Phys. 2011; 81(1):305–312.View ArticlePubMedPubMed CentralGoogle Scholar
 Mohammed N, Kestin L, Grills I, Shah C, GlideHurst C, Yan D, et al. Comparison of IGRT Registration Strategies for Optimal Coverage of Primary Lung Tumors and Involved Nodes Based on Multiple FourDimensional CT Scans Obtained Throughout the Radiotherapy Course. Int J Radiat Oncol Biol Phys. 2012; 82(4):1541–8.View ArticlePubMedGoogle Scholar
 Shi W, Li JG, Zlotecki RA, Yeung A, Newlin H, Palta J, et al. Evaluation of kV conebeam ct performance for prostate IGRT: a comparison of automatic greyvalue alignment to implanted fiducialmarker alignment. Am J Clin Oncol. 2011; 34(1):16–21.View ArticlePubMedGoogle Scholar
 Thongphiew D, Wu QJ, Lee WR, Chankong V, Yoo S, McMahon R, Yin FF. Comparison of online IGRT techniques for prostate IMRT treatment: adaptive vs repositioning correction. Med Phys. 2009; 36:1651–62.View ArticlePubMedGoogle Scholar
 Borst GR, Sonke JJ, Betgen A, Remeijer P, Herk van M, Lebesque JV. Kilovoltage conebeam computed tomography setup measurements for lung cancer patients; first clinical results and comparison with electronic portalimaging device. Int J Radiat Oncol Biol Phys. 2007; 68:555–61.View ArticlePubMedGoogle Scholar
 Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989; 45:255–68.View ArticlePubMedGoogle Scholar
 Konings H. Use of Deming regression in methodcomparison studies. Surv Immunol Res. 1982; 1:371–74.PubMedGoogle Scholar
 Martin RF. General deming regression for estimating systematic bias and its confidence interval in methodcomparison studies. Clinical Chem. 2000; 46:100–4.Google Scholar
 Milliken G, Graybill F. Extensions of the General Linear Hypothesis Model. J Am Stat Assoc. 1970; 65:797–807.View ArticleGoogle Scholar
 LaMotte L. A Canonical Form for the General Linear Model. Ann Stat. 1977; 5:787–9.View ArticleGoogle Scholar
 Fennessey J. The General Linear Model: A New Perspective on Some Familiar Topics. Am J Sociol. 1968; 74:1–27.View ArticleGoogle Scholar