Skip to main content
Download PDF

Artificial intelligence stenosis diagnosis in coronary CTA: effect on the performance and consistency of readers with less cardiovascular experience

BMC Medical Imaging volume 22, Article number: 28 (2022) Cite this article

Abstract

Background

To investigate the influence of artificial intelligence (AI) based on deep learning on the diagnostic performance and consistency of inexperienced cardiovascular radiologists.

Methods

We enrolled 196 patents who had undergone both coronary computed tomography angiography (CCTA) and invasive coronary angiography (ICA) within 6 months. Four readers with less cardiovascular experience (Reader 1–Reader 4) and two cardiovascular radiologists (level II, Reader 5 and Reader 6) evaluated all images for ≥ 50% coronary artery stenosis, with ICA as the gold standard. Reader 3 and Reader 4 interpreted with AI system assistance, and the other four readers interpreted without the AI system. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy (area under the receiver operating characteristic curve (AUC)) of the six readers were calculated at the patient and vessel levels. Additionally, we evaluated the interobserver consistency between Reader 1 and Reader 2, Reader 3 and Reader 4, and Reader 5 and Reader 6.

Results

The AI system had 94% and 78% sensitivity at the patient and vessel levels, respectively, which were higher than that of Reader 5 and Reader 6. AI-assisted Reader 3 and Reader 4 had higher sensitivity (range + 7.2–+ 16.6% and + 5.9–+ 16.1%, respectively) and NPVs (range + 3.7–+ 13.4% and + 2.7–+ 4.2%, respectively) than Reader 1 and Reader 2 without AI. Good interobserver consistency was found between Reader 3 and Reader 4 in interpreting ≥ 50% stenosis (Kappa value = 0.75 and 0.80 at the patient and vessel levels, respectively). Only Reader 1 and Reader 2 showed poor interobserver consistency (Kappa value = 0.25 and 0.37). Reader 5 and Reader 6 showed moderate agreement (Kappa value = 0.55 and 0.61).

Conclusions

Our study showed that using AI could effectively increase the sensitivity of inexperienced readers and significantly improve the consistency of coronary stenosis diagnosis via CCTA.

Trial registration Clinical trial registration number: ChiCTR1900021867. Name of registry: Diagnostic performance of artificial intelligence-assisted coronary computed tomography angiography for the assessment of coronary atherosclerotic stenosis.

Peer Review reports

Background

Many studies have demonstrated the high accuracy of coronary computed tomography angiography (CCTA) compared to invasive coronary angiography (ICA) in detecting coronary stenosis, particularly due to the high sensitivity and negative predictive value (NPV) of CCTA in coronary artery disease diagnosis. However, in previous studies, most of the results were interpreted by cardiovascular experts [1,2,3,4,5,6]. Reader diagnostic experience significantly impacts the identification of the degree of coronary artery stenosis, as well as the interobserver interpretation variability, with less experienced readers tending to miss lesions, resulting in relatively low diagnostic sensitivity [7, 8]. Less experienced readers also tend to overestimate lesions due to the calcium-blooming effects of hard plaques. Therefore, for radiologists with less cardiovascular experience, the development of an automated system to aid in diagnosis is attractive and promising.

Automated artificial intelligence (AI) algorithms have been applied in the diagnosis of a wide range of disease states [9,10,11,12,13,14]; for example, the use of AI in the detection and diagnosis of breast cancer and colon polyps has improved reader performance, especially for inexperienced or novice readers [10, 15, 16]. However, AI algorithms are still rarely applied in coronary artery disease [13, 17, 18]. Our study sought to assess the influence of AI based on deep learning on the diagnostic performance and consistency of inexperienced cardiovascular radiologists.

Methods

Study population

This study was a single-centre retrospective study approved by the Ethics Committee of Beijing Friendship Hospital, Capital Medical University (Central Office for Research Ethics Committee Reference 2020-P2-010-02). Between January 2017 and October 2018, 252 consecutive patients (aged over 18 years) with suspected or known coronary heart disease (CHD) underwent both CCTA and ICA examinations within six months. Patients with iodine contrast agent allergy, atrial fibrillation, renal failure or pregnancy were excluded from this study. Of the initially included patients, seventeen had incomplete CCTA or ICA data, 6 had abnormal coronary origins or had undergone bypass surgery, 7 had poor image quality, and 26 had three vessel lesions (left anterior descending (LAD) artery, left circumflex (LCx) artery and right coronary artery (RCA)) that could not be simultaneously evaluated. These patients were excluded due to severely extensive calcification (the standard of severe calcification was cross-sectional arc calcium > 180°) [19], stents and motion artefacts. Finally, 196 patients were enrolled (Fig. 1). ICA is the gold standard of diagnosis and was jointly interpreted for ≥ 50% stenosis by an expert panel of three cardiovascular experts with at least 10 years of experience in both ICA and CCTA.

Fig. 1
figure 1

Flow diagram of the patient selection process. CCTA coronary computed tomography angiography, ICA invasive coronary angiography

Full size image

CCTA image acquisition

A 256-section CT (GE Healthcare, Waukesha, Wisconsin, US), a 64-section CT (GE Healthcare, Waukesha, Wisconsin, US) and a 128-row multidetector CT (Philips Medical Systems, Eindhoven, The Netherlands) were used to capture patient image data. Prospective electrocardiographic gating was employed. IoproMide (IoproMide, Ultravist 370; Bayer Healthcare LLC, Whippany, New Jersey) or Iohexol (Omnipaque 350, GE Healthcare, Princeton, NJ) was injected at 5–6 ml/s into the antecubital vein. All scanners had a layer thickness and spacing of 0.625 mm.

CCTA analysis

All 196 CCTA patient datasets were reconstructed at a workstation (GE Advantage Workstation 4.6 or 4.7, GE Healthcare, Waukesha, Wisconsin) to transform the data into multiplanar reformation (MPR) and curved MPR (cMPR) images based on the original axial image. These images were then transferred to a picture archiving and communication system (PACS). Patients were defined as positive for significant coronary artery disease when ≥ 50% stenosis was observed. Six readers interpreted stenosis via CCTA for all patients. The readers had different levels of experience. Four readers (Readers 1 to 4) were general radiologists with less experience in cardiovascular imaging who had interpreted less than 50 cases of coronary artery stenosis via CCTA and had not been mentored [20]. Readers 5 and 6 were cardiovascular radiologists with at least 5 years of CCTA experience, corresponding to level II competency (independent practitioners, IP) [21]. Inexperienced Readers 1 and 2 and experienced Readers 5 and 6 evaluated all patient data on the same PACS without the AI system. Inexperienced Readers 3 and 4 evaluated the same patient data on the AI workstation and received AI assistance in coronary stenosis diagnosis. This study did not require the same readers to interpret the 196 CCTA datasets both with and without AI because CCTA interpretation experience is related to the number of cases evaluated [7, 20, 22]. Interpreting 196 CCTA datasets would have a significant impact on a reader’s experience; therefore, we selected four readers with similar experience levels rather than requiring the same readers to interpret the data twice. Thus, reader recall bias was effectively avoided. The AI system could independently perform automatic reconstruction and intelligently diagnose coronary artery stenosis. All readers analysed the four primary coronary arteries, the left main (LM) artery, LAD, LCx and RCA, and recorded the presence of both vessels and ≥ 50% stenosis. Vessels with severely extensive calcified plaque, stents or a diameter of ≤ 1.5 mm were excluded from this study.

AI system

Data acquisition

The AI system used was “CoronaryDoc clinical decision Support Platform V1.0” from Shukun (Beijing) Technology Co., Ltd. [23]. All CCTA data were transferred from a GE Advantage Workstation 4.6 or 4.7 to the AI workstation, and then the AI system extracted the centerline [24] and automatically reconstructed MPR and cMPR images based on the original axial image.

Coronary artery segmentation and naming

Coronary arteries were divided into 18 segments according to the Society of Cardiovascular Computed Tomography (SCCT) criteria [25]. An improved 3-dimensional (3D) U-Net coronary tree segmentation architecture was used. The AI system used an automatic identification algorithm to achieve coronary artery segmentation and naming [24].

Automatic reconstruction and intelligent diagnosis of coronary artery stenosis

The system was based on coronary tree segmentation, MPR, straightened rendering (SR), cMPR, maximum intensity projection (MIP) and volume rendering (VR) images, and it automatically reconstructed the data. Stenosis along the long axis of a vessel was calculated based on the radius of the lumen at the plaque location and the upstream and downstream blood vessel radii (details are provided in the Additional file 1: Appendices).

Statistical analysis

All data were analysed using SPSS (version 26.0, IBM, Washington, USA) and MedCalc (version 19.0.4, bvba, Ostend, Belgium) software. Assuming a 60% prevalence of coronary artery disease at the patient level at our single centre, the area under the receiver operating characteristic curve (AUC) of patients without AI and with AI was 68% and 77%, respectively. The sample size was estimated to be 143 patients, with 86 disease cases and 57 disease-free cases. Categorical variables are shown as percentages, and continuous variables are shown as the means and ranges. We used sensitivity, specificity, the positive predictive value (PPV), the NPV, and the AUC to describe diagnostic performance and accuracy. ICA was used as the gold standard to assess the results of ≥ 50% coronary artery stenosis detection by the 6 readers at the patient and vessel levels. In patient-level analysis, a patient with ≥ 50% stenosis in any vessel or segment was considered a positive case. In vessel-level analysis, any vessel with stenosis ≥ 50% was considered a positive case. AUC comparisons among the 6 readers were performed by the method of DeLong et al. [26]. We used Cohen's kappa coefficient to evaluate the interobserver consistency in detecting ≥ 50% stenosis lesions among the inexperienced readers without AI assistance, the inexperienced readers with AI assistance and the two cardiovascular radiologists. Vessels not evaluated by the six readers or the AI system were not statistically analysed in this study. A P value < 0.05 was considered statistically significant.

Results

Patients

The mean age of the patients was 63.9 ± 8.8 years, and the mean heart rate was 65.2 ± 12.5 beats/min. The clinical information of the patients is shown in Table 1. At the vessel level, seventeen vessels contained stents, ten had severely extensive calcified plaque, one had a vessel diameter ≤ 1.5 mm (RCA), and one had no LM. Finally, 755 vessels were included in the analysis. ICA showed 228 vessel stenoses ≥ 50% in 139 patients (71%), of which six were in the LM, 107 were in the LAD, 54 were in the LCx, and 61 were in the RCA (Fig. 1 and Fig. 2).

Table 1 Clinical features of the 196 patients
Full size table
Fig. 2
figure 2

Single-centre retrospective case of a noncalcified plaque lesion in the proximal LAD that caused severe stenosis (white arrow). a CPR image reconstructed manually, b ICA image of the left coronary artery, c CPR, VRT and straightened images automatically reconstructed by the AI system. ICA invasive coronary angiography, LAD left anterior descending, LCx left circumflex

Full size image

Patient-level analysis of the six readers

Table 2 shows the detailed reader performance and accuracy in identifying ≥ 50% stenosis at the patient level. At the patient level, Readers 3 (85.6%) and 4 (87.1%), who were aided by the AI system, had higher sensitivity than Readers 1 (70.5%) and 2 (78.4%), who did not use the AI system (range + 7.2–+ 16.6%). A statistically significant difference in sensitivity was found between Reader 1 and inexperienced readers with AI assistance (P = 0.001 and P < 0.001). Reader 2 demonstrated no difference in sensitivity (P = 0.164 and P = 0.81), a higher NPV (range + 3.7–+ 13.4%), lower specificity (range − 17.5 to − 8.8%), and no significant difference in PPV. Overall, the AUCs of the four inexperienced readers did not significantly differ (all P values > 0.05). The AUCs, as a measure of diagnostic accuracy, ranged from 0.68 to 0.71 for the inexperienced readers and were lower than those of the experienced readers (that of Reader 5 was 0.77, and that of Reader 6 was 0.76). Of the six readers’ AUCs, only those of Reader 4 and Reader 5 were significantly different (P = 0.02), while the other readers showed no significant differences (Fig. 3).

Table 2 Diagnostic performance of the six readers and the AI system alone for the same set of patients
Full size table
Fig. 3
figure 3

AUC comparison of the six readers at the patient level

Full size image

Vessel-level analysis of the six readers

Table 3 shows the detailed reader performance and accuracy in identifying ≥ 50% stenosis at the vessel level. At the vessel level, Readers 3 (67.1%) and 4 (69.3%), who were aided by the AI system, had a higher sensitivity than Readers 1 (53.2%) and 2 (61.2%), who did not have AI assistance (range + 5.9–+ 16.1%). A statistically significant difference in sensitivity was found between Reader 1 and the inexperienced readers with AI assistance (P = 0.001 and P < 0.001), while Reader 2 demonstrated no difference (P = 0.208 and P = 0.76) and a slightly higher NPV (range + 2.7–+ 4.2%). Reader 1 (89.2%) had higher specificity than Readers 3 (83.2%) and 4 (81.3%), while Reader 2 (78.3%) had slightly lower specificity than Readers 3 and 4. Overall, the AUCs of the four inexperienced readers did not significantly differ (all P value > 0.05). The AUC, as a measure of diagnostic accuracy, ranged from 0.80 to 0.83 for the inexperienced readers and was lower for the experienced readers (those of Readers 5 and 6 were both 0.85). Of the six readers’ AUCs, only the AUC of Reader 2 was significantly different from those of the experienced readers (P = 0.009 and P = 0.002), while Readers 4 and 6 showed slightly statistically significant differences (P = 0.045). The AUCs of other readers did not significantly differ (Fig. 4).

Table 3 Diagnostic performance of the six readers and the AI system alone for the same set of vessels
Full size table
Fig. 4
figure 4

AUC comparison of the six readers at the vessel level

Full size image

Diagnostic performance of the AI system

The CCTA data of all 196 patients were successfully uploaded and automatically reconstructed. To identify ≥ 50% stenosis, the AI system could output the degree of stenosis in each present vessel and segment. At the patient level, the AI system alone had 93.5% sensitivity, 57.9% specificity, 84.4% PPV, 78.6% NPV and 80.0% accuracy (Table 2). At the vessel level, the AI system alone had 78.1% sensitivity, 82.5% specificity, 65.9% PPV, 89.7% NPV and 84.5% accuracy (Table 3). Compared to ICA, AI indicated 9/139 (6.5%) false-positive and 24/57 (42.1%) false-negative patients at the patient level and 50/228 (21.9%) false-positive and 92/526 (17.5%) false-negative vessels at the vessel level. The AI system missed one lesion (LCx) at the vessel level, while no lesions were missed at the patient level. Overall, the diagnostic accuracy of the AI system was slightly higher than that of the experienced readers (IP) at the patient level and close to that of the experienced readers at the vessel level.

Diagnostic consistency of the six readers

Good interobserver consistency was found between the two inexperienced readers aided by the AI system (Readers 3 and 4) in interpreting lesions with more than 50% stenosis, with Kappa values of 0.75 (95% CI 0.64, 0.86) and 0.80 (95% CI 0.75, 0.84) at the patient and vessel levels, respectively. However, the two inexperienced readers without AI assistance (Readers 1 and 2) had very poor interobserver consistency, with kappa values of 0.25 (95% CI 0.11, 0.38) and 0.37 (95% CI 0.30, 0.44) at the patient and vessel levels, respectively. In interpreting lesions with more than 50% stenosis, the experienced cardiovascular radiologists without AI assistance (Readers 5 and 6) showed moderate agreement, with kappa values of 0.55 (95% CI 0.43, 0.66) and 0.61 (95% CI 0.54, 0.68) at the patient and vessel levels, respectively.

Discussion

Automated computerized detection and diagnostic systems have been introduced as auxiliary tools for radiologists in diverse diagnostic processes [9,10,11,12,13,14]. Coronary artery disease is one of the leading causes of life-threatening health problems in developing countries [27, 28], but AI is still rarely used for this disease diagnosis [18]. At present, several studies have compared the diagnostic performance of AI with that of ICA or experts [17, 29]. The preliminary conclusion is that AI has great value and promise for application and may be used as a diagnostic aid by inexperienced radiologists. The influence of AI on the diagnostic performance and consistency of inexperienced readers has relatively rarely been studied for the diagnosis of coronary artery disease with ≥ 50% stenosis.

Our results indicated that the use of AI as a diagnostic aid might have a positive effect on inexperienced radiologists in diagnosing coronary stenosis on CCTA. The most obvious effect was that inexperienced readers with AI assistance performed better than those without AI assistance at both the patient and vessel levels, and this difference was statistically significant. Additionally, the NPV of the former was also higher. Moreover, the consistency of inexperienced readers was significantly higher. When inexperienced readers used the AI system as an auxiliary reader, their sensitivity reached or surpassed that of more experienced readers (IP). In our analysis, based on the mildly improved diagnostic accuracy observed when using the AI system, the sensitivity and NPV were significantly higher, without a significant decrease in specificity or PPV. These findings were similar to those of previous studies evaluating the effect of AI in diagnosing coronary artery disease and other diseases [10, 13, 16]. Moreover, the AI training of less experienced radiologists was also dominated by an increase in sensitivity, which is in line with the results of previous studies [7, 30]. Increased sensitivity of less experienced readers could improve radiologists’ abilities to detect obstructed coronary arteries and reduce missed disease diagnoses. The reason for the increased sensitivity of inexperienced readers may be their tendency to rely the AI system when unsure of the presence of obstructive disease. Consequently, the AI system, with high sensitivity (93.5% at the patient level) and low specificity (57.9% at the patient level), also reflected the performance of inexperienced readers. Importantly, AI assistance clearly increased the consistency of the inexperienced readers (the kappa value increased from 0.25 to 0.75 at the patient level).

Experienced readers (IP) had higher diagnostic accuracy than inexperienced readers, suggesting that the reader’s experience influenced their diagnostic performance and that specific cardiovascular training is important. In clinical work, specialty training, although important, is time-consuming and demanding, especially for inexperienced practitioners. With the increasing use of deep learning algorithms, at present, AI has achieved higher sensitivity but lower specificity at the patient level. Therefore, the AI system had similar limitations to those of human readers in that it could not accurately measure the severity of lesions and overestimated them compared to ICA or quantitative coronary angiography (QCA) [31]. The algorithm is still undergoing optimization. Considering these factors, AI assistance still holds great potential for improving disease detection and excluding stenosis at the patient level for less experienced readers or novices, and it may provide an appropriate training alternative.

Our study had several limitations. First, we used ICA as the reference standard, and enrolled patients who underwent ICA were likely to have severe stenosis, which might have led to a high disease prevalence (71% at the patient level); thus, this study is subject to the same selection bias shown in previous comparative studies [13]. Second, we chose 50% coronary stenosis as the cut-off value based on previous research, but this analysis was different from actual clinical practice [13, 32]. Third, due to the relatively large number of patients enrolled and considering that case interpretation may have impacted inexperienced readers, diagnostic performance was not compared among the same readers with and without AI assistance. Instead, we selected different readers with similar experience levels, which might have impacted the obtained results; however, the overall trend was consistent with those of previous studies using the same readers for both tasks. Therefore, the study design effectively avoided methodological reader recall bias.

In conclusion, as a supplement, the AI system could effectively increase the diagnostic sensitivity of less experienced readers and significantly improve their consistency.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author upon reasonable request. All data generated or analysed during this study are included in this published article [and its supplementary information files].

Abbreviations

CCTA:

Coronary computed tomography angiography

ICA:

Invasive coronary angiography

AI:

Artificial intelligence

CHD:

Coronary heart disease

PACS:

Picture archiving and communication system

IP:

Independent practitioners

SCCT:

Society of Cardiovascular Computed Tomography

References

  1. Miller JM, Rochitte CE, Dewey M, Arbab-Zadeh A, Niinuma H, Gottlieb I, et al. Diagnostic performance of coronary angiography by 64-row CT. N Engl J Med. 2008;359(22):2324–36.

    Article  CAS  Google Scholar 

  2. de Graaf FR, Schuijf JD, van Velzen JE, Kroft LJ, de Roos A, Reiber JHC, et al. Diagnostic accuracy of 320-row multidetector computed tomography coronary angiography in the non-invasive evaluation of significant coronary artery disease. Eur Heart J. 2010;31(15):1908–15.

    Article  Google Scholar 

  3. Budoff MJ, Dowe D, Jollis JG, Gitter M, Sutherland J, Halamert E, et al. Diagnostic performance of 64-multidetector row coronary computed tomographic angiography for evaluation of coronary artery stenosis in individuals without known coronary artery disease. J Am Coll Cardiol. 2008;52(21):1724–32.

    Article  Google Scholar 

  4. Meijboom WB, Meijs MF, Schuijf JD, Cramer MJ, Mollet NR, van Mieghem CA, et al. Diagnostic accuracy of 64-slice computed tomography coronary angiography: a prospective, multicenter, multivendor study. J Am Coll Cardiol. 2008;52(25):2135–44.

    Article  Google Scholar 

  5. Hamon M, Biondi-Zoccai GG, Malagutti P, Agostoni P, Morello R, Valgimigli M, et al. Diagnostic performance of multislice spiral computed tomography of coronary arteries as compared with conventional invasive coronary angiography: a meta-analysis. J Am Coll Cardiol. 2006;48(9):1896–910.

    Article  Google Scholar 

  6. Park J, Kim HK, Park EA, Park JB, Lee SP, Lee W, et al. Coronary computed tomography angiography for the diagnosis of vasospastic angina: comparison with invasive coronary angiography and ergonovine provocation test. Korean J Radiol. 2019;20(5):719–28.

    Article  Google Scholar 

  7. Pugliese F, Hunink MG, Gruszczynska K, Alberghina F, Malago R, van Pelt N, et al. Learning curve for coronary CT angiography: what constitutes sufficient training? Radiology. 2009;251(2):359–68.

    Article  Google Scholar 

  8. Saur SC, Alkadhi H, Stolzmann P, Baumuller S, Leschka S, Scheffel H, et al. Effect of reader experience on variability, evaluation time and accuracy of coronary plaque detection with computed tomography coronary angiography. Eur Radiol. 2010;20(7):1599–606.

    Article  Google Scholar 

  9. Rueckel J, Kunz WG, Hoppe BF, Patzig M, Notohamiprodjo M, Meinel FG, et al. Artificial intelligence algorithm detecting lung infection in supine chest radiographs of critically ill patients with a diagnostic accuracy similar to board-certified radiologists. Crit Care Med. 2020;48:e574–83.

    PubMed  Google Scholar 

  10. Jiang Y, Edwards AV, Newstead GM. Artificial intelligence applied to breast MRI for improved diagnosis. Radiology. 2021;298(1):38–46.

    Article  Google Scholar 

  11. Rauschecker AM, Rudie JD, Xie L, Wang J, Duong MT, Botzolakis EJ, et al. Artificial intelligence system approaching neuroradiologist-level differential diagnosis accuracy at brain MRI. Radiology. 2020;295(3):626–37.

    Article  Google Scholar 

  12. Liu F, Zhou Z, Samsonov A, Blankenbaker D, Larison W, Kanarek A, et al. Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. Radiology. 2018;289(1):160–9.

    Article  Google Scholar 

  13. Thilo C, Gebregziabher M, Meinel F, Goldenberg R, Nance J, Arnoldi E, et al. Computer-aided stenosis detection at coronary CT angiography: effect on performance of readers with different experience levels. Eur Radiol. 2015;25(3):694–702.

    Article  Google Scholar 

  14. Lehman CD, Blume JD, DeMartini WB, Hylton NM, Herman B, Schnall MD. Accuracy and interpretation time of computer-aided detection among novice and experienced breast MRI readers. AJR Am J Roentgenol. 2013;200(6):W683-689.

    Article  Google Scholar 

  15. Freer TW, Ulissey MJ. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology. 2001;220(3):781–6.

    Article  CAS  Google Scholar 

  16. Baker ME, Bogoni L, Obuchowski NA, Dass C, Kendzierski RM, Remer EM, et al. Computer-aided detection of colorectal polyps: can it improve sensitivity of less-experienced readers? Preliminary findings. Radiology. 2007;245(1):140–9.

    Article  Google Scholar 

  17. Arnoldi E, Gebregziabher M, Schoepf U, Goldenberg R, Ramos-Duran L, Zwerner P, et al. Automated computer-aided stenosis detection at coronary CT angiography: initial experience. Eur Radiol. 2010;20(5):1160–7.

    Article  Google Scholar 

  18. Siegersma KR, Leiner T, Chew DP, Appelman Y, Hofstra L, Verjans JW. Artificial intelligence in cardiovascular imaging: state of the art and implications for the imaging cardiologist. Neth Heart J. 2019;27(9):403–13.

    Article  CAS  Google Scholar 

  19. Vavere AL, Arbab-Zadeh A, Rochitte CE, Dewey M, Niinuma H, Gottlieb I, et al. Coronary artery stenoses: accuracy of 64-detector row CT angiography in segments with mild, moderate, or severe calcification—a subanalysis of the CORE-64 trial. Radiology. 2011;261(1):100–8.

    Article  Google Scholar 

  20. Maroules CD, Cheezum MK, Joshi PH, Williams M, Simprini LA, Nelson KH, et al. SCCT curriculum guidelines for general (level 1) cardiovascular CT training. J Cardiovasc Comput Tomogr. 2015;9(2):81–8.

    Article  Google Scholar 

  21. Choi AD, Thomas DM, Lee J, Abbara S, Cury RC, Leipsic JA, et al. 2020 SCCT guideline for training cardiology and radiology trainees as independent practitioners (level II) and advanced practitioners (level III) in cardiovascular computed tomography: a statement from the Society of Cardiovascular Computed Tomography. J Cardiovasc Comput Tomogr. 2020.

  22. Patel MR, Calhoon JH, Dehmer GJ, Grantham JA, Maddox TM, Maron DJ, et al. ACC/AATS/AHA/ASE/ASNC/SCAI/SCCT/STS 2017 appropriate use criteria for coronary revascularization in patients with stable ischemic heart disease: a report of the American College of Cardiology Appropriate Use Criteria Task Force, American Association for Thoracic Surgery, American Heart Association, American Society of Echocardiography, American Society of Nuclear Cardiology, Society for Cardiovascular Angiography and Interventions, Society of Cardiovascular Computed Tomography, and Society of Thoracic Surgeons. J Am Coll Cardiol. 2017;69(17):2212–41.

    Article  Google Scholar 

  23. Han D, Liu J, Sun Z, Cui Y, He Y, Yang Z. Deep learning analysis in coronary computed tomographic angiography imaging for the assessment of patients with coronary artery stenosis. Comput Methods Programs Biomed. 2020;196:105651.

    Article  Google Scholar 

  24. Zhang C-J, Xia D, Zheng C, Wei J, Cui Y, Qu Y, et al. Automatic identification of coronary arteries in coronary computed tomographic angiography. IEEE Access. 2020;8:65566–72.

    Article  Google Scholar 

  25. Leipsic J, Abbara S, Achenbach S, Cury R, Earls JP, Mancini GBJ, et al. SCCT guidelines for the interpretation and reporting of coronary CT angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee. J Cardiovasc Comput Tomogr. 2014;8(5):342–58.

    Article  Google Scholar 

  26. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.

    Article  CAS  Google Scholar 

  27. Ma LY, Chen WW, Gao RL, Liu LS, Zhu ML, Wang YJ, et al. China cardiovascular diseases report 2018: an updated summary. J Geriatr Cardiol. 2020;17(1):1–8.

    PubMed  PubMed Central  Google Scholar 

  28. Zhou M, Wang H, Zeng X, Yin P, Zhu J, Chen W, et al. Mortality, morbidity, and risk factors in China and its provinces, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2019;394(10204):1145–58.

    Article  Google Scholar 

  29. Halpern EJ, Halpern DJ. Diagnosis of coronary stenosis with CT angiography comparison of automated computer diagnosis with expert readings. Acad Radiol. 2011;18(3):324–33.

    Article  Google Scholar 

  30. Herzog C, Kerl JM, De Rosa S, Tekin T, Boehme E, Liem S, et al. Influence of observer experience and training on proficiency in coronary CT angiography interpretation. Eur J Radiol. 2013;82(8):1240–7.

    Article  Google Scholar 

  31. Zhang H, Mu L, Hu S, Nallamothu BK, Lansky AJ, Xu B, et al. Comparison of physician visual assessment with quantitative coronary angiography in assessment of stenosis severity in China. JAMA Intern Med. 2018;178(2):239–47.

    Article  Google Scholar 

  32. Raff GL, Gallagher MJ, O’Neill WW, Goldstein JA. Diagnostic accuracy of noninvasive coronary angiography using 64-slice spiral computed tomography. J Am Coll Cardiol. 2005;46(3):552–7.

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge all members of the CCTA and ICA image interpretation and data collation groups; the staff of Beijing Friendship Hospital, Capital Medical University and Beijing Anzhen Hospital; Capital Medical University: Fang Li, Shiyu Zhang, Tian Ma, Zhaozhao Wang, Rujing Sun, Jianghong Chen, Xuechun Guan, Yu Liang, Xiaofeng Qu, Zhenxing Yang, Wenming Zhang, Jinfan Tian, Jianfeng Zheng, and Zichao Cheng; and the staff of other hospitals: Ye Xiang (Leshan Traditional Chinese Medicine Hospital), Yuanyi Zhu (Armed Police Sichuan Provincial Corps Hospital), Jincheng Duan (The First People's Hospital of Zhaotong City), and Ting Mo (Tanzhong People's Hospital of Liuzhou City).

Funding

This work was supported by the National Key Research and Development Program of China [Grant Number 2019YFE0107800], the Beijing Municipal Science and Technology Commission [Grant Number Z201100005620009] and the Beijing Hospitals Authority Clinical Medicine Development of Special Funding Support [Grant Number ZYLX202101].

Author information

Authors and Affiliations

  1. Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 YongAn Road, Beijing, 100050, People’s Republic of China

    Xianjun Han, Nan Luo, Lixue Xu, Jiaxin Cao, Yi He, Zhenchang Wang & Zhenghan Yang

  2. Shukun (Beijing) Technology Co., Ltd., Jinhui Bd, Qiyang Rd, Beijing, 100102, People’s Republic of China

    Ning Guo

  3. Department of Computer Software Engineering, Soonchunhyang University, Asan, South Korea

    Min Hong

  4. Beijing University of Technology, Beijing, People’s Republic of China

    Xibin Jia

Authors
  1. Xianjun Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lixue Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiaxin Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ning Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yi He
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Min Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xibin Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Zhenchang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Zhenghan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

XH contributed to the design, analysis and interpretation of the patient CCTA and ICA data and drafted and critically revised the manuscript. NL, LX and JC contributed to the conception, design, and collation of patient clinical information. YH, ZY and ZW contributed to the study design and supervised manuscript writing. NG, MH and XJ contributed to the statistical analysis and provided technical and AI software support. All authors commented on all drafts of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Zhenghan Yang.

Ethics declarations

Ethics approval and consent to participate

The Ethics Committee of Beijing Friendship Hospital, Capital Medical University approval was obtained (Central Office for Research Ethics Committee Reference 2020-P2-010-02). The need for written informed consent was waived by the Ethics Committee. The experiment was performed in accordance and with the Helsinki Declaration of 1964 and later versions.

Consent for publication

Not applicable, as no identifiable participant data, pictures or illustrations that require consent for publishing are included in this manuscript.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Deep-learning model (AI system).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, X., Luo, N., Xu, L. et al. Artificial intelligence stenosis diagnosis in coronary CTA: effect on the performance and consistency of readers with less cardiovascular experience. BMC Med Imaging 22, 28 (2022). https://doi.org/10.1186/s12880-022-00756-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12880-022-00756-y

Keywords

BMC Medical Imaging

ISSN: 1471-2342

Contact us