nnU-Net based segmentation and 3D reconstruction of uterine fibroids with MRI images for HIFU surgery planning

Abstract

High-Intensity Focused Ultrasound (HIFU) ablation represents a rapidly advancing non-invasive treatment modality that has achieved considerable success in addressing uterine fibroids, which constitute over 50% of benign gynecological tumors. Preoperative Magnetic Resonance Imaging (MRI) plays a pivotal role in the planning and guidance of HIFU surgery for uterine fibroids, wherein the segmentation of tumors holds critical significance. The segmentation process was previously manually executed by medical experts, entailing a time-consuming and labor-intensive procedure heavily reliant on clinical expertise. This study introduced deep learning-based nnU-Net models, offering a cost-effective approach for their application in the segmentation of uterine fibroids utilizing preoperative MRI images. Furthermore, 3D reconstruction of the segmented targets was implemented to guide HIFU surgery. The evaluation of segmentation and 3D reconstruction performance was conducted with a focus on enhancing the safety and effectiveness of HIFU surgery. Results demonstrated the nnU-Net’s commendable performance in the segmentation of uterine fibroids and their surrounding organs. Specifically, 3D nnU-Net achieved Dice Similarity Coefficients (DSC) of 92.55% for the uterus, 95.63% for fibroids, 92.69% for the spine, 89.63% for the endometrium, 97.75% for the bladder, and 90.45% for the urethral orifice. Compared to other state-of-the-art methods such as HIFUNet, U-Net, R2U-Net, ConvUNeXt and 2D nnU-Net, 3D nnU-Net demonstrated significantly higher DSC values, highlighting its superior accuracy and robustness. In conclusion, the efficacy of the 3D nnU-Net model for automated segmentation of the uterus and its surrounding organs was robustly validated. When integrated with intra-operative ultrasound imaging, this segmentation method and 3D reconstruction hold substantial potential to enhance the safety and efficiency of HIFU surgery in the clinical treatment of uterine fibroids.

Introduction

Uterine fibroids, constituting approximately 52% of benign gynecological tumors, are benign tumors originating from smooth muscle tissue and can impact up to 70–80% of women [1,2,3]. Treatment modalities for uterine fibroids encompass invasive approaches such as abdominal myomectomy [4], laparoscopy [5], hysteroscopy [6], uterine artery embolization (UAE) [7], microwave (radiofrequency) ablation [8], high-intensity focused ultrasound (HIFU) ablation [9], and other micro-non-invasive interventional therapies. HIFU, guided by ultrasound or MRI, employs high-intensity ultrasound to focus on the interior of uterine fibroids in vivo, creating a focal point with high energy density. This leads to a transient temperature rise above 60 degrees in the focal area, inducing thermal solidification and necrosis of tissues, while minimizing damage to tissues outside the focal region. Ensuring safety and efficacy is crucial in HIFU surgery for treating uterine masses [10].

MRI serves as a non-invasive and precise modality employed by medical practitioners in the diagnosis of uterine fibroids and guidance of High-Intensity Focused Ultrasound (HIFU) procedures. However, uterine fibroids manifest diverse morphological and size variations in magnetic resonance images. Therefore, the meticulous formulation of personalized HIFU surgical treatment plans for each patient is imperative. The segmentation of uterine fibroids, involving the precise measurement of the uterine and fibroid states on MRI images, encompassing volume, shape, and spatial location, constitutes the primary step in the HIFU treatment plan. Nevertheless, this constitutes a laborious and time-intensive undertaking, susceptible to variances arising from intra-expert and inter-expert differences, both pre- and post-treatment. Consequently, there is a pressing need for an automated and accurate segmentation methodology for uterine fibroids in practical applications. However, designing a precise uterine fibroid segmentation system remains a persistently challenging issue, yet to be resolved.

A meticulous approach was employed in selecting the segmentation labels, including the uterus, uterine fibroids, urethral orifice, bladder, sacrococcygeal bone, symphysis pubis, and endometrium. This approach aimed to accurately segment the tissues and organs within the uterine region, allowing for a comprehensive analysis of their interrelationships. An efficient assessment of the therapeutic range and effectiveness of HIFU was achieved, taking into account all pertinent factors in a thorough and unbiased manner. However, significant variations in the size, shape, and appearance of uterine fibroids present considerable challenges. These tumors’ irregular and complex shapes make segmentation difficult [11]. Additionally, due to varying degrees of contrast agent uptake or intrinsic tissue properties, the intensity levels of tumors in MRI images also vary. These factors increase imaging variability, necessitating advanced segmentation techniques to accurately delineate tumors [12]. Furthermore, MRI images often contain noise and artifacts, such as motion artifacts caused by patient movement or streak artifacts, which can obscure tumor boundaries [13]. These issues pose significant challenges for accurate segmentation, requiring robust algorithms to mitigate the impact of noise and enhance image clarity.

Therefore, the precise delineation of uterine fibroids and associated tissues is crucial in planning HIFU treatments, as it has a direct bearing on the ultimate success and surgical quality. The task of segmenting uterine fibroids poses significant challenges due to several factors. Firstly, the inherent anisotropy of the uterus and fibroids results in MRI images exhibiting a high degree of non-uniformity and irregular shapes. Consequently, the regions of interest exhibit considerable variations in both shape and size, as evident in Fig. 1. Secondly, the application of deep learning-based segmentation methods necessitates extensive annotated data. However, available data on uterine and fibroids is typically limited and complex, with inherent high heterogeneity leading to intra-class differences between the uterus and fibroids. These issues collectively impact the accuracy of uterine and fibroids segmentation. Furthermore, achieving rapid, precise, and reproducible segmentation of the uterine area remains a challenging task. Given these reasons, existing methods for uterine fibroid segmentation are often employed post-treatment, while pre-treatment segmentation is still performed manually by operators to mark the uterus, fibroids, and surrounding organs. Consequently, preoperative segmentation is essential to facilitate the development of effective treatment plans. Automatic segmentation techniques for uterine fibroids and associated tissues have the potential to overcome these challenges and provide precise outcomes.

Fig. 1

Exemplary uterus MR images

In recent years, the application of deep learning algorithms in medical image segmentation has demonstrated significant performance compared to previous methods. These algorithms have the potential to achieve fully automated segmentation tasks. Several researchers have developed tailored deep learning models for the automatic segmentation of uterine fibroids. The majority of these approaches have utilized the U-Net convolutional architecture and achieved promising outcomes, with reported Dice similarity coefficients (DSC) ranging from 80 to 86%. However, while the proposed method exhibits encouraging results, there may still be instances of boundary inaccuracies, particularly in patients with multiple fibroids.

Recently, the nnU-Net framework [14] has been proposed as an out-of-the-box tool for deep learning-based biomedical image segmentation, particularly in the context of uterine fibroid segmentation in routine MRI scans for computer-assisted HIFU surgery. This tool, publicly available, has demonstrated superior performance over existing models across 23 public datasets used in international biomedical segmentation competitions. Its accessibility and effectiveness make it potentially valuable for direct clinical applications, as it does not necessarily require expert knowledge to achieve competitive results. However, its performance specifically in uterine fibroid segmentation in clinical settings has yet to be fully assessed. In this study, our primary objective was to evaluate the performance of the nnU-Net framework for automatically segmenting uterine fibroids in routine MRI scans conducted for computer-assisted HIFU surgery. We utilized a dataset comprising 550 magnetic resonance images for this purpose. The nnU-Net model was trained to accurately segment uterine fibroids along with their surrounding tissues and organs. Performance evaluation of the model was conducted using the Dice similarity coefficient. The workflow encompassing patient selection, training, and evaluation processes is illustrated in Fig. 2. Furthermore, we compared the segmentation performance of the 3D nnU-Net with that of HIFUNET, with the comparison results presented in Table 2. In summary, this research employed a trained 3D nnU-Net model to achieve important advancements in the segmentation of uterine fibroids and surrounding organs using MRI images. The primary contributions of this work are as follows:

• (1) Construction of a high-quality uterine fibroid imaging dataset: A dataset comprising 550 MRI cases, meticulously annotated by professional physicians, providing a reliable data foundation for future research and clinical applications.

• (2) Achieved precise segmentation of uterine fibroids and their surrounding organs: The 3D nnU-Net model effectively and accurately segmented the uterus, fibroids, spine, endometrium, bladder, and urethral orifice in early examinations, with Dice Similarity Coefficients of 92.55%, 95.63%, 92.69%, 89.63%, 97.75%, and 90.45%, respectively, significantly outperforming existing methods.

• (3) Enhanced HIFU Surgery Planning Through 3D Reconstruction: By performing 3D reconstruction to the segmented targets, detailed spatial information was provided to guide HIFU surgery planning, enhancing the safety and efficacy of the procedure.

• (4) Demonstration of the Model’s Scalability and Potential for Clinical Integration: This study not only validated the effectiveness of the 3D nnU-Net in uterine fibroid segmentation but also highlighted its potential for real-time surgical guidance. The 3D nnU-Net model can be integrated into various medical management system as a decision support tool, improving segmentation accuracy, reproducibility, and operational efficiency in clinical settings.

Fig. 2

Data flow of patient selection, training, and evaluation process

The structure of this paper is as follows: In Section II, we provide an overview of related work. Section III introduces our proposed solution. Our experiments, along with performance comparisons with conventional and other deep learning methods, are detailed in Section IV. In section V, we draw some conclusions and perspectives.

The paper is structured as follows: In Section II, we provide an overview of recent related work. Section III introduces our proposed solution. Our experiments, along with performance comparisons with conventional and other deep learning methods, are detailed in Section IV. In Section V, we present conclusions and discuss future perspectives.

Related work

Advancements in medical machine learning techniques

In recent years, significant advancements have been made in the application of deep learning in the field of computer-aided diagnosis, particularly in medical imaging, biomedical signal processing, and drug discovery. For example, in the study of drug permeability, machine learning models have shown outstanding performance. For instance, Chandrasekar et al. [15] investigated various machine learning models for predicting drug permeability across the placental barrier, finding that KNN, SVC, and multilayer perceptron achieved prediction accuracies of 82%, 86.4%, and 90.8%, respectively. Ansari et al. [16] conducted a comprehensive analysis of machine learning models for predicting blood-brain barrier (BBB) permeability, revealing that random forest and extra trees models, using mol2vec fingerprints, achieved AUC_ROC values of 0.9453 and 0.9601, respectively. Al-Kababji et al. [17] have summarized key studies from recent years, showcasing the application of various machine learning algorithms in the segmentation of liver tissues, hepatic tumors, and hepatic vasculature structures. These algorithms include both supervised and unsupervised learning methods, reflecting the challenges posed by different segmentation tasks.

In the realm of biomedical signal processing, deep learning has similarly demonstrated excellent performance. Ansari et al. [18] reviewed methods for estimating age and gender from electrocardiogram (ECG) signals over the past decade, highlighting that elevated ECG-estimated age is associated with atherosclerotic cardiovascular disease, peripheral endothelial dysfunction, and high mortality rates. Additionally, deep learning has been applied to dietary recognition systems. Ansari et al. [19] developed a large-scale dataset of everyday foods from the Middle East to advance food recognition systems, showing that EfficientNet-V2 performed exceptionally well on this dataset while maintaining minimal resource utilization.

Noise handling is a critical issue in medical image processing, as noise can degrade image quality and negatively impact subsequent image analysis. Common denoising techniques include spatial domain denoising, frequency domain denoising, and deep learning-based denoising methods. For instance, Mohanty et al. proposed an innovative non-rigid image registration method that utilizes a diffeomorphism-based framework to address image registration between different modalities [20]. This method employs a non-stationary velocity field to minimize the impact of forces derived from image gradients and uses a similarity energy function based on gray-scale distribution to limit fluctuations and prevent local minima. The presence of noise not only affects registration accuracy but also has a significant impact on the segmentation process. In the realm of image segmentation, Regaya et al. introduced a method combining multi-resolution and statistical approaches for cerebral aneurysm segmentation [21]. This method uses Contourlet Transform in the 2D domain to extract image features and employs Hidden Markov Random Field with Expectation Maximization for segmentation. Their approach demonstrated high segmentation accuracy on Three-Dimensional Rotational Angiography datasets, underscoring its effectiveness in handling noise and enhancing segmentation quality. These advancements have significantly improved the quality and efficiency of medical image segmentation.

Hardware acceleration and parallel processing have become crucial for optimizing efficiency. Abbes et al. [22] demonstrated that using Zynq SoC to accelerate the Lattice Boltzmann Method (LBM) resulted in a 52-fold speed increase compared to a dual-core ARM processor. Mohanty et al. [23] enhanced LBM by developing a pipeline integrating blood flow simulation and real-time visualization, optimizing HemeLB with CUDA GPUs to process over 30 frames per second for 3D visualization. Additionally, Abbes et al. [24] ran HemeLB on heterogeneous system-on-chip platforms, showing that the Jetson TX1 implementation provided 19 times higher update rates than the Zynq SoC. These studies highlight that hardware acceleration and parallel processing significantly enhance the efficiency of medical image processing, offering new avenues for real-time clinical applications.

Moreover, other studies have addressed noise handling methods in medical image processing and segmentation. Han et al. [25] proposed an efficient model based on ConvNeXt, significantly reducing the number of parameters while performing well on various datasets. Ansari et al. [26] introduced a lightweight neural network (Res-PAC-UNet) for liver CT segmentation, which demonstrated excellent performance with a reduced number of parameters. Jafari et al. [27] combined the strengths of residual networks (ResNet) and densely connected networks (DenseNet) to propose an efficient network architecture that performed well on skin lesion segmentation and brain MRI datasets, with the code available on GitHub. Ansari et al. [28] also proposed an improved pyramid scene parsing (PSP) module for real-time liver ultrasound segmentation, achieving high Dice coefficients and real-time performance. Xie et al. [29] introduced a novel framework combining CNN and Transformer for 3D medical image segmentation, significantly improving multi-organ segmentation tasks, with the code also available on GitHub. Additionally, Ansari et al. [30] reviewed liver segmentation methods in clinical surgeries and interventions, emphasizing the importance of precision, accuracy, and automation. Akhtar et al. [31] assessed the risks of computer-aided diagnostic systems in hepatic resection, highlighting the potential of CAD systems in improving postoperative patient outcomes. Rai et al. [32] systematically reviewed the efficacy of fusion imaging for immediate post-ablation assessment of malignant liver neoplasms, underscoring its clinical potential. Finally, Ansari et al. [33] reviewed advancements in deep learning for B-mode ultrasound segmentation, analyzing the advantages and disadvantages of various neural network architectures.

Deep learning techniques for biomedical image processing

Recent years have seen the emergence of Convolutional Neural Networks (CNNs) as a groundbreaking addition to the field of Computer Vision [34], with deep CNNs demonstrating excellent results in the segmentation and classification of medical images [35]. Research has been shown that CNNs can be used to aid in the diagnosis of prostate cancer [36], and Brosch et al. [37] applied a deep three-dimensional convolutional network to perform multi-scale feature fusion and segment multiple sclerosis lesions. Ranneberger et al. [38] proposed the U-Net, which achieved good cell wall structure segmentation on the pathological tablet of He La cell suspension. In 2014, Long et al. [39] proposed the Fully Convolutional Network (FCN), which takes inputs of arbitrary size and produces correspondingly-sized outputs. FCNs have been widely successful and have been applied to dense prediction problems such as semantic segmentation, performing end-to-end image segmentation, particularly in encoder-decoder architectures.

U-Net, proposed by Ronneberger et al. [38], is one of the most renowned networks for medical image segmentation tasks [40]. It employs the same number of convolution operations as the FCN but features skip connections that link the encoder and decoder layers by copying and cropping feature maps from the encoder layer to the decoder layer. This makes U-Net the most advanced biomedical image segmentation method currently available. However, there is still has room for improvement. For instance, a deep residual U-Net network (ResUnet) for automatic myocardial segmentation was proposed in 2017 [41], while MultiResUNet [42] was designed to tackle multi-scale problems, and R2U-Net [43] was proposed to extract features of images using a residual structure multiple times. U-Net has been highly successful in a variety of medical applications, such as cardiac segmentation [44] and liver and tumor segmentation. nnU-Net [11] is a self-adapting CNN framework that enhances architecture for automated medical image segmentation, which proves to be highly complementary to one another when used in model ensembles. The authors of nnU-Net argued that a simple U-Net, if tuned properly and combined with adapted data preprocessing techniques, can outperform more complex CNNs. This framework was entered into 19 international competitions, and set a new state-of-the-art in the majority of 49 tasks.

Uterine fibroids segmentation

We outline here the traditional methods of segmenting uterus and uterine fibroids that have been proposed so far, and review the state-of-the-art MR image segmentation methods based on the CNN architecture.

Traditional methods of uterus and uterine fibroid segmentation

Traditional methods are mainly based on level sets, FCM, region-growing and so on. Ben-Zadok et al. [45] proposed a two-step semiautomatic method based on the level set, followed by an interactive level set segmentation framework to allow user feedback. This is a semi-automatic method where the user selects the seed-points, and the precision level of segmentation is comparable to that of all manual expert segmentation. Khotanlou et al. [46]. proposed a two-stage method that combines the region-based level set with the hybrid Bresen methods. Yao et al. [47] proposed a semiautomatic approach based on the level set for robustly segmenting fibroids on MRI and accurately measuring 3D volume. This method is based on a cascade of fast-moving and Laplacian level sets, rather than a single level set. Fallahi et al. [48]. used the FCM method in T1 and T1-Enhanced image and some morphological operations to segment uterine fibroids. Then, on the basis of [48], Fallahi et al. [49]proposed a two-step method is proposed, where in the second step, an improved probability fuzzy C-means (MPFCM) is adopted [35]. Militello et al. [50]. proposed a new fully automatic method for uterine and fibroid segmentation based on unsupervised fuzzy C-means clustering and iterative optimal threshold selection algorithm. Additionally, Militello et al. [51]performed both quantitative and qualitative evaluation of HIFU therapy by providing 3D models of fibroid areas using a semi-automatic approach based on area growth. Rundo et al. [52] proposed a new region-growth based approach for myoma segmentation in MRgHIFU therapy. The first stage of this method is automatic seed region selection and region detection, and the second stage is for uterine fibroid segmentation. Antila et al. [53] designed an automatic segmentation pipeline without user input, applying the active contour mode (ASM) to obtain deformed surfaces, and classifying PV (amount of perfusion: untreated tissue) and NPV (amount of perfusion: treated tissue) using the expectation maximization (EM) algorithm. Guyou et al. [54] developed VETOT, which used active contours and a fast marching level set to segment the fibroids and track the volume of fibroids over time. However, as far as we know, no single method can handle all complex fibroids and obtain accurate segmentation. Sasidharan et al. [55]. proposed the watershed algorithm for automatic fibroids segmentation, applying some post-processing operations to solve the over-segmentation issues.

These methods can be performed only in homogeneous and connected boundary regions while the fibroids are a calcified and infarct region presents. Compared with traditional methods, deep learning method does not require manual labeling of features, saving more work and time. In addition, the traditional image segmentation methods mostly use the surface information of the image, making them unsuitable for tasks that require extensive semantic information. Most methods based on deep learning are end-to-end learning. The whole segmentation process can be divided into two parts: training and testing. Usually, a set of data with manual labels is input into the deep learning network for training to get the network model, and then the image data in the test set is input into the network model to get the segmentation result.

Deep learning methods for uterus and uterine fibroid segmentation

Kurata et al. [56] were the first to use U-net for automatic MRI uterine segmentation, and concluded that automatic hysterectomy modified by U-net was clinically feasible, with the model’s segmentation performance unaffected by uterine lesions. Similarly, TANG et al. [57] proposed a novel Attention Res Net101-unet (Attention Res Net101-unet) for MRI T2W image segmentation of uterine fibroids. This network uses deep neural network Res Net101 as the feature extraction front end to extract image semantic information, and constructs the network structure combined with U-net design idea. An attention module is added before the up-sampling and down-sampling feature graphs are joined. Later, Zhang et al. [58]used the full convolutional network (HIFUNet) to perform multiple segmentation of uterus, uterine fibroids and spine in MR images before HIFU surgery. This approach uses a large kernel to capture multiple spatial scale contexts by expanding the effective acceptance domain. In addition, a deep multi-atom convolution block is used to enlarge the receptive field and extract denser feature maps. Moreover, the performance of HIFUNet is compared with that of the six latest segmentation networks (U-net, HRNet, etc.). The results show that the accuracy and robustness of HIFUNet are significantly improved compared with other segmentation networks, and the segmentation results are close to the level of radiologists.

However, these algorithms are proposed in 2D. Considering anatomical features of the data, it would be better if we use 3D algorithm. Due to the particularity of USgHIFU surgery, we need the segmentation of multiple tissue structures, so the above methods are not suitable for HIFU surgery of uterine fibroid.

Methods

Although many experts and scholars have proposed a variety of deep learning network structures, and have achieved good results in the field of medical image segmentation, few people have studied the segmentation of uterine fibroids.

Data

A database of 550 MR images from patients who underwent HIFU surgery was used in this study. We collected clinical data from Chongqing Haifu Hospital under the permission of the ethics committee and has no implication on patient treatment. Medical records of 550 MR images were retrospectively reviewed from January 2021 to July 2022. This group comprising 550 MR images from Chongqing Haifu Hospital, was used for training and validating.

The MRI dataset was collected retrospectively with a 1.5-T magnetic resonance scanner (Lianying, Shanghai, China) in Chongqing Haifu Hospital. The fat-suppressed T2-weighted MR images in the sagittal direction were used in this work. The MRIs were collected in DICOM format; these images of cases were used as the data samples for deep learning. The scan parameters and characteristics of MR images are shown in Table 1. The parameters for MRI were field-of-view = 240 mm, slice thickness = 6 mm, matrix size = 480*480, echo time = 75.06ms, repetition time = 4729ms.

Table 1 The scan parameters and characteristics of MR datasets

Each MR volume consists of 25 slices of 480*480 pixels. The ground truth has been generated through a proper annotation process with the guidance of professional doctor who has 7 years of experience in HIFU surgery.

Data pre- and post-processing

Data collection, sorting and labeling are all done by professionals, and the processing process is shown in the Fig. 3. All MRI scans had been segmented prior to our study during patient treatment. Ground truth segmentations were used for diagnosis and computer-aided surgical planning. It includes semi-automatic segmentations, manually refined by a first operator before slice-by-slice verification for validation by a senior operator focusing mainly on the regions of interest: external surface of Uterus, Fibroids, Spine, Endometrium, Bladder, Urethral Orifice. This process was repeated until the segmentations are approved and certified for clinical use.

Fig. 3

Training datasets preparation

A brief description of the network architecture is shown in Fig. 4. Datasets optimization is shown in Fig. 5.

Fig. 4

Network architecture

Fig. 5

Datasets optimization

Step 1 is data format conversion, and Step 2 and 3 is to improve training efficiency.

• (1) The DICOM format data convert to NIFTI package–convert 2D data to 3D data (including spatial coordinates, patient direction and azimuth information) and apply 3D UNet network;

• (2) Write the data information corresponding to label_data and image_data into the JSON file, which is convenient for batch writing to memory for training;

• (3) Convert the corresponding data and labels into binary files according to the JSON file completed in 2 to improve training efficiency.

Base model

We opted for nnU-Net as our segmentation network due to its user-friendly nature and its ability to adapt effectively to various biomedical image datasets. The no-new-Net (nnU-Net) segmentation framework in the medical domain is directly derived from the original U-Net architecture (refer to Fig. 6), a widely utilized convolutional neural network (CNN) for biomedical segmentation tasks, capable of dynamically adjusting to the intricacies of any given dataset. Our initial training/validation dataset was utilized to automatically configure preprocessing steps, network architecture, training procedures, and post-processing techniques. nnU-Net facilitates automatic determination of key hyperparameters such as training batch size, image block size, and downsampling levels. Following a 5-fold cross-validation approach, 3D full-resolution U-Net training was conducted on the training/validation dataset. Subsequent analysis of the cross-validation outcomes identified instances where the model misclassified certain voxels as uterine fibroids in scans devoid of such fibroids. Consequently, we implemented additional post-processing steps for refining the uterine fibroid mask, notably by eliminating all components smaller than a threshold determined empirically. Ultimately, inferences were drawn from our test dataset(Fig. 2).

Fig. 6

The model design ideas based on expert knowledge and experimental modification

During the training phase, the parameters of the shared layers between mask prediction and object score prediction are initialized using a pre-trained network specifically trained for image categorization on the ImageNet dataset. Subsequently, fine-tuning is applied to this model to refine object proposals, thereby improving its performance in this aspect. Our selection of the nnU-Net architecture, comprising eight 3 × 3 convolutional layers (followed by ReLU nonlinearities) and five 2 × 2 max-pooling layers, has consistently demonstrated outstanding performance.

Given our focus on deriving segmentation masks, the spatial information embedded within the convolutional feature maps holds significance. Consequently, we opt to discard all final fully connected layers of the nnU-Net model, along with the last max-pooling layer. The output of the shared layers experiences a downsampling factor of 16 due to the presence of four remaining 2 × 2 max-pooling layers. For an input image of dimensions 3 × h × w, the resulting feature map dimensions are 480 × h/16 × w/16.

In this paper, we choose the uterus, uterine fibroids, urethral orifice, bladder, sacrococcygeal bone, symphysis pubis and endometrium as the segmentation targets. Our final model architecture is illustrated in Fig. 7.

Fig. 7

The design ideas of nnU-Net

In preliminary experiments, we used the open-source PyTorch implementation of nnU-Net and ran it directly on our development dataset, which provided us with preliminary performance targets, as well as a recipe for the architecture, training procedure, and data preprocessing guidelines to best optimize U-Net for this task. Based on these preliminary experiments, we proceeded to implement a custom model and training pipeline.

Experimental setup

• 1) Training and Testing Phase: 550 MR images were used for training and testing. This study was conducted in two phases to utilize deep learning networks for 3D semantic segmentation of uterus fibroid, then to evaluate the performance of the models trained with different datasets. The model was trained and evaluated with 550 MRIs [80% (440 cases) training cases and 20%(110cases) validation cases] cases in the validation dataset were not used for training. In phase two, with the newly acquired data, the performance of the model is tested. After optimization, the model is re-added to the training set and trained iteratively. (Fig. 6) The images were segmented and rated by the same surgeons who rated the first set of images.

• 2) Parameter Settings and Platform: To optimize our network, we employ the Adam optimizer with an initial learning rate set to 2e-4. After each epoch, if validation loss fails to decrease consecutively for three times, the learning rate is reduced to 1/5 of its current value until it reaches 5e-7. Consequently, the number of training epochs is dictated by the diminishing learning rate. We set the batch size to 8. Consistency in updating hyperparameters is maintained across all comparative experiments. Additionally, in the ablation study, hyperparameters remain fixed while parts of the network are removed.

Our proposed network is using ResNet as the backbone network which is relatively suited for image segmentation. The implementation is carried out on the PyTorch platform. The training and testing bed are Windows Server 2016 with dual NVIDIA Tesla V100 GPUs (32GB x 2 memory) and CUDA 10.1.

Training details

While these studies may differ in implementation specifics and performance benchmarks, segmentation tasks commonly rely on the Dice Similarity Coefficient (DSC) as the primary metric for assessing approach effectiveness. To quantitatively evaluate the model’s performance, we compared predictions with ground truth masks across each of the 6 segmentation masks within our test set. The model underwent training utilizing a combination of Dice and Binary Cross Entropy (BCE) losses.

$$\:\text{Dice coefficient: }Dice = {\frac{2 | X Y |}{|X| + |Y|}}$$

(1)

$$\:\text{Dice Loss: }L_{Dice} = 1 - Dice$$

(2)

$$\:\text{Training Loss: }{L}_{training}=\sum_{i}\left({L}\left({p}_{i},{p}_{i}^{*}\right)+\sum_{j}{L}\left({S}_{i,j},{S}_{i,j}^{*}\right)\right)$$

(3)

Here $\:i$ is the index of a sampled window, $\:{p}_{i}$ is the predicted objectness score of the instance in this window, and $\:{p}_{i}^{*}$ is 1 if this window is a positive sample and 0 if a negative sample. $\:{S}_{i}$ is the assembled segment instance in this window, $\:{S}_{i}^{*}\:$is the ground truth segment instance, and$\:\:\:j\:$is the pixel index in the window. $\:\mathcal{L}$ is the logistic regression loss. We use the definition of positive/negative samples in [38], and the 256 sampled windows have a positive/negative sampling ratio of 1:1.

An initial learning rate of 1e-2 was used and was decayed with a polynomial schedule. A batch size of 4 was used. The Adam optimizer with Nesterov Momentum and gradient clipping was used. Augmentations included translation, rotation, left-right flipping, and gamma contrast. The model was trained for a maximum of 1000 epochs.

Evaluation Metrics

We assessed our model’s performance using various quantitative measures aimed at comprehensively evaluating and comparing segmentation performance with other methods. Spatial overlap was quantified using the Dice Similarity Coefficient (DSC). However, it’s important to note that DSC lacks a clear definition in cases where both compared volumes contain zero positive voxels, resulting in division by zero.

$$Precision\, = \,{{TP} \over {TP\, + \,FP}}$$

(4)

$$recall\, = \,{{TP} \over {TP\, + \,FN}}$$

(5)

$$DSC\, = \,2\, * \,{{precision\, * \,recall} \over {precision\, + \,recall}}$$

(6)

We use the DSC to compare the predicted segmentation results with the ground-truth manually labeled by HIFU surgeon.

Experiments

Results

As shown in Fig. 8, the first row displays the original MRI image. The second row visualizes the segmentation results of the uterus, fibroids, and spine using HIFUNet, where pink represents fibroids, red represents the uterus, and blue represents the spine. The third row shows the segmentation results using 3D nnU-Net for the uterus, fibroids, urethral orifice, bladder, sacrococcygeal bone, symphysis pubis, and endometrium. In these images, red representing fibroids, light blue representing the uterus, dark blue representing the urethral orifice, green representing the bladder, yellow representing the sacrococcygeal bone and symphysis pubis, and pink representing the endometrium. From left to right, the images correspond to four different patients, labeled as Patient 1, Patient 2, Patient 3, and Patient 4.

Fig. 8

Comparison the segmentation results of HIFUNET and nnU-Net

For Patient 1, HIFUNet roughly segments the uterus and fibroids, but the boundaries near the bladder are blurred. In contrast, nnU-Net provides clearer segmentation in the same area, accurately distinguishing the uterus and fibroids while also clearly differentiating the bladder and sacrococcygeal bone. For Patient 2, HIFUNet struggles with the complex morphology of the fibroids, especially at the junction of the fibroids and the uterus, resulting in unclear boundaries. In comparison, nnU-Net delivers more detailed segmentation results, excelling in the precise delineation of fibroid boundaries and uterine contours. For Patient 3, HIFUNet shows over-segmentation in some areas when dealing with fibroids near the spine. 3D nnU-Net, however, accurately segments the spine, uterus, and fibroids in the same region, clearly displaying the positions of other related organs. For Patient 4, HIFUNet performs poorly when handling multiple small fibroids, often missing some. 3D nnU-Net, on the other hand, accurately identifies and segments multiple small fibroids while maintaining clear boundaries for all organs.

In summary, HIFUNet performs well in segmenting the uterus and fibroids but struggles with complex structures such as fibroids near the spine and bladder, resulting in blurred boundaries. This issue arises because HIFUNet employs large convolution kernels and multi-atom convolution blocks to capture multi-scale spatial contexts, which can sometimes lead to over-segmentation in complex regions. In contrast, 3D nnU-Net dynamically adjusts its preprocessing steps, network architecture, training procedures, and post-processing techniques to adapt to specific datasets, achieving more accurate and robust segmentation across various anatomical structures. 3D nnU-Net not only accurately segments the uterus and fibroids but also clearly distinguishes other important organs such as the urethral orifice and bladder.

After segmentation, the final processing step is to reconstruction, applying volume-rendering techniques, of the 3D model of ablated fibroid areas. Figure 9 shows the 3D model of the spine, fibroids, Endometrium, Bladder, Urethral Orifice, Symphysis Pubis of a patient. This can be displayed in front of the doctor in three dimensions, increasing the doctor’s three-dimensional sense. All summary quantitative results reported in the experimental results demonstrate that our conditional 3D nnU-Net achieves promising performance for multiorgan segmentation on the union of partially labeled datasets.

Fig. 9

3D model reconstructed from predicted segmentation masks (Spine, Fibroids, Endometrium, Bladder, Urethral Orifice, Symphysis Pubis)

The mean results of DSC for each segmentation label of our test set are shown in Table 2.

Table 2 Quantitative comparison of DSC of different segmentation methos on testing dataset

Table 2 presents the quantitative comparison of the Dice Similarity Coefficient (DSC) for different segmentation methods on the testing dataset across various anatomical structures. The 3D nnU-Net shows significantly higher segmentation performance compared to other methods in all structures.

Specifically: Uterus: 3D nnU-Net achieves a DSC of 92.55%, compared to 87.99% for ConvUNeXt, 82.37% for HIFUNet, 84.46% for U-Net, 86.82% for R2U-Net, and 88.63% for 2D nnU-Net. The 3D nnU-Net shows the highest accuracy in uterus segmentation, outperforming the other methods by 4.56%, 10.18%, 8.09%, 5.73%, and 3.92%, respectively. Fibroids: 3D nnU-Net reaches a DSC of 95.63%, compared to 92.49% for ConvUNeXt, 83.51% for HIFUNet, 87.01% for U-Net, 89.15% for R2U-Net, and 92.05% for 2D nnU-Net. In fibroid segmentation, the 3D nnU-Net outperforms the other methods by 3.14%, 12.12%, 8.62%, 6.48%, and 3.58%, respectively. Spine: 3D nnU-Net achieves a DSC of 92.69%, compared to 90.76% for ConvUNeXt, 85.01% for HIFUNet, 89.98% for U-Net, 89.26% for R2U-Net, and 90.98% for 2D nnU-Net. The 3D nnU-Net outperforms the other methods in spine segmentation by 1.93%, 7.68%, 2.71%, 3.43%, and 1.71%, respectively. Endometrium: 3D nnU-Net reaches a DSC of 89.63%, compared to 88.88% for ConvUNeXt, 86.35% for U-Net, 87.65% for R2U-Net, and 87.32% for 2D nnU-Net. In endometrium segmentation, the 3D nnU-Net outperforms the other methods by 0.75%, 3.28%, 1.98%, and 2.31%, respectively. Bladder: 3D nnU-Net achieves a DSC of 97.75%, compared to 95.65% for ConvUNeXt, 92.17% for U-Net, 93.89% for R2U-Net, and 95.34% for 2D nnU-Net. In bladder segmentation, the 3D nnU-Net outperforms the other methods by 2.10%, 5.58%, 3.86%, and 2.41%, respectively. Urethral Orifice: 3D nnU-Net reaches a DSC of 90.45%, compared to 88.27% for ConvUNeXt, 85.87% for U-Net, 86.29% for R2U-Net, and 88.13% for 2D nnU-Net. In urethral orifice segmentation, the 3D nnU-Net outperforms the other methods by 2.18%, 4.58%, 4.16%, and 2.32%, respectively.

To further illustrate the effectiveness of the 3D nnU-Net algorithm, an in-depth analysis of precision and recall was conducted, as shown in Table 3.

Table 3 Precision and recall values for various segmentation methods

Fig. 10

Segmentation examples on new MRI data with nnU-Net model. We show segmentations with different colors as before; red denotes the fibroids, blue denotes the uterus, blue denotes the urethral orifice, green denotes the bladder, yellow denotes the sacrococcygeal bone and symphysis pubis, pink denotes the endometrium

From Table 3, it is evident that the 3D nnU-Net not only outperforms in terms of DSC but also in precision and recall. For example, in uterus segmentation, the 3D nnU-Net achieves a precision of 93.98% and a recall of 92.11%, significantly higher than the other models. Similar trends are observed for fibroids, spine, endometrium, bladder, and urethral orifice segmentation, where the 3D nnU-Net consistently shows superior performance.

Overall, the 3D nnU-Net significantly outperforms HIFUNet, U-Net, R2U-Net, ConvUNeXt, and 2D nnU-Net in all segmentation tasks. These results indicate that 3D nnU-Net has a significant advantage in segmenting uterine fibroids and surrounding organs, demonstrating high accuracy and robustness in complex anatomical structures.

More results

To further validate the effectiveness of our method, we perform multiple segmentation validation on recent hospital MRI data. We performed segmentation verification on the recent data and accumulated 60 cases of data, and the results showed that the segmentation results met the expected results. Segmentation examples on new MRI data with 3D nnU-Net model show in Fig. 10.

Discussion and conclusion

The main goal of this study was to evaluate the performance of the nnU-Net framework for automatic segmentation of uterine fibroids and their surrounding organs based on MRI images for the planning of HIFU surgeries. We chose this deep learning framework because it has the ability to automatically configure itself without manual intervention. It has delivered state-of-the-art segmentation results on a large diversity of biomedical datasets, surpassing most highly specialized algorithms. We compared our method with the HIFUNET, and both models were implemented using the same datasets. It has been validated that the proposed deep learning model nnU-Net is able to automatically segment uterine fibroids and their surrounding organs on MRI images, with better consistent and more accurate results than the HIFUNET.

Different colors were used to denote different classes (red denotes the uterine fibroids, blue the uterus and green the bladder, purple the urethral orifice, yellow the sacrococcygeal bone, green the symphysis pubis and pink the endometrium). The images show that our method provided more accurate results. One drawback of the proposed method is that it requires high computational resources and it takes quite long to do the segmentation. However, in clinical applications, the accuracy of the segmentation is much more important than the computation cost. We found it acceptable that the increases in computational costs are negligible for the improvement in accuracy. The computational cost of our method at test time can be borne by a standard GPU. Reliably segmenting uterine fibroids and THE related tissues is the cornerstone of HIFU treatment planning.

This model can be integrated into the current USgHIFU management software as a medical decision support system. The proposed solution can provide effective support to physicians or radiologists to improve the accuracy, reproducibility, and execution time of the healthcare services where the segmentation is currently required.

Finally, the clinical value of DSC metrics in the context of small object segmentation was demonstrated by our segmentation results. In the future, we will optimize the current framework by designing a more efficient adaptive loss function for multi-target segmentation and introducing an attention mechanism or other strategies.