 Review
 Open access
 Published:
A review on deep learning MRI reconstruction without fully sampled kspace
BMC Medical Imaging volumeÂ 21, ArticleÂ number:Â 195 (2021)
Abstract
Background
Magnetic resonance imaging (MRI) is an effective auxiliary diagnostic method in clinical medicine, but it has always suffered from the problem of long acquisition time. Compressed sensing and parallel imaging are two common techniques to accelerate MRI reconstruction. Recently, deep learning provides a new direction for MRI, while most of them require a large number of data pairs for training. However, there are many scenarios where fully sampled kspace data cannot be obtained, which will seriously hinder the application of supervised learning. Therefore, deep learning without fully sampled data is indispensable.
Main text
In this review, we first introduce the forward model of MRI as a classic inverse problem, and briefly discuss the connection of traditional iterative methods to deep learning. Next, we will explain how to train reconstruction network without fully sampled data from the perspective of obtaining prior information.
Conclusion
Although the reviewed methods are used for MRI reconstruction, they can also be extended to other areas where groundtruth is not available. Furthermore, we may anticipate that the combination of traditional methods and deep learning will produce better reconstruction results.
Introduction
Magnetic resonance imaging (MRI) plays an important role in clinical medicine, and it can visualize human organs and tissues to help followup diagnosis. However, MRI has always faced the challenge of long scan time. Before the advent of deep learning, two common methods were used to accelerate MRI, one was compressed sensing (CS) utilizing image compressibility, and another was parallel imaging using redundant information between coils [1,2,3]. Although these methods have made certain achievements, it still faces the challenges of long iteration time and low acceleration rate.
Recently, deep learning has become a method for accelerating MRI. Compared with traditional methods, it not only improves the quality of reconstructed images but has the advantages of realtime imaging. The quality of images is measured comprehensively by the peak signaltonoise ratio (PSNR) and mean structure similarity index measure (MSSIM). Higher PSNR means less noise and Higher MSSIM entail better structure similarity with the ground truth. Meanwhile, realtime imaging is important for some clinic applications, for example, deep learning can achieve realtime adaptive magnetic resonance imaging (MRI)guided radiotherapy by achieving higher acceleration factors to reduce total delays [4] and provide a powerful diagnostic tool for dynamic assessment of wrist function [5]. However, most of them require a large amount of data to perform network learning in a supervised learning manner.
Traditional optimization methods and deep learning in a supervised manner have done a lot of work and related reviews can be found in [6, 7]. Due to physiological constraints such as organ motion or physical constraints such as signal decay, it is difficult, impractical and impossible to obtain fully sampled data. Some researchers try to utilize transfer learning to solve this problem [8], while they still require a small number of fully sampled data to adjust the pretrained network. Hence, how to perform network learning and image reconstruction in the absence of fully sampled data is an active research topic. Here, the main related methods in deep learningbased MRI reconstruction without fully sampled data are reviewed.
The remainder of this paper is organized as follows. First, we give a brief overview of the traditional reconstruction model, meanwhile, some reconstruction algorithms involved in the following review are roughly discussed. Then deep learning model for MRI reconstruction is illustrated in a supervised manner. Next, we review how to train a reconstruction network without fully sampled data from the perspective of obtaining prior information. Finally, we emphasize the necessity of deep learning reconstruction without fully sampled data and the current challenges and look forward to the future.
Traditional methods for MRI reconstruction
Reconstruction model
Reconstructing a highquality image from undersampling data is a typical inverse problem. A multicoil imaging model can be expressed as follows,
where \({\mathbf{x}} \in {\mathbb{C}}^{N}\) is the image to be reconstructed, \({\mathbf{y}} \in {\mathbb{C}}^{M}\) is the noisy measured data \(\left( {M < N} \right)\), \({{\varvec{\upeta}}}\) is the noise, and \({\mathbf{A}}\) denotes a measurement operator consisting of a sampling matrix \({\mathbf{U}} \in {\mathbb{R}}^{M \times N}\), Fourier transform operator \({\mathbf{F}}\) and the sensitivity map matrix \({\mathbf{S}}_{i}\) for the ith coil. A common reconstruction model is to add a regularization term to constrain its solution space,
where \({\mathbf{y}}  {\mathbf{Ax}}_{2}^{2}\) ensures consistency with the measured data, \(\Re ({\mathbf{x}})\) is a regularization item, and \(\lambda\) is a tradeoff between the data consistency and the regularization terms. In most cases, the difference lies in whether it is singlechannel reconstruction or multichannel [1,2,3] reconstruction. Meanwhile, multiple regularization items can be chosen such as 2D wavelet [9], total variation (TV) [10], dictionary [11], 3D wavelet [12], 3D kt sparse, 3D lowrank (kt SLR) [13]. Some methods are often used to iteratively solve the above optimization problems [6].
Sparsity or lowrankness constraints are often used as priors to reduce the artefacts of the reconstruction image when the acceleration rate is high. Lustig et al. [14, 15] firstly applied compressed sensing to MRI reconstruction and achieved reliable results. Afterwards, researchers found that the key to MRI reconstruction based on compressed sensing lies in the design of the sparse domain, which mainly includes preconstructed [9, 14,15,16,17,18] or adaptive [12, 19,20,21] basis and dictionary [11, 22, 23]. Besides, lowrankness methods are mainly used for dynamic and highdimensional imaging by exploring the relationship between multiple images [24]. The structured lowrankness of kspace is discovered and used for reconstruction [25]. Meanwhile, the lowrankness of the structured matrix is used to jointly reconstruct the image with other aspects, including transformdomain weighted kspace [26,27,28,29] and slowly varying image phases [30, 31]. Although good achievements have been achieved, traditional optimization reconstruction methods complete iterations with more time.
Optimization algorithm
Practical and effective optimization algorithms are essential. A large number of algorithms have been studied to solve various optimization problems. Deep learning networks and optimization iterative reconstruction algorithms still have a certain connection, thus some of them will be reviewed. Here, we only briefly introduce the algorithms that will be involved in the review, including variablesplitting with the quadratic penalty (VSQP) [32], proximal gradient descent (PGD) [33], iterative shrinkagethresholding algorithm (ISTA) [34], alternate directions method of multipliers (ADMM) [35].
VSQP
We use variablesplitting with the quadratic penalty (VSQP) for Eq.Â (2), the formulation is as follow,
where \({\mathbf{z}}^{i}\) is the auxiliary intermediate variable, \({\mathbf{x}}^{i}\) is the image to be reconstructed in the ith iteration, and \(\mu\),\(prox_{\Re } ( \cdot )\) is the secondary penalty parameter and proximity operator respectively. In a deep network, this algorithm can be unrolled for a fixed number of iterations as network architecture, Eq.Â (3) is mainly related to the choice of priors, and can be interpreted as a denoising operation [50], which is executed in the manner of a neural network. EquationÂ (4) depends on the selection of the forward model and corresponds to the dataconsistency (DC) layer in the network, which is usually solved by
where \(\cdot^{H}\) denotes conjugate transpose, Eq.Â (5) can be updated by using a conjugate gradient (CG) [36] to avoid the matrix inversion process and cope with multicoil reconstruction scenarios.
PGD
In VSQP, if the following formula is used to update \({\mathbf{x}}^{i}\) instead of Eq.Â (4), the proximal gradient descent (PGD) is formulated as.
where \(\rho\) is a gradient descent step size. Since the simplest gradient descent is used to update, it often requires more iterations to achieve better results.
ISTA
As discussed above, both VSQP and PGD use a network to directly learn the approximation mapping for Eq.Â (3). Here, an iterative shrinkagethresholding algorithm (ISTA) is used to guide the completion of Eq.Â (3). As well known, ISTA comes from solving \(l_{1}\) norm problems; however, magnetic resonance images tend to be sparse in a certain domain rather than selfsparse, hence there is no simple closedform solution. To explain in more detail, we use the following substitution in Eq.Â (2).
we can get the final solution by alternately iterating the following subproblem and equation Eq.Â (6).
where \({{\varvec{\Psi}}}\) represents a tight frame, \(\Gamma_{\kappa } (x)\) denotes shrinkage operator such that
when ISTA meets deep learning, a nonlinear transform operator \(\hbar ({\mathbf{x}})\) is used here instead of \({{\varvec{\Psi}}}\) and \(\widetilde{\hbar }({\mathbf{x}}^{i  1} )\) is an inverse operator of \(\hbar ({\mathbf{x}}^{i  1} )\), meanwhile, \(\widetilde{\hbar }({\mathbf{x}}^{i  1} )\) and \(\hbar ({\mathbf{x}}^{i  1} )\) are implemented with a neural network respectively, \(\kappa\) is a new parameter that includes \(\rho\). More details can be acquired in [37]. Moreover, when tight frame sparsity is enforced, Liu et al.[17] proposed aÂ projected iterative softthresholding algorithm (pFISTA) to address the problem that ISTA can not be directly applied to MRI reconstruction, meanwhile, Zhang et al. [38]proved the convergence of pFISTA applied to parallel imaging. Subsequently, Lu et al. [39] constructed pFISTASENSEResNet network based on pFISTA and achieved better results compared with traditional parallel imaging in terms of MSSIM and PSNR.
ADMM
For formula Eq.Â (2), we can make the following Augmented Lagrangian function by utilizing a new variable
where \({\mathbf{u}}\)Â andÂ \(\nu\) denote Lagrangian multiplier and penalty parameter, respectively. EquationÂ (10) can be solved by three alternate iteration subproblems. For simplicity, we do the following substitutions:
then the alternate iteration subproblem is as follows:
Alternate directions method of multipliers (ADMM) can be combined with TV [40] and dictionary learning [11] to complete MRI reconstruction together. When the ADMM algorithm is expanded into a network, different network versions can be constructed according to the learning situation of the network, for instance, image transformation has been replaced by the network for ADMMnetI [41], ADMMnetII [41] also learns data consistency except for image transformation.
Deep learning with fully sampled kspace data
In the past few years, deep learning has achieved outstanding performance in the medical field, including biological magnetic resonance spectroscopy [42,43,44,45,46] and accelerated MRI [36, 39, 47,48,49,50,51,52]. MRI reconstruction based on deep learning can be roughly divided into two categories, datadriven and modeldriven. The former uses the redundant information in the original input to learn the potential mapping relationship from input to output, including learning the mapping from zerofiled to artefactfree images [47] and the interpolation rules of kspace [49, 50].
For the sake of allowing the network to exploit the information of the imaging system, the researchers proposed physical modeldriven deep learning methods [36, 39, 51, 52]. The network uses a fixed number of iterations to unroll the traditional optimization iterative algorithm, which not only achieves better reconstruction results but makes the network more interpretable. When there are a large number of training sample pairs, the supervised reconstruction can be expressed as follows,
where \({\mathbf{x}}_{ref}^{i}\) is the reference image of ith subject, \({\text{L}} ( \cdot , \cdot )\) is the loss function between the network output image and the reference image, and N is the number of fully sampled datasets in the training database. Let \(f({\mathbf{y}}^{i} ,{\mathbf{A}}^{i} ,\theta )\) denotes output image of network for undersampled kspace data \({\mathbf{y}}^{i}\) and measurement operator \({\mathbf{A}}^{i}\) of ith subject, where the network is parameterized by \(\theta\). EquationÂ (13) can be implemented using stochastic gradient descent (SGD) [53], whose basic form is
where \(\rho^{k}\) is the gradient descent size. According to actual needs, it can change with the number of iterations, or be a constant. Due to the slow convergence of the basic SGD, researchers use other variants of SGD to speed up the convergence of the algorithm and avoid convergence to the saddle point [54,55,56]. Furthermore, unrolled network based on the traditional optimization algorithm is often used to improve the interpretability of the network and reconstruction quality [36, 39, 57, 58]. A comprehensive review of modeldriven MRI deep learning reconstruction can be found in [7].
Deep learning without fully sampled kspace data
As discussed above, a network in a supervised manner can learn maps to complement the missing information in the input from fully sampled data. However, for the scenario without fully sampled data, it is difficult to find the optimal solution from infinite latent solutions without other information. For traditional optimization algorithms, the regularization term is usually manually predefined to obtain the optimal solution by compressing the solution space. Hence, how to better discover effective prior information is very important for deep learning without fully sampled kspace data, the flowchart is shown in Fig.Â 1.
Next, we will show the MRI reconstruction process in a deep learning manner from the perspective of a prior acquisition.
Deep image prior
The experiment in [59] showed that only a generation network can still achieve good results in the absence of any other reference data, which illustrate that the convolutional neural network can replace the regularization term in Eq.Â (2) by capturing the lowlevel image implicit prior.
Yazdanpanah et al. [60] and Senouf et al. [61] apply this idea to MRI reconstruction, only use the zerofilled image as the input of the network, and then iteratively update the network parameters to approximate the kspace of the output image to the undersampled kspace data. This idea was further extended to dynamic MRI by Jin et al. [62]. However, there are also obvious shortcomings,Â sinceÂ it required welldesigned network architectures and was easy to overfit to noise with the iteration gradually approaches the target as the optimization function is based on lossy data. Hence the regularization term was introduced to alleviate this situation [63]. Meanwhile, many works illustrate that unroll network based on the physical model can improve the interpretability of the network and reconstruction quality, andÂ some of these traditional algorithms are briefly introduced in "Traditional methods for MRI reconstruction" section.
A. Wang et al. [64] proposed to combine a VQSPbased iterative network (as depicted in Fig.Â 2) with the high robustness of the classic iterative algorithm. The loss function adds a regularization term based on [60] as follows,
where \(I_{\theta } ({\mathbf{y}}_{{\mathbf{i}}} )\) is an output of the network, the \(\Re ( \cdot )\) is predefined such as TV. Although the addition of the \(\Re ( \cdot )\) can ameliorate the situation in [60] that is easy to overfit to noise, it will encourage the solution to approach the characteristics of the predefined regularization term. The experimental performance revealed that the model in [64] was more robust than the supervised learning without using the above loss function, and had a better reconstruction utility compared with the classical method using the same loss function.
Selfpartition kspace data
Yaman et al. [32] proposed to divide the measured undersampling space \(\Omega\) into two disjoint subsets satisfying \(\Omega = \Lambda \cup \Theta\) to train VSQP based unrolled reconstruction network, where \(\Lambda\) was used as the network input for training and \(\Theta\) was used to calculate the loss function, which could be called selfsupervised learning via data undersampling (SSDU). The experimental results showed that at a certain moderately acceleration rate the mentioned method can achieve comparable performance to supervised learning with fully sampled data and be better than traditional compressed sensing and parallel imaging. Since the undersampled data needs to be divided, the information provided to the network for learning is further reduced, which will result in a poor network reconstruction performance at a high acceleration time. Thereby, Yaman et al. [65] further proposed a multimask method increasing the use of undersampled data to improve the quality of reconstruction at higher acceleration rates. Here, the undersampling data was divided into multiple disjoint subsets \(\Omega = \Lambda_{j} \cup \Theta_{j}\) for \(j\)â€‰=â€‰1, â€¦, K denoting the number of partitions for each scan. We can visualize the process through Fig.Â 3. The result illustrated that the multimask method outperforms SSDU at high acceleration rates. Even so, since the essence that partition will decrease information to the network can not be changed, acceleration rates are still limited. Furthermore, Hosseini et al. [66] tried to finetune the pretrained reconstruction network in a scanspecific manner by using SSDU to reduce the risks of generalization to rare pathological conditions.
kspace Information complement each other
The lack of fully sampled data motivates to study how to use the information available in lossy images [67]. The key problem that we can not use supervised learning for network training is that we do not have missing information as a label in each sample.
Inspired by Noise2Noise (N2N) [67], the artefactcontaminated dataset obtained by sampling the same object multiple times will supplement the information missing in each sample, which enables the training of the imaging priors. However, sampling multiple times for the same object violates the original intention of accelerating MRI reconstruction and not requiring fully sampled data, hence it has a little restriction on application scenarios.
Different body parts and regions of interest have unique complications, such as the liver that needs to hold the breath for imaging and cardiac that produce motion. Thereby, specific deep learning models need to be adapted for specific tasks [68, 69]. Gan et al. [70] use the middle adjacent layers having the most relevant brain regions in each object from open dataset OASIS3 [71] to simulate multiple sampling of the same object. While the training data comes from the different breathing phases within the same slice of the liver in [72], which has obvious shortcomings for patients who do not have periodic breathing. Similarly, for organs such as cardiac that produce motion, Ke et al. [73] used a timeinterleaved acquisition scheme to build a series of fully encoded data as reference images for network training by merging the kspace of several adjacent frames along the time dimension. The remaining part will explain more details.
Gan et al. [70] proposed to train two networks simultaneously, one was used for reconstruction by utilizing information supplement between different samples, and another to register the image as the object may have moved in the actual scanning process. The experimental results showed that the method was superior to the unregistered Noise2Noise method and the traditional total variation (TV) method in terms of sharpness, contrast and deartefacts.
Additionally, experiments have shown that converting advanced denoising devices into regularization items can achieve good results [63, 74, 75], regularization by denoising (RED) [74] and the plugandplayprior (PnP) [76] are two common skills. Meanwhile, the deep network can be flexible to extract useful information from the data set compared with handcrafted prior. Hence Liu et al. [72] proposed to pretrain a deartefact network as imaging prior through information complement between undersampled data, the regularization function used the basic framework of RED, where the denoiser was replaced by the pretrain network \(I_{\theta } ({\mathbf{x}})\) in the following form.
where \(\tau\) is a regularization parameter. Bring Eq.Â (16) into Eq.Â (2), then iteratively update \({\mathbf{x}}\) by using the simplest gradient descent algorithm for Eq.Â (2). Due to the use of data fidelity information, compared to the results obtained directly through \(I_{\theta } ({\mathbf{x}})\), the proposed method reconstruction results on liver data were better and outperformed traditional iterative algorithms, such as kt SLR. Moreover, we can use more advanced traditional iterative algorithms to update \({\mathbf{x}}\) such as ADMM and so on.
In [73], Ke et al. averaged all acquired frames to improve the signaltonoise ratio (SNR) and relieve memory pressure. It should be noted that the merge operation only occurs in the training sample synthesis stage, and it is not required in the subsequent testing stage. Therefore, the reconstructed image will not result in a lower temporal resolution. Concretely, a network is established to learn the correlation between the coils instead of obtaining through ESPIRIT [77], the physical modelbased ADMMNetIII [41] was used as the reconstruction network. The method structure diagram is shown in Fig.Â 4. The experimental results showed that the reconstruction quality was better than conventional reconstruction methods, such as kt SLR [13], Lâ€‰+â€‰S [78], KLR [79], etc., and the reconstruction time was shorter [73]. Nevertheless, as the breathing pattern is inconsistent with different people, the generalization ability of the model will be affected.
Assuming known probability distribution as prior
Generative Adversarial Networks (GAN) [80] has shown strong advantages in unsupervised learning and can estimate the basic data distribution while obtaining higher image quality [81, 82]. In principle, the purpose of adversarial training is to approximate, in terms of distances, the probability distribution of the label set. Hence, there is no need for the ground truth corresponding to the input. Nevertheless, the real probability distribution between the generated output and the input of the discriminator will directly affect the utility of the final generator output as both of them approximating when the generator and discriminator obtaining equilibrium.
Cole et al. [83] proposed to build an ISTAbased GAN network for MRI reconstruction in the absence of fully sampled data. The network framework shown in Fig.Â 5 is based on the assumption that the randomly undersampled kspace \({\mathbf{y^{\prime}}}\) has the approximate distribution as the initially obtained measured kspace \({\mathbf{y}}\), where \({\mathbf{y}}\prime\) is obtained by performing forward measurement operation including random undersampled mask on the output of the generator. Hence, since the distribution \({\mathbf{A}}\) is known, the true underlying distribution of \({\mathbf{x}}\) can be uniquely determined \({\mathbf{y}}\prime\) [84].
Meanwhile, Wasserstein distance is continuous and differentiable compared with JensenShannon to serve as the measure of the distance between two probability distributions [85], and it is variation WGANGP which has proven to have the best convergence performance [86] was selected as the loss function of the network here. Ultimately, the experiment was carried out on dynamic contrastenhanced (DCE) data and knee data, and the results showed that the reconstruction quality has competed to supervised counterpart and better than CS.
Besides, experiments have proved that image style transfer tasks do not need groundtruth can be finished by unpaired training with only adversarial training [87], which is based on the assumption that there is a potential distribution relationship between unpaired samples and try to let the network learn this relationship. Hence, Lei et al. [88] suggested 2D images that are easier to obtain can be used as training labels for DCE images to train a PGDbased GAN network, Sim et al. incorporate the cycle consistent generative adversarial network (cycleGAN) [87] and forward physics in MRI using optimal driven theory to complete unpaired samples training [89]. For this type of method, the degree of joint probability distribution between unpaired samples will directly affect the experimental results. Thereby, the experimental results can be predicted to be no better than CS because the unpaired samples are completely disjoint in [88].
In addition to the mentioned methods, there are other ways to solve the problem of network training without fully sampled images. For example, traditional parallel compressed sensing imaging can be used as the true label for network training [90], but the final reconstruction quality of the network will not be significantly better than traditional parallel compressed sensing [83].
Conclusions
In summary, deep learning can complete related tasks by learning an implicit mapping relationship. Although the learning process cannot be explained in detail, many experimental results show that deep learning is effective and feasible. In normal conditions, the performance of supervised learning is better than that of network learning without labels. But in many scenarios, it is very difficult and infeasible to obtain labels, which seriously hinders the application of supervised learning, and also highlights the importance of deep learning without ground truth. Despite the reviewed methods are applied to MRI reconstruction, it can also be extended to other areas where it is difficult to obtain real data, such as dynamic positron emission tomography (PET) [91] or computed tomography (CT) [92].
While deep learning shows strong learning capabilities, it has also been criticized for its poor interpretability. Accordingly, the theoretical research of deep learning has become a new hot research direction. Researchers try to analyze the effective network structure from different angles to guide the construction of new networks, such as differential equations [93] and matrix decomposition [94].
Nevertheless, exploring the effective prior information in the lossy data and the inherent characteristics of the network is still a direction that needs to be studied. In addition, an appropriate sampling strategy for a specific body part is important for deep learning reconstruction performance. Typically, fast coronary imaging usually uses the spiral undersampling scheme [95] and some studies try to learn sampling strategy and reconstruction at the same time through the network [96].
Fortunately, inspired by the unfolding network based on the physical model. As traditional iterative algorithms and deep learning have their advantages, the former is computationally complex but has strong guidance significance, and the latter has the advantages of realtime imaging and powerful learning capabilities. We may anticipate that the deeper integration of traditional methods and deep learning, which will not only guide the construction of networks to bring more interpretability butÂ also more importantly, can obtain better results. Additionally, information complementarity between multicontrast images may be used as priors to participate in reconstruction.
Availability of data and materials
No datasets were generated or analyzed during the current study in this review.
Abbreviations
 ADMM:

Alternate directions method of multipliers
 CG:

Conjugate gradient
 CT:

Computed tomography
 CS:

Compressed sensing
 DC:

Data consistency
 DCE:

Dynamic contrastenhanced
 FFT:

Fast Fourier transform
 GAN:

Generative adversarial networks
 ISTA:

Iterative shrinkagethresholding algorithm
 MRI:

Magnetic resonance imaging
 MSSIM:

Mean structure similarity index measure
 N2N:

Noise2Noise
 PSNR:

Peak signaltonoise ratio
 PET:

Positron emission tomography
 pFISTA:

Projected iterative softthresholding algorithm
 PGD:

Proximal gradient descent
 PnP:

Plugandplayprior
 RED:

Regularization by denoising
 SGD:

Stochastic gradient descent
 SSDU:

Selfsupervised learning via data undersampling
 TV:

Total variation
 VSQP:

Variablesplitting with the quadratic penalty
References
Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sensitivity encoding for fast MRI. Magn Reson Med. 1999;42(5):952â€“62.
Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang J, Kiefer B, Haase A. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn Reson Med. 2002;47(6):1202â€“10.
Lustig M, Pauly JM. SPIRiT: Iterative selfconsistent parallel imaging reconstruction from arbitrary kspace. Magn Reson Med. 2010;64(2):457â€“71.
Terpstra ML, Maspero M, Dâ€™Agata F, Stemkens B, Intven MP, Lagendijk JJ, van den Berg CA, Tijssen RH. Deep learningbased image reconstruction and motion estimation from undersampled radial kspace for realtime MRIguided radiotherapy. Phys Med Biol. 2020;65(15):155015.
Radke KL, WollschlÃ¤ger LM, Nebelung S, Abrar DB, Schleich C, Boschheidgen M, Frenken M, Schock J, Klee D, Frahm J, Antoch G, Thelen S, Wittsack HJ, Lutz AM. Deep learningbased postprocessing of realtime MRI to assess and quantify dynamic wrist movement in health and disease. Diagnostics. 2021;11(6):1077.
Fessler JA. Optimization methods for MR image reconstruction (long version); 2019. https://arxiv.org/abs/1903.03510.
Zhang HM, Dong B. A review on deep learning in medical image reconstruction. J Oper Res Soc. 2020;8:311â€“40.
Dar SUH, Ã–zbey M, Ã‡atlÄ± AB, Ã‡ukur T. A transferlearning approach for accelerated MRI using deep neural networks. Magn Reson Med. 2020;84(2):663â€“85.
Qu X, Zhang W, Guo D, Cai C, Cai S, Chen Z. Iterative thresholding compressed sensing MRI based on contourlet transform. Inverse Probl Sci Eng. 2010;18(6):737â€“58.
Knoll F, Bredies K, Pock T, Stollberger R. Second order total generalized variation (TGV) for MRI. Magn Reson Med. 2011;65(2):480â€“91.
Zhan Z, Cai JF, Guo D, Liu Y, Chen Z, Qu X. Fast multiclass dictionaries learning with geometrical directions in MRI reconstruction. IEEE Trans Biomed Eng. 2016;63(9):1850â€“61.
Qu X, Hou Y, Lam F, Guo D, Zhong J, Chen Z. Magnetic resonance image reconstruction from undersampled measurements using a patchbased nonlocal operator. Med Image Anal. 2014;18(6):843â€“56.
Lingala SG, Hu Y, DiBella E, Jacob M. Accelerated dynamic MRI exploiting sparsity and lowrank structure: kt SLR. IEEE Trans Med Imaging. 2011;30(5):1042â€“54.
Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med. 2007;58(6):1182â€“95.
Lustig M, Donoho DL, Santos JM, Pauly JM. Compressed sensing MRI. IEEE Signal Process Mag. 2008;25(2):72â€“82.
Block KT, Uecker M, Frahm J. Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint. Magn Reson Med. 2007;57(6):1086â€“98.
Liu Y, Zhan Z, Cai JF, Guo D, Chen Z, Qu X. Projected iterative softthresholding algorithm for tight frames in compressed sensing magnetic resonance imaging. IEEE Trans Med Imaging. 2016;35(9):2130â€“40.
Eslahi SV, Dhulipala PV, Shi C, Xie G, Ji JX. Parallel compressive sensing in a hybrid space: application in interventional MRI. In: Annual international conference of the IEEE engineering in medicine and biology society (EMBC); 2017. p. 3260â€“63.
Lai Z, Qu X, Liu Y, Guo D, Ye J, Zhan Z, Chen Z. Image reconstruction of compressed sensing MRI using graphbased redundant wavelet transform. Med Image Anal. 2016;27:93â€“104.
Qu X, Guo D, Ning B, Hou Y, Lin Y, Cai S, Chen Z. Undersampled MRI reconstruction with patchbased directional wavelets. Magn Reson Imaging. 2012;30(7):964â€“77.
Ravishankar S, Bresler Y. Datadriven learning of a union of sparsifying transforms model for blind compressed sensing. IEEE Trans Comput Imaging. 2016;2(3):294â€“309.
Ravishankar S, Bresler Y. MR image reconstruction from highly undersampled kspace data by dictionary learning. IEEE Trans Med Imaging. 2010;30(5):1028â€“41.
Wang Y, Ying L. Compressed sensing dynamic cardiac cine MRI using learned spatiotemporal dictionary. IEEE Trans Biomed Eng. 2013;61(4):1109â€“20.
Yu Y, Jin J, Liu F, Crozier S. Multidimensional compressed sensing MRI using tensor decompositionbased sparsifying transform. PLoS ONE. 2014;9(6):e98441.
Shin PJ, Larson PE, Ohliger MA, Elad M, Pauly JM, Vigneron DB, Lustig M. Calibrationless parallel imaging reconstruction based on structured lowrank matrix completion. Magn Reson Med. 2014;72(4):959â€“70.
Ongie G, Jacob M. Offthegrid recovery of piecewise constant images from few Fourier samples. SIAM J Imaging Sci. 2016;9(3):1004â€“41.
Jin KH, Lee D, Ye JC. A general framework for compressed sensing and parallel MRI using annihilating filter based lowrank Hankel matrix. IEEE Trans Comput Imaging. 2016;2(4):480â€“95.
Hu Y, Liu X, Jacob M. A generalized structured lowrank matrix completion algorithm for MR image recovery. IEEE Trans Med Imaging. 2019;38(8):1841â€“51.
Zhang X, Guo D, Huang Y, Chen Y, Wang L, Huang F. Image reconstruction with lowrankness and selfconsistency of kspace data in parallel MRI. Med Image Anal. 2020;63:101687.
Haldar JP. Lowrank modelling of local kspace neighborhoods (LORAKS) for constrained MRI. IEEE Trans Med Imaging. 2014;33(3):668â€“81.
Haldar JP, Zhuo J. PLORAKS: lowrank modelling of local kspace neighborhoods with parallel imaging data. Magn Reson Med. 2016;75(4):1499â€“514.
Yaman B, Hosseini SAH, Moeller S, Ellermann J, UÄŸurbil K, AkÃ§akaya M. Selfsupervised learning of physicsguided reconstruction neural networks without fully sampled reference data. Magn Reson Med. 2020;84(6):3172â€“91.
Combettes PL, Pesquet JC. Proximal splitting methods in signal processing. In: Fixedpoint algorithms for inverse problems in science and engineering. Springer; 2011. p. 185â€“212.
Daubechies I, Defrise M, De Mol C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun Pure Appl Math. 2004;57(11):1413â€“57.
Boyd S, Parikh N, Chu E. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn. 2011;3(1):1â€“122.
Aggarwal HK, Mani MP, Jacob M. MoDL: Modelbased deep learning architecture for inverse problems. IEEE Trans Med Imaging. 2019;38(2):394â€“405.
Zhang J, Ghanem B. ISTANet: interpretable optimizationinspired deep network for image compressive sensing. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR); 2018. p. 1828â€“37.
Zhang X, Lu H, Guo D, Bao L, Huang F, Xu Q. A guaranteed convergence analysis for the projected fast iterative softthresholding algorithm in parallel MRI. Med Image Anal. 2021;69:101987.
Lu T, Zhang X, Huang Y, Guo D, Huang F, Xu Q, Hu Y, Yang L, Lin J, Yan Z, Qu X. pFISTASENSEResNet for parallel MRI reconstruction. J Magn Reson. 2020;318:106790.
Yang J, Zhang Y, Yin W. A fast alternating direction method for TVL1L2 signal reconstruction from partial Fourier data. IEEE J Sel Top Signal Process. 2010;4(2):288â€“97.
Cheng J, Wang H, Zhu Y, Liu Q, Zhang Q, Su T. Modelbased deep medical imaging: the roadmap of generalizing iterative reconstruction model using deep learning; 2019. https://arxiv.org/abs/1906.08143.
Qu X, Huang Y, Lu H, Qiu T, Guo D, Agback T, Orekhov V, Chen Z. Accelerated nuclear magnetic resonance spectroscopy with deep learning. Angew Chem Int Ed. 2020;59(26):10297â€“300.
Chen D, Wang Z, Guo D, Orekhov V, Qu X. Review and prospect: deep learning in nuclear magnetic resonance spectroscopy. Chem Eur J. 2020;26(46):10391â€“401.
Wang Z, Guo D, Huang Y, Tu Z, Orekhov V, Qu X. Accelerated NMR spectroscopy: merge optimization with deep learning; 2020. https://arxiv.org/abs/2012.14830.
Hu W, Chen D, Qiu T, Chen H, Chen X, Yang L. Denoising single voxel magnetic resonance spectroscopy with deep learning on repeatedly sampled in vivo data; 2021. https://arxiv.org/abs/2101.02254.
Huang Y, Zhao J, Wang Z, Guo D, Qu X. Complex exponential signal recovery with deep hankel matrix factorization; 2020. https://arxiv.org/abs/2007.06246.
Wang S, Su Z, Ying L, Peng X, Zhu S, Liang F. Accelerating magnetic resonance imaging via deep learning. In: IEEE 13th international symposium on biomedical imaging (ISBI); 2016. p. 514â€“17.
Han Y, Yoo J, Kim HH, Shin HJ, Sung K, Ye JC. Deep learning with domain adaptation for accelerated projectionreconstruction MR. Magn Reson Med. 2018;80(3):1189â€“205.
AkÃ§akaya M, Moeller S, WeingÃ¤rtner S, UÄŸurbil K. Scanspecific robust artificialneuralnetworks for kspace interpolation (RAKI) reconstruction: databasefree deep learning for fast imaging. Magn Reson Med. 2019;81(1):439â€“53.
Han Y, Sunwoo L, Ye JC. Kspace deep learning for accelerated MRI. IEEE Trans Med Imaging. 2020;39(2):377â€“86.
Sun J, Li H, Xu Z. Deep ADMMNet for compressive sensing MRI. In: Advances in neural information processing systems; 2016. p. 10â€“18.
Adler J, Ã–ktem O. Learned primaldual reconstruction. IEEE Trans Med Imaging. 2018;37(6):1322â€“32.
Bottou L. Largescale machine learning with stochastic gradient descent. In: Proceedings of the 19th international conference on computational statisticsparis france (COMPSTAT). PhysicaVerlag HD; 2010. p. 177â€“86.
Kingma DP, Ba J. Adam: a method for stochastic optimization; 2014. https://arxiv.org/abs/1412.6980.
Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011;12(7):2121â€“59.
Hinton G, Srivastava N, Swersky K. Neural networks for machine learning. Coursera, video lectures. 2012;264(1):2146â€“53.
Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med. 2018;79(6):3055â€“71.
Qin C, Schlemper J, Caballero J, Price AN, Hajnal JV, Rueckert D. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Trans Med Imaging. 2019;38(1):280â€“90.
Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2018. p. 9446â€“54.
Yazdanpanah AP, Afacan O, Warfield SK. Nonlearning based deep parallel MRI reconstruction (NLDpMRI). In: Medical imaging 2019: image processing, vol. 10949. International Society for Optics and Photonics (SPIE); 2019. p. 1094904.
Senouf O, Vedula S, Weiss T, Bronstein A, Michailovich O, Zibulevsky M. Selfsupervised learning of inverse problem solvers in medical imaging. Cham: Springer; 2019. p. 111â€“9.
Jin KH, Gupta H, Yerly J, Stuber M, Unser M. Timedependent deep image prior for dynamic MRI; 2019. https://arxiv.org/abs/1910.01684.
Mataev G, Milanfar P, Elad M. Deepred: deep image prior powered by red. In: Proceedings of the IEEE international conference on computer vision (ICCV). 2019.
Wang AQ, Dalca AV, Sabuncu MR. Neural networkbased reconstruction in compressed sensing MRI without fullysampled training data. Cham: Springer; 2020. p. 27â€“37.
Yaman B, Hosseini SAH, Moeller S, Ellermann J, UÄŸurbil K, AkÃ§akaya M. Multimask selfsupervised learning for physicsguided neural networks in highly accelerated MRI; 2020. https://arxiv.org/abs/2008.06029.
Hosseini SAH, Yaman B, Moeller S, AkÃ§akaya M. Highfidelity accelerated MRI reconstruction by scanspecific finetuning of physicsbased neural networks. In: Proceeding in 2020 42nd annual international conference of the IEEE engineering in medicine and biology society (EMBC); 2020. p. 1481â€“1484.
Lehtinen J, Munkberg J, Hasselgren J, Laine S, Karras T, Aittala M. Noise2noise: learning image restoration without clean data; 2018. https://arxiv.org/abs/1803.04189.
Sun L, Fan Z, Ding X, Huang Y, Paisley J. Regionofinterest undersampled MRI reconstruction: a deep convolutional neural network approach. Magn Reson Imaging. 2019;63:185â€“92.
Ramzi Z, Philippe C, Jean JS. Benchmarking MRI reconstruction neural networks on large public datasets. Appl Sci. 2020;10(5):1816.
Gan W, Sun Y, Eldeniz C, Liu J, An H, Kamilov US. Deep image reconstruction using unregistered measurements without groundtruth; 2020. https://arxiv.org/abs/2009.13986.
LaMontagne PJ, Benzinger TL, Morris JC, Keefe S, Hornbeck R, Xiong C, et al. OASIS3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease. 2019. https://doi.org/10.1101/2019.12.13.19014902v1.
Liu J, Sun Y, Eldeniz C, Gan W, An H, Kamilov US. RARE: Image reconstruction using deep priors learned without ground truth. IEEE J Sel Top Signal Process. 2020;14(6):1088â€“99.
Ke Z, Cheng J, Ying L, Zheng H, Liang D. An unsupervised deep learning method for multicoil cine MRI. Phys Med Biol. 2020;65(23):235041.
Romano Y, Elad M, Milanfar P. The little engine that could: regularization by denoising (RED). SIAM J Imaging Sci. 2017;10(4):1804â€“44.
Reehorst ET, Schniter P. Regularization by denoising: clarifications and new interpretations. IEEE Trans Comput Imaging. 2019;5(1):52â€“67.
Venkatakrishnan SV, Bouman CA, Wohlberg B. Plugandplay priors for model based reconstruction. In: Proceeding of the 2013 IEEE global conference on signal and information processing (GlobalSIP); 2013. p. 945â€“8.
Uecker M, Lai P, Murphy MJ, Virtue P, Elad M, Pauly JM, et al. ESPIRiT: an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magn Reson Med. 2014;71(3):990â€“1001.
Otazo R, Candes E, Sodickson DK. Lowrank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magn Reson Med. 2015;73(3):1125â€“36.
Nakarmi U, Wang Y, Lyu J, Liang D, Ying L. A Kernelbased lowrank (KLR) model for lowdimensional manifold recovery in highly accelerated dynamic MRI. IEEE Trans Med Imaging. 2017;36(11):2297â€“307.
Goodfellow I, PougetAbadie J, Mirza M, Xu B, WardeFarley D, Ozair S. Generative adversarial nets. In: Advances in neural information processing systems. New York; 2014. p. 2672â€“80.
Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks; 2015. https://arxiv.org/abs/1511.06434.
Zhu JY, KrÃ¤henbÃ¼hl P, Shechtman E, Efros AA. Generative visual manipulation on the natural image manifold. In: Proceeding of the European conference on computer vision (ECCV); 2016. p. 597â€“613.
Cole EK, Pauly JM, Vasanawala SS, Ong F. Unsupervised MRI reconstruction with generative adversarial networks; 2020. https://arxiv.org/abs/2008.13065.
Bora A, Price E, Dimakis AG. AmbientGAN: Generative models from lossy measurements. In: Proceeding of the international conference on learning representations (ICLR). 2018.
Villani C. The Wasserstein distances. In: Optimal transport, vol. 338. Springer; 2009. p. 93â€“111.
Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC. Improved training of wasserstein gans. In: Advances in neural information processing systems; 2017. p. 30.
Zhu JY, Park T, Isola P, Efros AA. Unpaired imagetoimage translation using cycleconsistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision (ICCV); 2017. p. 2223â€“32.
Lei K, Mardani M, Pauly JM, Vasanawala S. Wasserstein GANs for MR imaging: from paired to unpaired training. IEEE Trans Med Imaging. 2021;40(1):105â€“15.
Sim B, Oh G, Kim J, Jung C, Ye JC. Optimal transport driven CycleGAN for unsupervised learning in inverse problems. SIAM J Imaging Sci. 2020;13(4):2281â€“306.
Cheng JY, Chen F, Alley MT, Pauly JM, Vasanawala SS. Highly scalable image reconstruction using deep neural networks with bandpass filtering; 2018. https://arxiv.org/abs/1805.03300.
Gong K, Catana C, Qi J, Li Q. Direct patlak reconstruction from dynamic PET using unsupervised deep learning. In: Proceeding of the 15th international meeting on fully threedimensional image reconstruction in radiology and nuclear medicine, vol. 11072. International Society for Optics and Photonics (SPIE); 2019. p. 110720R.
Gallegos IO, Koundinyan S, Suknot AN, Jimenez ES, Thompson KR, Goodner RN. Unsupervised learning methods to perform material identification tasks on spectral computed tomography data. In: Radiation detectors in medicine, industry, and national security XIX, vol. 10763. International Society for Optics and Photonics (SPIE); 2018. p. 107630G.
Lu Y, Zhong A, Li Q, Dong B. Beyond finite layer neural networks: bridging deep architectures and numerical differential equations. In: Proceedings of international conference on machine learning (ICML). 2018; p. 3276â€“3285.
Ye JC, Han Y, Cha E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J Imaging Sci. 2018;11(2):991â€“1048.
Irarrazabal P, Nishimura DG. Fast three dimensional magnetic resonance imaging. Magn Reson Med. 1995;33:656â€“62.
Aggarwal H K, Jacob M. JMoDL: joint modelbased deep learning for optimized sampling and reconstruction; 2019. https://arxiv.org/abs/1911.02945.
Acknowledgements
The authors thank Zhangren Tu, Chen Qian, and Haoming Fang for valuable discussions.
Funding
This work is supported in part by the National Natural Science Foundation of China (61871341, 61901188, 61672335), in part by the Natural Science Foundation of Fujian Province of China under Grant (2021J011184), HealthEducation Joint Research Project of Fujian Province (2019WJ31), andÂ the science and technology fund of Fujian education department (JT180280).
Author information
Authors and Affiliations
Contributions
GZ and DG conceived the idea of the manuscript and were major contributors in writing the manuscript. YG and JZ prepared some figures and polish the English. XQ and ZW revised the manuscript. ZL and XD replied and revised comments. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
Xiaobo Qu, works as a Senior Editor for BMC Medical Imaging. The other authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Zeng, G., Guo, Y., Zhan, J. et al. A review on deep learning MRI reconstruction without fully sampled kspace. BMC Med Imaging 21, 195 (2021). https://doi.org/10.1186/s12880021007279
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12880021007279