Intelligent neonatal monitoring based on a virtual thermal sensor

Background Temperature measurement is a vital part of daily neonatal care. Accurate measurements are important for detecting deviations from normal values for both optimal incubator and radiant warmer functioning. The purpose of monitoring the temperature is to maintain the infant in a thermoneutral environmental zone. This physiological zone is defined as the narrow range of environmental temperatures in which the infant maintains a normal body temperature without increasing his or her metabolic rate and thus oxygen consumption. Although the temperature measurement gold standard is the skin electrode, infrared thermography (IRT) should be considered as an effortless and reliable tool for measuring and mapping human skin temperature distribution and assist in assessing thermoregulatory reflexes. Methods Body surface temperature was recorded under several clinical conditions using an infrared thermography imaging technique. Temperature distributions were recorded as real-time video, which was analyzed to evaluate mean skin temperatures. Emissivity variations were considered for optimal neonatal IRT correction for which the compensation vector was overlaid on the tracking algorithm to improve the temperature reading. Finally, a tracking algorithm was designed for active follow-up of the defined region of interest over a neonate’s geometry. Results The outcomes obtained from the thermal virtual sensor demonstrate its ability to accurately track different geometric profiles and shapes over the external anatomy of a neonate. Only a small percentage of the motion detection attempts failed to fit tracking scenarios due to the lack of a properly matching matrix for the ROI profile over neonate’s body surface. Conclusions This paper presents the design and implementation of a virtual temperature sensing application that can assist neonatologists in interpreting a neonate’s skin temperature patterns. Regarding the surface temperature, the influence of different environmental conditions inside the incubator has been confirming.


Background
Recently, the rapid improvement in medical thermography technologies in various clinical fields has promoted the use of thermography imaging as a contactless physiological sensor. In particular, neonatal intensive medicine is a clinical field in which infrared thermography may play a future role in non-invasive monitors.
Initially, Clark et al. [1] performed the first clinical trials using direct thermography measurement in neonates, which was dated back to 1980. To perform non-invasive skin temperature measurements, the setup included a hole in the roof of the incubator and the assistance of a mirror system; these additions [1,2] allowed for real-time measurements of thermal reputation.
Adams et al. [3] achieved successful direct thermography imaging in the earliest minutes of life by using a long-wave infrared (LWIR) system. In that project, continuous thermal monitoring of the neonate was accomplished at intermittent intervals ranging between 20 and 30 minutes at the initial stage. Then, a modified protocol was defined to monitor preterm infants inside a convective incubator, kangaroo mother care, and open radiant warmer. The results were compared with values obtained from multiple weighted measurements of resistance temperature device (RTD) sensors.
Pavlidis et al. [4][5][6] developed a tracking system for infrared thermography as part of an augmented computer vision system. This development was based on a coalitional tracking approach in which a distinct region of interest (ROI) was defined over the neonate's face and its position was tracked over numerous infant motion planes.
Recently, Abbas et al. [7] developed a concept for noncontact respiration monitoring in infants based on IR thermography (IRT). This technique also tracks the nostrils' thermal signature to detect the infant's breathing rate at a distance, and it provides an insightful analysis of possible error sources within the neonatal IRT (NIRT) imaging technique. The need for a robust and intelligent temperature monitoring methodology has increased, which makes NIRT imaging a suitable candidate for contactless temperature measurement and observation inside neonatal intensive care unit (NICU) facilities [2,8].
The NIRT method demonstrates good outcomes for the real-time and continuous quantification of a neonate's surface and core temperatures; however, it lacks the ability to estimate the real temperature value on a neonate's body surface accurately. This lack of reliability is mainly due to the unknown emissivity, ε. For reference, the experimenter could utilize an emissivity value ε for a known material surface or could utilize fabric supplies, such as a hand band or head caps, sutured with a material of known emissivity, such as copper, polished steel, or polyvinyleflouride electrical tape (e.g., scotch-764). However, in such clinical study, it is impossible to use like material due to the hygienic and disinfection concern that roses within the utilization of these material inside infant incubators.

Thermal imaging
Radiation in the long wave infrared (LWIR) bands (8-14 μm) is important because the human body emits most of its thermal radiation, which encodes valuable physiologic information, in this region of electromagnetic spectrum. This vital information, if properly processed and analyzed, may be used in many biomedical applications, such as mean body temperature mapping and arterial pulse measurements [6,9,10]. A solid base that includes an understanding of the physics of image formation principles, the choice of imaging IR band, and instrumentation is crucial for successful biometrics signature processing. Such signatures include superficial vessel blood flow [11], forehead mean temperature, and nostril thermal patterns [4,[12][13][14].
Possible IRT tracking and monitoring sites on a neonate's body are displayed in Figure 1; these spatial points will be the reference sites for virtual temperature sensing as the issue is discussed further in this paper.
Thermography imaging offers a high-quality concept for the observation and monitoring of different physiological processes [8,15,16]. Recently, we used IR thermal imaging to monitor and map the temperature distribution over the preterm infant's body [12,17,18]. We believe that this technique will become an alternative technique in the future to gold-standard technologies in neonatal temperature monitoring and control [19].

Methods
All measurements were performed using a VarioCAM® hr head (InfraTec GmbH, Germany) IR camera (LWIR, 7 μm to 14 μm). The camera transferred the thermal map to a PC via the IEEE 1394 FireWire interface. The neonate's thermal images were taken inside a convective infant incubator (Caleo, Draeger AG, Germany) and converted to a 2D array containing temperature information within the LabVIEW software platform. Additionally, these data were used to test the algorithm software's ability to track the specified virtual temperature sensor points on a neonate's skin after motion. Figure 2 illustrates a typical setting for NIRT clinical study inside a convective incubator.

Thermography imaging experiment design
Only ten newborn infants were selected to participate in the clinical study, five of them were under radiant warmer therapy and the rest are placed inside convective incubator. A referential ground truth measurement was implemented by using skin temperature electrodes as gold standards. The accuracy of these clinical skin electrodes is (± 0.1°C). The NIRT imaging and measurement was performed at the Department of Neonatology (RWTH Aachen University Hospital), and this has been approved by the medical ethics committee of the RWTH Aachen University Hospital, issued on 19 August 2009 with reference code (EK032/09). The acquired thermography datasets used for testing the tracking algorithm. Each dataset contained one measurement scene consisting of a newborn infant undergoing thermography inside a convective incubator or under a radiant warmer. The tracking time was approximately 20 minutes for each subject with a frame rate of 25 fps, and the measurements were conducted as a real-time imaging operation. In principle, a higher frame rate (up to 50 fps) could be achieved; however, a higher frame rate would increase the size of the thermography data to an out-of-memory level in many PCs.
Principally, the selected thermography datasets often included involuntary movements of the neonate during the 20 minutes of thermography acquisition time. The thermography data featured out-of-plane rotation of the facial tissue, hands, feet, and main trunk as the neonates rotated their heads left, right, up, down, or in a random motion. For covering all planes and geometry of the neonate, we configure and selected ROI over the neonate's skin to guarantee effective temperature detection over examination time (Figure 3).
A ring-projection transformation was selected in the tracker hierarchy to be compared against the active ROI tracker. The calibration phase of the IR camera was performed directly throughout the measurement time. The typical NIRT protocol sequence used in this study explained in Figure 4, in which the NIRT measurement phase indicates different intervals throughout time.

IR thermal camera calibration setting
The calibration process of the thermal camera took place inside the NICU ward in synchronization with the NIRT measurement phases [20]. This process is called automatic non-uniformity calibration (ANUC), and the procedure compensates for temperature drift during measurements. In addition, the selected field of view (FOV) for the camera assured that there is no influence on thermography resolution during NIRT imaging despite the inclined side angle of the thermal camera within the allocated FOV. This was confirmed during analysis and modeling of heat fluxes dissipated from neonate within NIRT measurement [19].
Temperature and humidity variations inside the convective incubator are commonly considered the main  factors that prevent accurate temperature calibration. Therefore, to avoid any incorrect temperature registration and physical related errors in NIRT imaging, the calibration process was implemented during the clinical measurement using the IRBIS® Professional software of the IR camera. Objects of interest (OOIs) inside the acquired thermogram were selected and the environmental, incubator and object settings were performed through an IR transparent window (with 0.01 mm thickness) made of polyethylene (PE) material [3].
The transmission of IR radiation through the foil is between 0.92 and 0.94. Therefore, this transparent foil was chosen to block the opened incubator clapper while allowing the baby inside the incubator to be visualized because the Plexiglas® material of the incubator hood is an IR-reflecting material with emissivity values reaching 0.97 [1,21].
A geometric correction was applied to the acquired thermography using selected region of interests (ROI) over the neonate's skin and setting the physical parameters (e.g., incubator air temperature, outside window temperature, humidity, IR transmission of PE thin-foil and body temperature) for optimal thermography correction. Figure 5 shows the difference in calibration setting between different thermography scenes where in scene (a) the thermography imaging performed through IR-transparent window and in scene (b) thermography imaging performed directly without interfering media [19].
Moreover, the data were registered against an emissivity equal to unity (considering neonatal skin as a typical blackbody radiator), although the actual value of emissivity was equal to 0.972 [22,23]. This correction strategy plays a vital role in accurate temperature mapping because any slight difference in the emissivity value will  tend to add inaccuracies to the temperature reading from the IR camera.

Thermal virtual sensor architecture
The term "Virtual InfraRed SENSor" (VIRSENS) relates to a sensing method based on augmented visual or physical measurements. In this work, a virtual temperature sensor was developed wherein contactless temperature measurements essentially replace the clinical gold standards. Furthermore, virtual sensor tracking software was developed using Lab-VIEW® Vision Assistance (National Instruments®) as an integrated toolkit. This software allowed the thermal camera to be connected directly the LabVIEW console by using a native interface file provided by the manufacturer (Figure 6).
Thermography acquisitions began after IR camera calibration and were followed by the extraction of the thermal data from the color space of the image; this task formed a crucial step of the VIRSENS concept. Moreover, the selection of the ROI array was initiated afterward to set the tracking coordinates of the neonate's body regions to be implemented the image-processing loop and architecture ( Figure 7).

Tracking technique
The key aspect for robust virtual sensing is the tracking method, which should accurately monitor the motion of the target surface even in the presence of partial occlusion or deformation [24]. This tracking system is applied to follow the motion of the target's outline (and not only superficial features) [25][26][27][28]. Generally, motion tracking is not a straight forward process; it depends on the proper definition of the tracked anatomical geometry and the ability to follow-up and mark the defined ROI over multiple thermography frames (Figures 7 and 8).
Primarily, the tracking algorithm can be divided into five main stages, as illustrated in Figure 8: IR thermography acquisition, ROI geometry profile definition, object coordinate tracking, information extraction, and sensor display. The manner in which the active ROI moves through the image frames is illustrated in Figure 9, where the yellow rectangle moves with the relative motion of the baby inside the camera's field of view (FOV).
When template matching, the ring projection template (RPT) process was used to address rotational variations within the thermography-imaging scene. The RPT reduces a 2D thermogram image into a 1D vector. In general, this task is used as a pre-processing step in the VIRSENS approach.
We define the initial template to be T(x,y) of size (M × N). The RPT process begins by deriving a center point on the Template T(x,y) that is denoted as (x c ,y c ). Subsequently, the Figure 5 Two thermograms showing the effect of geometric correction of the neonate, it is noticed that the neonates skin temperature in thermogram (a) with higher value than in thermogram (b) of the same neonate, therefore, correction step prior to NIRT imaging is important for accurate thermography acquisition. Cartesian frame coordinate Template T(x,y) is transformed into polar frame coordinates based on the following relations: x = r cosθ (for horizontal reference) , y = r sinθ (for vertical reference) where Basically, the ring projection in the selected template T(x,y) at radius r is denoted as P T (r) and is defined as follows: where S r is the total number of pixels falling on the circle of radius r = 0,1,2,…,R and k denotes the number of correlation iterations in template matching kernel. Note  that P T (r) is defined as the mean pixel intensity along a circle whose radius to the center of the template has equal order in the correlation computation. Because the RPT is synthesized along circular rings of increasing radii, the derived 1D RPT on the thermography image is invariant to the rotation of its corresponding 2D image template. To effectively obtain the RPT computation along concentric circles, the method employs a look-up table (LUT) whose diameter is set to the size of the template in the ring projection process.
Finally, the RPT is obtained simply by summing up the pixel values along a concentric circle within the template results. For the matching process, the normalized correlation (NC) is adopted in the similarity measurement. Therefore, we consider the following: Generally, the representations of the reference template ring-projection vectors (P With this definition, the value is unaffected by rotational and linear changes (at constant gain and contrast offset in the thermal imaging) in the reference template and thermography scene subimage. In addition, the dimensional length of the ring projection vector is only (R + 1). This significantly increases the computational efficiency for the vector Parametric vector approach for template matching The method proposed here is inspired by the PT method, which is characterized by a decrease in computational complexity when the thermography image involves a change of scale and rotation. Therefore, it is considered a robust solution for the largescale image data generated in medical thermography [26,29,30]. To obtain rotation/scale invariance in the matching process, a simple approach using a P T vector (template image) and a P S vector (scene subimage) was proposed.
In the VIRSENSE approach, a PT vector P → T P was constructed from a base-ring projection set ( P → t 0 ; P → t 1 ; …; P → t N ) consisting of the RPTs and including the template image and differently scaled images as follows: The NC between the scene subimage vector P → S and a PT vector P → T P becomes P → T ; P → S D E ; then, the problem under consideration can be solved by constrained optimization, that is, Essentially, the Lagrangian multiplier (LM) method can solve this problem of difference optimization. The solution of ω → is given by where The next step of the algorithm is producing the scaling value sq estimation of the scene subimage, which initiates in terms of the following equation where s i for 0 ≤ i ≤ N denotes the different scaling values generated by scaling the template image. The approach enables fast matching in the ROI tracking algorithm. The computational efficiency is significantly increased because the RPT process reduces a 2D thermography image array into a 1D vector. Additionally, the correlation matrix (L) can be determined in the training phase while the optimal parameters ω → , the scaling value obtained directly from the correlation vector F → , and the correlation matrix L are determined in the matching phase [30]. In fact, there is no iteration step involved in this tracking template-matching-based algorithm. Therefore, the computational time is considerably reduced.
Generally, this data description is appended to the input template image. During the matching phase, the template descriptor (the ROI descriptor, P ROI (x T , y T )) is extracted from the template image and used to search the template in the inspection image [31][32][33].
The mathematical process of image cross-correlation is simple; the RPT is overlaid on the source thermogram image, and the intensity values for each corresponding pixel are multiplied individually. Additionally, all of the matched templates are summed to produce a single correlation value [32,33].
The correlation value matrix is then scanned for its peak value. This position generally conforms to the position in the source image that most closely matches the template [22,34,35]: where P(x,y) is the reference template position on the thermography image. The correlation matrix can include several high values that correspond to several instances (events) of tracked templates in the source thermography image [36][37][38].

Scale (shift)-and rotation-invariant technique
One of the greatest flaws in cross-correlation is its inability to match objects in a source image that are either a different size or rotated compared to the reference template. These two template-matching mechanisms are used in the ROI descriptor tracking (corresponding to the projected template) in the frame matrix. The mathematical approximation of such a template inside a rectangular contour with T k (x k ,y k ) is as follows: To overcome and compensate for this issue throughout the NIRT data frames, the template must be rescanned over the thermography scene image using different rotations and sizes (variances in both the x-and y-axes). This process can be extremely time consuming; consider performing a cross-correlation 360 times just to perform a rotation-invariant match without even sub-degree precision [35,39,40].
If the tracked portion always has the similar size and no spatial distortion exists, then the virtual sensor does not scan for size variations [4,26,27,41]. The identical principle is applicable for rotation variance if the body part will be repeatedly positioned at the same orientation ( Figure 10). In that case, the source thermography image is rescanned using a range of different angles (crosscorrelation can typically detect object rotations of approximately ±5°without rescanning) is not necessary.
The detection of object rotations can be accomplished at up to ±12°-18°angle of rotation without rescanning and initializing the reference ROI template. However, the inability of cross-correlation to match objects in a source image that are either a different size or rotated compared to the template is still one of the shortcomings in the rotation-and shift (scale)-invariant method for the object detection system [26,30,[42][43][44].

Results and discussion
In summary, the results obtained from the virtual sensor demonstrate its ability to accurately track different geometric profiles over the external anatomy of a neonate. Only a small percentage of the motion detection trials failed to track due to the lack of a properly matching matrix for the ROI descriptor under study (see Table 1).
The main clinical application of the presented virtual sensor approach is the continuous monitoring of patients without loss of the ROI due to unexpected movements or involuntary motions initiated by the patient.
The VIRSENS approach offers the flexibility to perform stress-test infrared thermography, e.g., on treadmills, or to monitor unconscious patients (e.g., under intensive or critical care). Furthermore, this non-contact temperature monitor may become a tool in high-risk missions, such as for pilots or submarine staff [9,12,45], to provide online monitoring of respiration activity through convective heatloss during expiration and inspiration [7,20,41,46].
To further advance the use of VIRSENS in neonatal medicine, we used embedded contactless temperature monitoring and regulation in a neonatal incubator closedloop control system. This approach can reduce the need for skin temperature electrodes and the problems associated with their use, such as sensor dislocation, motion artifacts, calibration drift, wire crowding, false connections, and the possibility of infection for newborn infants.
Moreover, this tracking method requires additional validation tests and clinical trials to provide beside the proof-of-concept (POC) of this technology feasibility in the neonatal monitoring field. Figure 10 Imaging-plane layout of ROI tracking over a neonate's different body regions displaying the out-of-plane rotation coordinates that were used to develop an ROI tracking algorithm for medical IRT. The table also illustrates the correlation of the tracked ROI descriptor over the measurement scene with respect to a newly chosen position of the ROI descriptor. 1 The table presents the comparison of different success rates for the virtual temperature sensor used within the NIRT imaging for illustrating the scoring percentage of fitted and tracked ROI over the misallocated ones.
In addition, the ability of VIRSENS to perform geometric identification of selected body parts (e.g., face, hands, legs, interscapular, and maxillary region) (see Additional files 1, 2 and 3) adds a crucial role in anatomical posture identification for neurological reflexes and postural control of neonates. Because the VIRSENS has several misallocated ROI over the neonate's geometry during the tracking process (Figure 11), which indicates that this method need further optimization and feasibility studies. This believed to be solved when more stable and precise tracking algorithms used in the VIRSENS architecture to become more stable monitoring technique. Table 2 provides some of quantitative analysis for performance measuring in different thermography datasets within NIRT study. This table showing the scoring of matches of tracked ROI per anatomical regions for seven infants participating in the study. As we can see from Table 2 that the higher success rate of this scoring occurs, in the facial, plane where there is a prominent landmark such as nose, orbital, forehead and maxillofacial regions. Therefore, this is highly discriminated from other anatomy such as hand, arms, legs and trunk can be use the facial tracking as referential template for tracking accuracy and validation procedure of virtual thermal sensor.

Conclusion
In this study, a thermal imaging tracking method was proposed and tested based on a template-matching algorithm. The developed method uses a spatially trained ROI tracker whose interactions are modeled using cross-correlations of the ROI template and a searchable IR image. The method's output provides pixel-level tracking accuracy even in the presence of multidimensional target transformation. The proposed tracking method was effectively tested in thermal and visual datasets featuring facial regions and other anatomical objects. The thermography tracking system for neonatal monitoring was implemented and tested for clinical monitoring inside NICU unit. The main conclusion from this experiment is that the tracking can be robust over wellcalibrated thermography frames and for lesser jerky movements of the neonate. In fact, thermography measurements performed at a distance are beneficial from a psychological viewpoint for both staff and the patient's relatives but produce challenges from the medical perspective. The tracking problem, which is pivotal in this study, was particularly challenging due to the functional nature of thermal IR imaging and its application in realtime operation.
Moreover, NIRT imaging depicts physiological changes; therefore, it is highly dynamic, non-linear, unpredictable in its uncertainties, and difficult to model. In addition, the estimation of the emissivity value at certain tracking points requires further optimization and development before it can be included in prospective NIRT applications, such as the detection of respiration signatures with the IRTR method or evaluation of superficial blood perfusion over active metabolic regions (e.g., liver and brain). Because these applications would appear to be difficult tasks due to the slow hemodynamic activities of the superficial vessels, the method requires further development and improvement for clinical convention in contactless blood perfusion and hemodynamics parameters. Furthermore, this physiological tracking application based on thermography might consider a good candidate for running on smartphones and other mobile communication devices. These applications can be a part of the widespread adoption and use of mobile and computing vision technologies is opening new and innovative ways to improve health care delivery. This in turn can transform a mobile platform into a regulated medical monitoring system.

Statement of consent
An oral consent was gained from the parents of the patent for publication of optical and thermography images and their related files according to medical ethics approval from Medical Ethics committee of the RWTH Aachen University Hospital, issued on 19 August 2009 (EK032/09).

Additional files
Additional file 1: Thermography video file for virtual temperature sensor (VIRSENS) used in NIRT imaging application (respiratory monitoring) for a neonate one cared in open radiant warmer.  This table gives the quantitative index for the total numbers of fitted ROIs and missed ROIs over the total number of these selected ROIs for different spatial positions over neonate's body.