Cloud Hosting

Patient characteristics

A total of 26,233 and 152,979 patients who met our selection criteria, as described under Methods, were identified from the Memorial Hermann Hospital System (MHHS) and Medical Information Mart for Intensive Care (MIMIC)-IV databases, respectively. Those patients had 56,233 and 393,713 index culture events over time in MHHS and MIMIC-IV datasets. The aggregated patient characteristics are described in Table1. Some patients were classified into MRSA and non-MRSA groups when they had both MRSA and non-MRSA events at different index time. Patient features were used once if the patient had two or more events in the same group. Demographic features at the time of index culture were used to describe the characteristics when patients were classified more than twice into one group. Overall, the MRSA group had a higher number of intensive care unit (ICU) admissions (MHHS: 4.3% vs. 0.7%, MIMIC-IV: 31.7% vs. 16.7%) and emergency department (ED) patients (MHHS: 66.4% vs. 13.3%, MIMIC-IV: 51.3% vs 35.0%). As MIMIC-IV was originally developed based on an ICU database, the MIMIC-IV dataset included a higher number of ICU patients. Intermediate unit (IMU) status was not included in the MIMIC-IV data. Table2 summarizes types of antibiotics and cultures before index time. Vancomycin was the most commonly used antibiotic, followed by cefepime in the MHHS dataset, whereas ceftriaxone was the second most commonly used antibiotic in the MIMIC-IV dataset. As expected, given the origin of the EHRs (MHHS from Houston and MIMIC-IV from Boston), the MHHS dataset had more Hispanic patients compared to MIMIC-IV (10.510.6% vs. 3.63.9%). Across groups, Caucasian was the most common race, and 5565 years was the most common age group. Gender was equally distributed in all groups. Blood and urine cultures were other common cultures taken during the study periods.

Table3 summarizes the bacteria and diagnostic codes identified within the event periods. S. aureus were the most common bacteria in MRSA groups, whereas E. coli was the most common in the non-MRSA group. Bacteremia (MHHS: 6.7% vs. 2.1%, MIMIC-IV: 8.6% vs. 1.9%) and skin soft tissue infection (MHHS: 24.8% vs. 5.6%, MIMIC-IV: 13.2% vs. 2.6%) were more common in MRSA groups.

Table4 shows the prediction accuracy of the models. For the MHHS dataset, the deep learning model PyTorch_EHR exhibited the highest Area Under Curve of Receiver Operating Characteristics (AUROC) of 0.911 [0.900 0.916] (see ROC curve in Supplementary Fig.5-1) compared to other machine learning models (logistic regression [LR]: 0.857 [0.8490.865] and light gradient boost machine [LGBM]: 0.892 [0.8850.899]). Similar results were obtained for the MIMIC-IV dataset (PyTorch_EHR: 0.859 [0.8490.869], LR: 0.816 [0.8040.828], and LGBM: 0.838 [0.8230.849]; see ROC curve in Supplementary Fig.5-2). We also evaluated the AUROC in each patient group with a specific diagnosis during the event. Although the AUROC decreased by 0.500.10, we had acceptable accuracy in each infection in the MHHS dataset. We also evaluated confusion matrices based on our models high-risk and low-risk predictions (see Supplementary Table4). In high-risk groups, Pytorch_EHR showed a specificity of 95.0% and 99.0%, and a sensitivity of 48.1% and 19.3% in MHHS and MIMIC-IV datasets, respectively, whereas LGBM showed a specificity of 95.0% and 99.0%, and a sensitivity of 44.5% and 14.9%. In low-risk groups, Pytorch_EHR had a sensitivity of 95.0% and 90.0% and a specificity of 62.9% and 58.7% in MHHS and MIMIC-IV datasets, respectively, whereas LGBM showed a sensitivity of 95.0% and 90% and a specificity of 62.8% and 57.2%.

Given the imbalanced distributions of positive events in both datasets, for high-risk patients, positive predictive values (PPV) were relatively low: 65.6% and 22.4% for Pytorch_EHR and 63.6% and 17.5% for LGBM in MHHS and MIMIC-IV datasets, respectively. However, negative predictive values (NPV) were high: 90.3% and 98.9% for Pytorch_EHR and 89.7% and 98.8% for LGBM in MHHS and MIMIC-IV datasets, respectively. For low-risk patients, PPV was low: 37.6% and 3.0% for Pytorch_EHR and 33.5% and 2.9% for LGBM in MHHS and MIMIC-IV datasets, respectively. However, NPV were particularly high: 98.6% and 99.8% for Pytorch_EHR and 98.5% and 99.8% for LGBM in MHHS and MIMIC-IV datasets, respectively.

Fig.1 shows the cumulative incidence curve of MRSA-positive cultures over two weeks from the index culture. In both datasets, our model clearly differentiated the patients with high and low risks of MRSA-positive cultures. The cumulative incidence of MRSA-positive cultures in the MRSA group in the MHHS dataset was 61.2%, whereas the incidence in the MIMIC-IV dataset was approximately 18.2%. The low incidence in MIMIC-IV despite a high risk was likely due to the overall incidence of positive MRSA cultures in the MIMIC-IV dataset.

a and b show cumulative incidence of MRSA cultures in Memorial Hermann Hospital System (MHHS) and Medical Information Mart for Intensive Care (MIMIC)-IV datasets, respectively. Both figures were generated based on the risk predicted by our model in test datasets. Given the significant imbalance in the MIMIC-IV dataset, even high-risk patients achieved 20% positivity compared to the MHHS dataset. In contrast, the low-risk patient group had fewer false negatives. The shaded area in the graph represents the 95% confidence intervals. MHHS Memorial Hermann Hospital System, MIMIC Medical Information Mart for Intensive Care, MRSA Methicillin Resistant Staphylococcus aureus.

AUROC curves over multiple index events were evaluated in MHHS and MIMIC-IV test datasets. (See Supplementary Fig.10) When evaluated on patients with only the first event in MHHS dataset, LGBM model performance was better than that of PyTorch_EHR and LR models. However, when evaluated on patients who had repeated events, i.e., a longer duration of observation in the dataset, PyTorch_EHR model performance improved significantly and sustained superiority against the LR and LGBM models. Similar results were obtained for the MIMIC-IV dataset, with a longer duration of observation providing better performance in the PyTorch_EHR model.

Table5 summarizes the potential clinical impact of the PyTorch_EHR model. In patients predicted as low risk, our model exhibited NPV of 98.6% and 99.8% in MHHS and MIMIC-IV datasets, respectively. In addition, among those low-risk patients who had true negative results, MRSA-specific antimicrobials were given by treating clinicians in 21.6% (1505/6975) and 2.3% (1069/45,533) of events, which translated to 7949 and 1397 doses of MRSA-specific antimicrobials in MHHS and MIMIC-IV, respectively. The main antimicrobials used for those patients were vancomycin (6833 and 1254 doses in MHHS and MIMIC-IV, respectively), followed by linezolid (852 and 88 doses) and daptomycin (264 and 55 doses). Further, 1.4% (98/6,975) and 0.2% (108/45,533) events were false negatives in our model. Among them, only 0.3% (23/6,975) and 0.04% (27/45,533) events received MRSA-specific antimicrobials, which could be missed by our model.

In high-risk patients, our model exhibited PPV of 65.6% and 22.4% in MHHS and MIMIC-IV datasets, respectively (Supplementary Table4). The model predicted 12% (1437/11,922) and 1.2% (957/78,548) of events as high risk. Among high-risk groups, patients did not receive any MRSA-specific antimicrobials in 34.6% (497/1437) and 19.7% (189/957) of events in MHHS and MIMIC-IV datasets, respectively. On the contrary, with our models high-risk prediction, 15.8% (227/1437) and 71.1% (671/957) events may receive unnecessary MRSA-specific antimicrobials (potential harm from our model).

Finally, we evaluated the performance of our model in patients who had MRSA bacteremia. As summarized in Tables5, 31.8% (457/1437) and 7.3% (70/957) of high-risk events in MHHS and MIMIC-IV datasets, respectively, had MRSA bacteremia. These rates were much higher than the rates in low-risk events in MHHS (0.5%; 32/6975) and MIMIC-IV (0.04%; 35/48,455). Based on these findings, high-risk group had 69.3 and 101.2 higher relative risk of MRSA bacteremia compared to low-risk patient group. In addition, our model identified 58.0% (265/457) and 50.0% (35/70) of high-risk patients with true MRSA bacteremia did not receive MRSA-specific antimicrobials, considered optimal antibiotics for MRSA bacteremia, within 12h of the index cultures.

These results were also evaluated in other models and any MRSA antimicrobials (see Supplementary Table5). Overall, PyTorch_EHR model exhibited higher net-benefits against treating clinicians decisions compared to LGBM and LR models, except for MRSA bacteremia in MIMIC-IV dataset. LGBM model provided better net benefit compared to PyTorch_EHR model (18 vs. 10 MRSA bacteremia cases may receive early MRSA antimicrobials, respectively.)

We obtained the contribution scores for positive MRSA cultures in the datasets. Supplementary Fig.7 shows the top 14 median contribution scores of admission diagnoses in our model for MHHS data. Interestingly, our model identified multiple diagnoses often related to MRSA infections, such as cutaneous abscesses or boils. Supplementary Fig.8 shows the top 10 overall contribution scores for antimicrobial exposures before the index time in the datasets. Some common antibiotics had high scores in both datasets, but it was difficult to interpret the scores clinically.

We also present individual feature importance as a bar graph for an example patient among the patients we visualized (see Supplementary Fig.9). The patient is female and between 45 54 years of age, with multiple underlying comorbidities listed on admission two days (2 days) before the index culture (blood culture on index date). Our model identified a risk score of 0.541 (predicted as a positive patient). After the patient was admitted to the hospital, vancomycin and meropenem were initiated, and a blood culture was ordered. Subsequently, cultures identified MRSA over two weeks.

See the original post here:
Deep learning model for personalized prediction of positive MRSA culture using time-series electronic health records - Nature.com