Patient characteristics
A total of 26,233 and 152,979 patients meeting the inclusion criteria were identified from the Memorial Hermann Hospital System (MHHS) and Intensive Care Medical Information Mart (MIMIC)-IV databases, respectively, as described in Methods. These patients had 56,233 and 393,713 index culture events over time in the MHHS and MIMIC-IV datasets. Summary patient characteristics are shown in Table 1. Some patients were classified into MRSA and non-MRSA groups if both their MRSA and non-MRSA events occurred at different index times. Patient characteristics were used once if a patient had two or more events within the same group. Demographic characteristics at the time of index culture were used to describe the characteristics if a patient was classified into one group more than once. Overall, the MRSA group had significantly lower intensive care unit (ICU) admissions (MHHS: 4.3% vs. 0.7%, MIMIC-IV: 31.7% vs. 16.7%) and emergency department (ED) admissions (MHHS: 66.4%). ) was high. vs. 13.3%, MIMIC-IV: 51.3% vs. 35.0%). Because MIMIC-IV was originally developed based on the ICU database, the MIMIC-IV dataset included more ICU patients. Intermediate unit (IMU) status was not included in the MIMIC-IV data. Table 2 summarizes the antibiotic type and pre-index time cultures. In the MHHS dataset, vancomycin was the most commonly used antibiotic, followed by cefepime, while in the MIMIC-IV dataset, ceftriaxone was his second most commonly used antibiotic. As expected, given the origins of the EHRs (his MHHS in Houston and his MIMIC-IV in Boston), the MHHS dataset contained more Hispanic patients compared to his MIMIC-IV ( 10.5-10.6% vs. 3.6-3.9%). The most common race across the group was white, and the most common age range was 55 to 65 years old. Gender was evenly distributed across all groups. Blood and urine cultures were other common cultures taken during the study period.
Types of infectious diseases and other pathogens
Table 3 summarizes the bacteria and diagnosis codes identified within the event period. Staphylococcus aureus Although it was the most common bacteria in the MRSA group, Escherichia coli It was most common in the non-MRSA group. Bacteremia (MHHS: 6.7% vs. 2.1%, MIMIC-IV: 8.6% vs. 1.9%) and skin and soft tissue infections (MHHS: 24.8% vs. 5.6%, MIMIC-IV: 13.2% vs. 2.6% ) is more common in the MRSA group.
model prediction
Table 4 shows the predictive accuracy of the model. For the MHHS dataset, the deep learning model PyTorch_EHR showed the highest area under the receiver operating characteristic curve (AUROC) of 0.911. [0.900 – 0.916] Comparison with other machine learning models (logistic regression) (see ROC curve in Supplementary Figure 5-1) [LR]: 0.857 [0.849–0.865] and light gradient boost machine [LGBM]: 0.892 [0.885–0.899]). Similar results were obtained on the MIMIC-IV dataset (PyTorch_EHR: 0.859 [0.849–0.869],LR: 0.816 [0.804–0.828]LGBM: 0.838 [0.823–0.849]; see ROC curve in Supplementary Figure 5-2). We also assessed her AUROC for each patient group with a specific diagnosis during the event. Although the AUROC decreased by 0.50-0.10, acceptable accuracy was obtained for each infection in the MHHS dataset. We also evaluated the confusion matrix based on the model’s high- and low-risk predictions (see Supplementary Table 4). In high-risk groups, Pytorch_EHR showed specificity of 95.0% and 99.0% and sensitivity of 48.1% and 19.3% in MHHS and MIMIC-IV datasets, respectively, while LGBM showed specificity of 95.0% and 99.0%. The sensitivity was 44.5% and 14.9%. In the low-risk group, Pytorch_EHR had a sensitivity of 95.0% and 90.0% and a specificity of 62.9% and 58.7% in the MHHS and MIMIC-IV datasets, respectively, whereas LGBM had a sensitivity of 95.0% and 90% and a specificity of 62.9% and 58.7%, respectively. It was 58.7%. 62.8% and 57.2%.
Considering the unbalanced distribution of positive events in both datasets, the positive predictive value (PPV) for high-risk patients is relatively low: 65.6% and 22.4% for Pytorch_EHR in MHHS and MIMIC-IV datasets and 22.4% in LGBM. They were 63.6% and 17.5%. Each. However, the negative predictive value (NPV) was high: 90.3% and 98.9% for Pytorch_EHR and 89.7% and 98.8% for LGBM in the MHHS and MIMIC-IV datasets. For low-risk patients, PPV was low: 37.6% and 3.0% for Pytorch_EHR and 33.5% and 2.9% for LGBM in the MHHS and MIMIC-IV datasets, respectively. However, the NPV was particularly high: 98.6% and 99.8% for Pytorch_EHR and 98.5% and 99.8% for LGBM in the MHHS and MIMIC-IV datasets, respectively.
Figure 1 shows the cumulative incidence curve of MRSA-positive cultures over 2 weeks from the index culture. In both datasets, our model clearly differentiated between patients at high and low risk for MRSA-positive cultures. His cumulative incidence of MRSA-positive cultures in his MRSA group in the MHHS dataset was 61.2%, whereas the incidence in the MIMIC-IV dataset was approximately 18.2%. The low incidence of MIMIC-IV despite the high risk may be due to the overall incidence of positive MRSA cultures in the MIMIC-IV dataset.
be and b The figures show the cumulative incidence of MRSA cultures in the Memorial Hermann Hospital System (MHHS) and Intensive Care Medical Information Mart (MIMIC)-IV datasets, respectively. Both numbers were generated based on the risk predicted by the model in the test dataset. Considering the significant imbalance in the MIMIC-IV dataset, even in high-risk patients he achieved a 20% positivity rate compared to the MHHS dataset. In contrast, there were fewer false negatives in the low-risk patient group. The shaded area in the graph represents the 95% confidence interval. MHHS Memorial Hermann Hospital System, MIMIC Intensive Care Medical Information Mart, MRSA Methicillin Resistance Staphylococcus aureus.
AUROC curves across multiple index events were evaluated on the MHHS and MIMIC-IV test datasets. (See Supplementary Figure 10) The performance of the LGBM model was better than that of the PyTorch_EHR and LR models when evaluated on patients with only the first event in the MHHS dataset. However, when evaluated in patients with repeat events, i.e. patients with a longer observation period in the dataset, the performance of the PyTorch_EHR model improved significantly and maintained its superiority over the LR and LGBM models. Similar results were obtained on the MIMIC-IV dataset, where the longer the observation period, the better the performance of the PyTorch_EHR model.
Potential clinical impact
Table 5 summarizes the potential clinical impact of the PyTorch_EHR model. In patients predicted to be low risk, our model showed her NPV of 98.6% and 99.8% in the MHHS and MIMIC-IV datasets, respectively. Additionally, among low-risk patients with a true-negative result, MRSA-specific antibiotics were administered by the treating clinician in 21.6% (1505/6975) and 2.3% (1069/45,533) of the events. corresponds to 7949 and 1397 doses. MHHS and MIMIC-IV, respectively, of her MRSA-specific antibiotics. The main antibiotics used in these patients were vancomycin (6833 and 1254 doses for MHHS and MIMIC-IV, respectively), followed by linezolid (852 and 88 doses) and daptomycin (264 and 55 doses). ) followed. Additionally, the model had 1.4% (98/6,975) and 0.2% (108/45,533) of events as false negatives. Of these, only 0.3% (23/6,975) and 0.04% (27/45,533) events were in which MRSA-specific antibiotics were administered, which could be missed by our model.
In high-risk patients, our model showed a PPV of 65.6% and 22.4% in the MHHS and MIMIC-IV datasets, respectively (Supplementary Table 4). The model predicted 12% (1437/11,922) and 1.2% (957/78,548) of events to be high risk. Among high-risk groups, in 34.6% (497/1437) and 19.7% (189/957) of events in the MHHS dataset and her MIMIC-IV dataset, respectively, the patient did not receive her MRSA-specific antibiotics. I had not received it. Conversely, our model’s high-risk predictions indicate that 15.8% (227/1437) and 71.1% (671/957) of events are likely to receive unnecessary MRSA-specific antibiotics (from our model potential harm).
Finally, we evaluated the performance of our model in patients with MRSA bacteremia. As summarized in Table 5, 31.8% (457/1437) and 7.3% (70/957) of high-risk events in the MHHS and MIMIC-IV datasets, respectively, were MRSA bacteremia. These rates were much higher than the rates of low-risk events in MHHS (0.5%; 32/6975) and MIMIC-IV (0.04%; 35/48,455). Based on these findings, the high-risk group was found to have a 69.3 and 101.2 higher relative risk of MRSA bacteremia compared to the low-risk patient group. Furthermore, in our model, 58.0% (265/457) and 50.0% (35/70) of high-risk patients with true MRSA bacteremia received the “best” antibiotic for MRSA bacteremia. It was identified that a possible MRSA-specific antibiotic had not been administered within 12 hours. Index of culture.
These results were also evaluated in other models and with any MRSA antibiotic (see Supplementary Table 5). Overall, the PyTorch_EHR model showed higher net benefits for clinicians’ treatment decisions compared to the LGBM and LR models, except for his MRSA bacteremia in the MIMIC-IV dataset. The LGBM model had a better net benefit compared to the PyTorch_EHR model (18 and 10 MRSA bacteremic cases could receive earlier MRSA antibiotics, respectively).
Importance of features
We obtained a contribution score for MRSA-positive cultures in the dataset. Supplementary Figure 7 shows the median of the top 14 contributing scores for admission diagnoses in the model for the MHHS data. Interestingly, our model identified multiple diagnoses often associated with MRSA infection, including skin abscesses and boils. Supplementary Figure 8 shows the top 10 overall contribution scores to antimicrobial exposure before the index time in the dataset. Although some common antibiotics showed high scores in both datasets, the scores were difficult to interpret clinically.
We also display the importance of individual features as a bar graph for patient examples among the visualized patients (see Supplementary Figure 9). The patient was female, aged 45 to 54 years, and had multiple underlying medical conditions described at admission 2 days (-2 days) before the index culture (blood culture on index day). Our model identified a risk score of 0.541 (predicted as a positive patient). After the patient was admitted, vancomycin and meropenem were started, and blood cultures were ordered. Two weeks later, cultures identified MRSA in her.