Predicting Transvaginal Surgical Mesh Exposure Outcomes Using an Integrated Dataset of Blood Cytokine Levels and Medical Record Data: Machine Learning Approach

Introduction

Patients experiencing difficulty from symptoms of pelvic organ prolapse (POP) are prescribed nonsurgical option as a first-line treatment, including pelvic floor physical therapy to strengthen the muscles supporting the pelvic organs and the use of pessaries that are inserted into the vagina to provide structural support and alleviate symptoms. Lifestyle modifications are also suggested, such as weight management and avoidance of heavy lifting, to reduce symptoms. However, conservative treatments often fail to provide adequate relief, leading many patients to pursue surgical interventions. Notably, patients who undergo surgical treatments are more likely to report complete satisfaction 1 year after the procedure compared to those who opt for nonsurgical options [1]. The choice of surgical approach depends largely on the type of prolapse. Among these, transvaginal mesh surgery was once a widely used treatment when reinforcement of the pelvic floor structures was deemed necessary to treat severe or recurrent prolapse, though its use was later restricted following safety concerns raised by the Food and Drug Administration. This paper focuses on transvaginal mesh surgery to explore strategies for identifying patient populations that are more likely to experience favorable surgical outcomes, with the aim of minimizing the number of patients who face less successful results.

Surgical treatment via transvaginal mesh implantation uses polypropylene mesh due to its low cost, nontoxicity, and mechanical resilience. Its superior tensile strength and elasticity compared to native pelvic tissue enabled it to withstand the pressure changes within the pelvic cavity [2]. Despite these advantageous properties, the usage of polypropylene mesh in transvaginal repair of POP has been prohibited in multiple countries due to postsurgical complications, notably mesh exposure through the vaginal wall [2,3]. In fact, studies have reported mesh exposure rates ranging from 4% to 12.3% in POP patients following surgery [4-6]. Conversely, however, nearly 90% of patients experience successful POP treatment via transvaginal mesh. Consequently, accurately identifying risk factors and developing personalized predictive models can greatly benefit both patients and surgeons in making informed surgical decisions.

Building a machine learning model to predict postsurgical mesh exposure holds considerable promise, as it offers a robust framework that harnesses large datasets, identifies relevant patterns, and provides quantitative predictions that can be used as clinical decision-making tools. In fact, supervised machine learning models have demonstrated efficacy in predicting various surgical outcomes, including patient survival, symptom improvement, and disease progression [7-9]. This efficacy is particularly evident when using datasets that integrate patient health care information and biological assessments. For instance, Jung et al [7] developed an individualized predictive model incorporating patient health care information from medical records as well as pre- and postoperative laboratory measurements (eg, total bilirubin and sodium levels) to anticipate graft failure in pediatric liver transplant surgery. Similarly, Chowdhury et al [8] applied biomarkers, such as mucus cytokines, along with patient demographics and clinical characteristics to machine learning algorithms to predict chronic rhinosinusitis improvement when treated with endoscopic sinus surgery. Kawakita et al [9] developed a model to predict delayed graft function with kidney transplantation surgery by integrating donor and recipient health care characteristics (eg, BMI, race, age, diabetes diagnosis) and clinical laboratory data (eg, blood urea nitrogen levels, serum glutamic pyruvic transaminase). Despite these successful examples of incorporating demographic, clinicopathological, therapeutic, and biomolecular data into personalized clinical decision-making tools aimed at predicting poor clinical outcomes, such methods have yet to gain acceptance in the current clinical practice of mesh surgery.

In this preliminary study, we explored the benefit of integrating distinct datasets when developing supervised machine learning models to predict mesh exposure in patients who have undergone POP repair surgery involving transvaginal insertion of polypropylene mesh. Models were trained with 3 different datasets: patient medical record data (such as demographics, social history, and medical history), biomaterial-induced blood cytokine levels, and the integration of both datasets. Models trained with these 3 different datasets were able to identify pertinent health risk factors as well as important predictors among the cytokines. Moreover, models trained with the integrated dataset relative to individual datasets demonstrated superior performance. These outcomes illustrate the potential of leveraging integrated datasets with machine learning models to accurately predict transvaginal mesh implantation outcomes and identify crucial risk factors associated with such outcomes. Such predictive models offer invaluable guidance to patients and surgeons. This approach holds potential to enhance the decision-making process for both transvaginal and other mesh surgeries, leading to improved patient care and better surgical outcomes.

Methods

Study Population

As described previously [10], recruited patients with surgical intervention for POP using transvaginal polypropylene mesh included both those with mesh exposure, a postsurgical complication where the implanted mesh protrudes through the vaginal wall, and those without. A cohort of 20 healthy, nonpregnant female patients, divided evenly between the 2 categories, was determined necessary for statistical observation of a difference in blood cytokine biomarkers with a 10% chance of error. Briefly, this statistical analysis used a 1-tailed t test based on a priori power analysis, a Cohen d of 1, and a conservative study power of 0.80.

Participants were selected retrospectively from a list of patients who had undergone surgical intervention for POP using polypropylene mesh at Prisma Health Greenville Memorial Hospital, as detailed in Waugh et al [10]. The patients were randomly identified and recruited via phone call until 10 patients with mesh exposure and 10 without were enrolled for this study. Participants with POP recurrence or those taking medications that could affect inflammatory responses were excluded from the study.

Ethical Considerations

The study protocol was approved by the Institutional Review Board of Prisma Health (number 1984257). Informed consent was waived due to the retrospective nature of the data. All collected data were deidentified and stored in a secure, password-protected folder with restricted access. As this was a retrospective study with no subject participation, no compensation was provided.

Patient Medical Record Data Collection

Medical record data were collected from 20 female patients who had experienced prior surgical intervention for POP using transvaginal polypropylene mesh, selected retrospectively and randomly as described earlier. The data categories include vital signs (systolic blood pressure [BP], diastolic BP, and pulse pressure), previous medical history (hypertension, diabetes, renal disease, hyperlipidemia, hemorrhoids, endometriosis, gastroesophageal reflux, and hysterectomy), social history (alcohol and tobacco usage, sexual activity, marital status, and education level), and other relevant health statistics (age at surgery, BMI, BMI change since surgery, POP stage diagnosis at surgery, and number of births). Mean and SD values were calculated for continuous variables (systolic and diastolic BP, pulse pressure, age at surgery, BMI, and BMI changes) along with the range of the variables (Table 1). Unpaired t tests were conducted on each variable to determine significance (P<.05).

Patient Blood Cytokine Measurements

Patient blood samples were collected from the same 20 female patients and incubated with sterile polypropylene mesh (2 cm × 2 cm), as detailed in Waugh et al [10]. Plasma layers collected following centrifugation (1500xg, 4 °C) of blood samples were analyzed via multiplex assay to quantify cytokine levels, as in Waugh et al [10]. These cytokines include interleukin (IL)-1α, IL-1β, IL-2, IL-4, IL-6, IL-8, IL-10, IL-12 p40, IL-12 p70, IL-17A, interferon-γ, tumor necrosis factor-α (TNF-α), and granulocyte-macrophage colony-stimulating factor. In total, 3 independent measurements were observed to detect the cytokine levels, with each sample evaluated in duplicate.

Data Analysis

Raw datasets, including blood cytokine levels and various medical record data, were analyzed using the statistical programming language R (RStudio, Inc). The imported data contained 60 observations (20 subjects x 3 independent cytokine measurements) and 35 total variable fields (21 medical record variables+13 cytokines+1 target variable). The target variable of exposure referred to the presence or absence of surgical mesh exposure through the vaginal wall following POP surgery. Univariate and multivariate analysis was implemented to explore the dataset, including identifying missing values, analyzing outliers, and standardizing categorical variables.

Supervised machine learning models were created using the caret package (version 6.0‐90) in the R programming language. The 4 models trained were decision tree (DT), logistic regression (LR), Naïve Bayes (NB), and artificial neural network (ANN). This approach focused on three different datasets: (1) patient medical record data, (2) blood cytokine levels following incubation with surgical mesh, and (3) an integrated dataset of both medical record data and blood cytokine levels. Prior to creating the predictive models, the original data were split using an industry standard of 70% for training and 30% for testing. Each group contained an equal distribution of subjects who did or did not experience postsurgical mesh exposure through the vaginal wall, the prediction target for each model. Each model was trained using the 70% (42/60) subset and a cross-validation training control. A 10-fold cross-validation with 25% (15/60) left out replicated 3 times was used on each model to avoid bias and overfitting. From this, training accuracies are reported. Additional testing was performed for the prediction accuracy of each model using the 30% (18/60) test data. Prediction accuracies are reported along with sensitivity and specificity for the prediction of subjects to experience postsurgical mesh exposure. After computing predictive statistics for each dataset, analysis using the varImp function was conducted to identify which variables contributed most to the predictive outcome within each model. This analysis allowed for visualization of individual variable importance regarding each predictive model.

Results

Descriptive Analysis of Patient Medical Record Data

Health care data of patients with POP were collected from electronic medical records and analyzed in Table 1. The cohort included 20 patients with POP, among whom were the 10 that experienced vaginal mesh exposure and 10 that did not. Overall, no significant differences (P<.05) in systolic and diastolic BP, pulse pressure, age at surgery, BMI, and BMI changes were noted between patients with or without mesh exposure. Higher numbers of patients with no mesh exposure had previous medical diagnoses of diabetes and renal diseases. In contrast, higher numbers of patients with mesh exposure reported social histories of alcohol usage as well as sexual activity.

Predictive Analysis

Each of 4 supervised machine learning models, DT, NB, LR, and ANN, was implemented on 3 separate datasets: patient medical record data (21 variables), blood cytokine levels following incubation with surgical mesh (13 variables), and an integrated dataset of both medical data and blood cytokine levels (34 variables). The 5 most important factors that predict mesh exposure in each model are presented in Figure 1.

‎

When the models were trained with patient medical record data alone, history of alcohol usage was the most important factor for NB, LR, and ANN models, and it also accounted for greater than 60% of the importance in the DT model. BMI change was among the 5 most relevant factors in 3 of the models, DT, LR, and ANN. In total, 3 additional factors were among the 5 most relevant factors in 2 of the 4 models: medical history of endometriosis (LR and ANN, >77% importance), POP stage (LR and ANN, >60% importance), and age at surgery (DT and ANN, >40% importance).

When models were trained with blood cytokine levels alone, IL-1β was the most important cytokine to predict mesh exposure in LR and ANN models and among the 5 most important cytokines in NB and DT models, while IL-12 p40 was the most important cytokine in DT and NB models and among the 5 most important cytokines in the ANN model. In total, 2 cytokines exhibited>60% importance in 3 models: granulocyte-macrophage colony-stimulating factor in DT, NB, and LR models and IL-12 p70 in DT, LR, and ANN models. In total, 2 cytokines exhibited>80% importance in 2 models: TNF-α in DT and NB models and IL-6 in NB and ANN models.

Finally, the predictive models were also performed on an integrated dataset of both medical record data and blood cytokine levels. IL-8 was among the 5 most important factors (>70% importance) in 3 models, DT, LR, and ANN. In total, 2 factors were among the 5 most important in 2 models: TNF-α in DT and NB models and hemorrhoids in LR and ANN models. Factors of highest importance in all 4 models included a combination of health care characteristics and blood cytokine levels. For the NB model, the 5 most important factors in the integrated dataset matched those identified to be of highest importance in the individual datasets. In the DT model, 3 factors identified to be of high importance in individual datasets were also of the highest importance in the integrated dataset, while 2 previously unidentified factors rose to high importance within the integrated dataset. In contrast, ANN and LR models identified four unique factors to be of highest importance within the integrated dataset.

Statistics were generated for the performance of each supervised machine learning model (Table 2). Each model achieved the highest prediction accuracy when using the integrated dataset, ranging from 78% (14/18) to 94% (17/18), compared to either medical record data or blood cytokine levels alone, which scored ranges of 33% (6/18) to 50% (9/18) and 50% (9/18) to 83% (15/18), respectively. Similarly, the highest training accuracy for each model was observed when using the integrated dataset, as opposed to either medical record data or blood cytokine levels alone. When comparing different models, both LR and ANN achieved the highest training and prediction accuracies with the integrated dataset. In fact, when the LR and ANN models were trained with the integrated dataset to predict mesh exposure, a training accuracy of 91% (38/42) and a prediction accuracy of 94% (17/18) were observed in combination with sensitivity and specificity ranging from 89% (16/18) to 100% (18/18).

Discussion

Summary

Machine learning is gaining acceptance as a tool for predicting medical conditions, including treatment prognosis, disease progression, and surgical outcomes [7,9,11,12]. In particular, literature reports have illustrated the utility of predictive models that incorporate both health care data and biomarkers [13,14]. These successes lay the groundwork for a pioneering method that integrates patient medical records and biomarker measurements for predicting mesh-related postsurgical outcomes. Patients with POP, in particular, could benefit from such a personalized approach. These patients currently make surgical decisions based solely on risk factors derived from demographic statistics of the general population of surgical patients with POP, while individual medical history and personalized responses to the biomaterial used in the surgery could better inform their decision. Our previous study demonstrated the efficacy of predictive models that use biomaterial-induced blood cytokine-expression profiles to accurately predict mesh exposure following surgical intervention for POP using transvaginal polypropylene mesh [10]. In this exploratory study, we examined the benefit of combining continuous and categorical data from health care records and biomarker measurements to build supervised machine learning models and develop more inclusive and efficient predictive models with a higher rate of prediction accuracy, rendering such models more favorable as a novel, noninvasive, personalized clinical decision tool.

Principal Findings and Comparison to Prior Work

Comparative statistical analysis of medical record data of patients with POP with transvaginal mesh is illustrated in Table 1. Even though this study had a small population of 20 patients, the median outcomes in mesh-exposure and nonexposure groups were comparable to those reported in other studies with a larger study cohort [15,16]. The current dataset revealed that sexually active patients were more prone to experiencing mesh exposure, a trend similarly observed by Kaufman et al [4]. This observation could be attributed to the mechanical challenges posed by sexual activity, potentially leading to friction-induced issues with the mesh implant [17]. In contrast, within the current dataset, variables such as BP, medical diagnoses, education levels, BMI, age at surgery, and parity numbers were not significant risk factors when comparing patients with or without mesh exposure. Similarly, Long et al [18] found no disparity in BMI between the 2 groups. Additionally, Chavez et al [16] noted that risk factors including obesity, diabetes, and smoking status were not associated with vaginal mesh exposure, aligning with our result.

While significant associations were not evident within many individual health care data categories, implementing supervised machine learning models on such data both reinforced groups identified via comparative statistics and revealed new groups associated with patients who experienced mesh exposure. For instance, the history of alcohol usage, indicated as important via comparative statistics, was also considered the most important factor in NB, LR, and ANN models, while BP variables were newly identified as important in the DT model. From a clinical perspective, alcohol use could contribute to malnutrition and inflammation, which may affect the healing process and influence the outcomes of mesh surgery [19]. Similarly, changes in BP and pulse pressure could indicate cardiovascular or systemic issues that may impact surgical planning or postoperative recovery [20]. However, using supervised machine learning models on patient medical record data alone yielded the lowest prediction accuracy rate when compared to models trained with either cytokine levels or an integrated dataset (Table 2). Similarly, low predictive power was reported in Taneja et al [21] when health care data alone was employed in their predictive model [21]. Thus, relying solely on medical record data to construct predictive models does not provide a comprehensive representation. Given an individual’s varying inflammatory response [10,22], as well as their race and genetic components [23,24], merely identifying commonalities among medical record data may not accurately represent predictors of the mesh exposure postsurgery.

Previous studies have indicated a relationship between elevated levels of pro-inflammatory cytokines and postsurgical complications [25-27]. When creating predictive models trained with 13 biomaterial-induced cytokines known to influence wound healing, the models predicted with a higher prediction accuracy of 50% (9/18) to 83% (15/18) compared to models trained with patient medical record data (Table 2). These supervised machine learning models demonstrated that IL-1β was one of the most important cytokines to predict mesh exposure across all four models. Elevated levels of IL-1β have been associated with various diseases, including obesity, cardiovascular diseases, cancer, and periodontitis [28-30]. Moreover, IL-1β was increased in a murine model following hernia polypropylene mesh implantation [31], mirroring our findings. Additionally, IL-12 p40 played a pivotal role in DT and NB models. This pro-inflammatory cytokine, which serves as a critical link between the innate and adaptive immune response, is observed in inflammatory conditions, such as pulmonary sarcoidosis and inflammatory bowel diseases [32-34].

All 4 supervised machine learning models achieved their highest prediction accuracies when trained with an integrated dataset comprising both patient medical record data and biomaterial-induced blood cytokine levels. This integrated dataset proved particularly effective when used in LR and ANN models, achieving 94% (17/18) prediction accuracy. The approach also exceeds predictive health care models for other surgical outcomes that used only cytokine-based [35] or electronic health record-based approaches [36]. In parallel, Taneja et al [21] demonstrated that their sepsis predictive models had higher predictive power when combined data were used, rather than either electronic medical record or biomarker data alone. Interestingly, this integrated approach also identifies new factors of high importance. In contrast to the models trained on cytokine expression alone, models trained with integrated data observed that IL-8 was among the important predictors in DT, LR, and ANN models, and TNF-α in DT and NB models. IL-8 and TNF-α are 2 prominent pro-inflammatory cytokines elevated in patients postsurgery, including mesh implantation and pelvic surgery [37,38]. In contrast to the models trained on medical record data alone, hemorrhoids were considered one of the important factors in LR and ANN models trained with the integrated dataset. Models trained with the integrated dataset also outperformed those trained with either dataset alone in terms of training accuracy, sensitivity, and specificity across all four models (Table 2).

Limitations and Future Directions

Despite employing rigorous research methodologies, it is imperative to acknowledge certain limitations inherent in this study. As a pilot study, the population size was limited to 20 patients within a single hospital system. Even though it was on a small scale, this study yielded a prediction accuracy as high as 94% (17/18), surpassing comparable predictive models that ranged from 71% to 86% in prediction accuracy across a population size of 198-467 [35,39], thus affirming the validity of our approach. Future research will involve a larger participant pool drawn from multiple hospitals to further confirm these results. To mitigate potential confounders, patients with POP recurrence or taking medication that would alter inflammatory response were excluded. Future studies with a larger population will also enable matching participants with respect to these and other potentially confounding variables. In the analysis of the integrated dataset, medical record data were repeated 3 times to match the structure of the cytokine dataset, which contained 3 repeated measurements. This approach was employed to balance the contribution of both data types. This may represent to a potential limitation, although the resulting variable importance graphs for the integrated dataset demonstrated a relatively equal contribution from both the cytokine and medical record data, mitigating concerns that the repetition of the medical record data led to overfitting. Despite these limitations, the results of this pilot study highlight the importance of integrating inflammatory biomarkers with medical record data in the prediction of postsurgical mesh exposure.

Conclusions

Supervised machine learning models demonstrated significantly higher prediction accuracy when trained with an integrated dataset comprising both patient medical record data and biomaterial-induced blood cytokine levels, surpassing the performance of models trained using either dataset alone. These results demonstrate for the first time the advantage of an approach integrating medical record data and blood biomarkers. This innovative approach to predicting surgical outcomes following POP repair via transvaginal insertion of polypropylene mesh holds promise for both patients and surgeons. Additionally, such a method could be applied to the study of other mesh-related surgeries, such as those used to repair POP transabdominally or to treat hernia and stress urinary incontinence. By presenting a more personalized decision-making tool, this approach offers a more accurate depiction of potential outcomes, thereby enhancing informed decision-making and improving patient care.