Article Text
PDF
Abstract
Objective Various risk prediction models are available to stratify patients before non-cardiac surgery and pave the way for anticipative and preventive measures. The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator is an extensive tool that predicts the risk for major adverse cardiovascular events (MACE) and other perioperative outcomes. This study validated the calculator in a Filipino population and compared its predictive ability with the more widely used Revised Cardiac Risk Index (RCRI).
Methods The study included 424 patients referred for preoperative stratification before non-cardiac surgery in St Luke’s Medical Center Quezon City. The development of all-cause mortality, morbidity, pneumonia, cardiac events, venous thromboembolism, urinary tract infection, renal failure and return to operating room were observed. The discriminative ability of the ACS NSQIP to predict these outcomes was evaluated using the area under the receiver operating characteristic curve (AUC) while calibration was measured using the Brier score. The AUC of the ACS NSQIP was compared with that of the RCRI.
Results The ACS NSQIP Surgical Risk Calculator had excellent predictive ability for MACE and was comparable with the RCRI (AUC 0.93 vs 0.93). It also had acceptable predictive ability for pneumonia (AUC 0.93), all-cause mortality (AUC 0.89) and morbidity (AUC 0.88). It had poor to fair predictive ability for renal failure, return to operating room, surgical site infection, urinary tract infection and venous thromboembolism. Calibration was excellent for all-cause mortality, morbidity, pneumonia, venous thromboembolism and renal failure.
Conclusion The ACS NSQIP Surgical Risk Calculator is a valid tool for predicting MACE and other important perioperative outcomes among Filipinos.
- general practice
- primary care
- community cardiology
Statistics from Altmetric.com
Background
Preoperative surgical risk evaluation has evolved through the years. With the longer life expectancy of presurgical patients, there is a need for comprehensive risk calculators that account for their individuality with respect to demographics and comorbidities. Various risk calculators are available to predict the risk for perioperative major adverse cardiac events (MACE), defined as the occurrence of cardiac death or myocardial infarction.1 Among the most frequently used internationally is the Revised Cardiac Risk Index (RCRI), developed in 1999. It is composed of six independent predictors (box 1) for ‘major cardiac complications’, including myocardial infarction, pulmonary oedema, ventricular fibrillation or primary cardiac arrest, and complete heart block.
Revised Cardiac Risk Index (RCRI) predictors and risks for major cardiovascular events according to the number of risk factors.
Independent predictors of major adverse cardiac events (MACE)
High risk surgery (intraperitoneal, intrathoracic, infrainguinal, vascular).
History of ischaemic heart disease (myocardial infarction, positive treadmill exercise test, current chest pain attributed to myocardial infarction, current nitrate therapy, ECG with pathologic Q waves).
History of congestive heart failure (pulmonary oedema, bilateral rales or S3 gallop, paroxysmal nocturnal dyspnoea, chest x-ray showing pulmonary vascular redistribution).
History of cerebrovascular disease (transient ischaemic attack or stroke).
Preoperative treatment with insulin.
Preoperative creatinine >2 mg/dL.
Number of risk factors and risk of MACE
No risk factors—0.5% (95% CI 0.2% to 1.1%).
One risk factor— 1.3% (95% CI 0.7% to 2.1%).
Two risk factors—3.6% (95% CI 2.1% to 5.6%).
Three or more risk factors— 9.1% (95% CI 5.5% to 13.8%).
The RCRI has an area under the receiver operating characteristic curve (AUC) of 0.806±0.034 in predicting MACE.2 However, while it was excellent in predicting the composite outcome of MACE, its performance for predicting all-cause mortality was poor (AUC 0.62).3 Moreover, the applicability of RCRI deserves to be revisited. Its definition of ‘high risk surgery’ includes all intraperitoneal, intrathoracic, infrainguinal and vascular surgeries. With the advancements in minimally invasive surgery, it is now questionable whether laparoscopic or endovascular techniques should still be classified as high risk. This calls for a newer risk prediction model that is specific to the surgical procedure and updated with the evolving advancements of the surgical field: the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator.
Created in 2013, the ACS NSQIP Surgical Risk Calculator is a web-based tool that incorporates 21 preoperative risk factors (table 1) to predict 13 perioperative complications: all-cause mortality, serious complication, any complication, pneumonia, cardiac complication, urinary tract infection, surgical site infection, venous thromboembolism, renal failure, readmission, return to the operating room (OR), discharge to nursing or rehabilitation facility, and length of hospital stay. It is ‘procedure-specific’, accounting for over 1500 unique surgical procedures across different subspecialties: general surgery, orthopaedic surgery, otolaryngology, plastic surgery, cardiothoracic surgery, vascular surgery, urology, neurosurgery and gynaecology, and discriminates open from minimally-invasive procedures. Excluded from this list are ophthalmologic and endoscopic procedures.
American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) variables for preoperative risk factors and predicted perioperative outcomes
The preoperative variables were derived from the ACS NSQIP database consisting of 1 414 006 patients in the USA. Regression models were developed to predict the 30-day perioperative outcomes. The ACS NSQIP Risk Calculator had good predictive capability with an AUC of 0.944 for mortality, 0.816 for morbidity, 0.870 for pneumonia, 0.895 for cardiac events, 0.817 for surgical site infections, 0.806 for urinary tract infection, 0.819 for venous thromboembolism, and 0.903 for renal failure.4 Although it was proven to be valid, it lacked external validation outside of the ACS NSQIP database, particularly in an Asian population.
The ACS NSQIP risk calculator is continuously recalibrated to improve its accuracy as more data are added to the US-based database. The last recalibration was done this 2017. With the updated calculator, it was the primary objective of this study to externally validate it in the Philippines, particularly in St Luke’s Medical Center Quezon City, in order to provide a more comprehensive tool that encompasses the uniqueness of each patient to better prognosticate perioperative outcomes and influence risk–benefit considerations. Differences in the genetic profile of Asians compared with Caucasians, as well as the differences in the level of care in low and middle income countries compared with high income countries, could impact the performance of the calculator, hence the need for this external validation. Furthermore, external validation of the ACS NSQIP calculator in other countries yielded variable results, reiterating the need for validation in different populations. The secondary objective of this study was to compare the ACS NSQIP risk calculator with the RCRI to determine its validity for predicting MACE.
Methods
Population and study design
The study was a single-centre prospective validation cohort study. All patients aged 19 years and older admitted to St Luke’s Medical Center-Quezon City from January 2016 to March 2017, who had been referred for preoperative evaluation and cardiopulmonary risk stratification before non-cardiac surgery, were included.
Surgeries eligible for inclusion included open, laparoscopic and percutaneous abdominal surgeries, anorectal surgeries, breast surgeries, thyroid surgeries, head and neck surgeries, orthopaedic surgeries, urologic surgeries, excision and incision biopsies of superficial masses, wound debridement, vascular surgeries, and neurosurgical procedures. Ophthalmologic and endoscopic procedures were excluded.
Data collection and risk stratification
Each patient that met the inclusion criteria above was assigned a unique code. Two investigators then extracted the demographic data and the following preoperative risk factors via chart review:
Planned surgical procedure.
Functional status:
Independent: able to do all activities of daily living without assistance.
Partially dependent: required some assistance.
Totally dependent: fully-assisted in all activities of daily living.
Emergency case—whether the surgeon deemed the case as emergent.
ASA class—American Society of Anesthesiologists (ASA) class obtained from the anaesthesia record:
Class I (healthy patient).
Class II (mild systemic disease).
Class III (severe systemic disease).
Class IV (severe systemic disease/constant threat to life).
Class V (moribund/not expected to survive surgery).
Steroid use—regular administration of oral or parenteral corticosteroid medications or immunosuppressants within 30 days before surgery.
Ascites—fluid accumulation in the peritoneal cavity noted on physical examination, abdominal ultrasound, or abdominal CT/MRI within 30 days before surgery.
Systemic sepsis within 48 hours before surgery—any of the following:
Systemic inflammatory response syndrome (SIRS)—two or more of the following: (1) fever >38°C or hypothermia <36°C; (2) respiratory rate >24 cycles per minute; (3) heart rate >90 beats per minute; (4) leucocytosis >12 000/µL, leucopenia <4000/µL or >10% bands.
Sepsis—SIRS with proven source of infection.
Septic shock—sepsis with hypotension (systolic blood pressure <90 mm Hg) for at least 1 hour despite adequate fluid resuscitation or need for vasopressors to maintain systolic blood pressure ≥90 mm Hg.
Ventilator dependence—need for ventilator-assisted respiration at any time during the 48 hours before surgery.
Disseminated cancer—presence of a primary cancer that metastasised to a major organ and met at least one of the following:
Cancer was actively treated within 1 year before surgery.
Patient elected not to receive treatment for the metastatic disease.
Patient’s metastatic cancer was deemed untreatable.
Acute lymphocytic leukaemia (ALL), acute myelogenous leukaemia (AML), and stage IV lymphoma were considered disseminated cancers.
Diabetes mellitus—use of exogenous insulin or oral hypoglycaemic agents.
Hypertension—elevated blood pressure requiring antihypertensive medication(s) within 30 days before surgery.
Congestive heart failure (CHF)—symptomatic CHF in the 30 days before surgery.
Dyspnoea—presence of dyspnoea at the time of surgery.
Current smoker—smoking at least one cigarette within 1 year before surgery.
Severe chronic obstructive pulmonary disease (COPD)—emphysema and/or chronic bronchitis resulting in one of the following:
Functional disability.
Hospitalisation for treatment of COPD.
Chronic bronchodilator therapy (oral or inhaled).
Forced expiratory volume in 1 s (FEV1) <75% of predicted.
Dialysis—acute or chronic renal failure requiring treatment with peritoneal or haemodialysis within 2 weeks before surgery; this included patients advised to have dialysis but refused.
Acute renal failure—rapid decline in renal function resulting in increased blood urea nitrogen (BUN) on two measurements and two creatinine results >3 mg/dL.
Using the obtained preoperative variables, the patients were stratified using the RCRI and ACS NSQIP Surgical Risk Calculator (http://riskcalculators.facs.org) as shown in figure 1. Patients admitted in 2017 were risk-stratified using the 2017 calibration of the calculator. As for patients admitted in 2016, they were re-stratified using the 2017 update of the ACS NSQIP Surgical Risk Calculator in order to achieve consistency in risk prediction.
Screenshot of the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator interactive web-based tool. BMI, body mass index; COPD, chronic obstructive pulmonary disease.
A graphic chart of the patient’s individual risk compared with the average risk for developing postoperative complications after a particular surgical procedure was generated (figure 2). Whether the calculator labelled the patient as ‘below average’ risk, ‘average’ risk, or ‘above average’ risk for a particular outcome was noted and encoded into Microsoft Excel.
Sample of perioperative risks generated using the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator. HTN, hypertension; OR, operating room.
Measurement of outcome variables
The patients were observed for the following outcomes within 30 days of the surgical procedure or until discharge from the hospital:
Primary outcome: MACE—cardiac arrest, acute myocardial infarction (ST elevation or non-ST elevation acute coronary syndrome), heart failure.
Secondary outcomes:
All-cause mortality.
Morbidity—any postoperative complication.
Pneumonia— infection of the lungs based on both radiologic and clinical criteria.
Surgical site infection (SSI)—included superficial incisional, deep, or organ space infections.
Urinary tract infection—bladder infection diagnosed using combined clinical and laboratory parameters (eg, urine dipstick or culture).
Venous thromboembolism—identification of a new thrombus within the deep venous system identified through imaging.
Renal failure—rapid rise in creatinine of >2 mg/dL from preoperative value or acute renal failure requiring dialysis; this outcome measure was not applied to those with preoperative acute renal failure.
Return to OR— return to the OR for additional surgery that was not planned at the time of the initial surgery.
Statistical analysis
Categorical data and frequency distribution of patients according to risk factors were expressed using percentage. Those with ‘average’ and ‘above average’ risk for developing a particular outcome were grouped into one category labelled ‘above average’ risk since according to the ACC/AHA Perioperative Clinical Practice Guidelines, the recommendations for both risk categories are similar and their merging into one category simplifies the recommendations.1 Differences in the frequency of outcomes between those with ‘below average’ and ‘above average’ risk levels were compared using a Χ2 test. Differences in mean values such as age were compared using the Student’s t-test.
The accuracy of the ACS NSQIP Surgical Risk Calculator was measured using discrimination and calibration.5 Discrimination measures how well the calculator will be able to distinguish those who will be stratified as ‘average’ to ‘above average’ risk for MACE who will eventually develop MACE from those who will be stratified as ‘below average’ risk for MACE and not develop it. It is measured by the area under the receiver operating characteristic curve (AUC), which plots the true positive rate (sensitivity) against the false positive rate (1−specificity). A perfect risk calculator will be able to stratify all those who will develop MACE as ‘average’ to ‘above average’ risk and stratify those who will not develop MACE as ‘below average’ risk. The discriminative ability of the ACS NSQIP Surgical Risk Calculator was considered excellent if the AUC was 0.90 to 1.0, good if 0.80 to 0.89, fair if 0.70 to 0.79, poor if 0.60 to 0.69, and fail if 0.50 to 0.59. The AUC curves were generated using the Johns Hopkins web-based calculator for receiver operating characteristic curves.6 Statistical differences between the two AUC curves were then compared using P value.
Calibration or the agreement between observed and predicted outcomes was tested using the Brier score. The Brier score ranges from 0 to 1. The closer the score is to 0, the more accurate is the risk prediction model. A risk prediction model with a score of 0.25 and higher is considered non-informative while scores approaching 1 are inaccurate.
Ethical considerations
The study protocol was submitted and approved by the Institutional Ethics Review Board of the Research and Biotechnology Division of St Luke’s Medical Center. Confidentiality of patient identity was respected through the use of designated codes. All research documents were secured in a private location and accessible only to the study investigators. Spreadsheets used for encoding data did not contain patient identifiers apart from their unique codes and were encrypted with passwords only known to the investigators. Informed consent was not required for this study since there was no direct contact between the investigators and the patients. All data were obtained via chart review.
Results
Demographics
A total of 424 patients were included in the study. The distribution of patient characteristics and types of surgery performed are presented in table 2.
Distribution of patients’ preoperative characteristics and types of surgeries performed
The ACS NSQIP Surgical Risk Calculator classified 385 (91%) patients as ‘below average’ risk for MACE while the RCRI classified 369 (87%) patients as ‘low risk’. The ACS-NSQIP classified 19 (4%) patients as ‘average’ risk for MACE, and 20 (5%) patients as ‘above average’ risk for MACE. For the RCRI, patients with two or more risk factors were classified as ‘elevated risk’ and these constituted 55 (13%) of the patients.
MACE occurred in 12 (3%) of patients. Of these, seven occurred after abdominal surgery, three after abdominal pigtail insertion, and two after otolaryngology procedures. The mean age for patients with MACE was 67 (SD 16.46) years. Ten out of the 12 deaths were due to septic shock secondary to hospital acquired pneumonia, one was due to fatal arrhythmia secondary to severe hyperkalaemia, and one due to myocardial infarction.
Among the secondary outcomes, pneumonia was most frequent at 14%. Of note, 75% of those who underwent video-assisted thoracic surgery, 38% of those who underwent otolaryngology procedures, 21% of those who underwent open abdominal surgery versus only 2% of those post laparoscopic abdominal surgery developed pneumonia. Those who developed pneumonia were older (mean age 64.95 vs 53.04 years, P<0.0001) with higher frequency of steroid use (25% vs 9%, P=0.0004), ascites (20% vs 10%, P=0.029), sepsis (42% vs 5%, P<0.0001), preoperative ventilatory support (32.7% vs 3.2%, P<0.0001), cancer (45% vs 21%, P=0.0001), diabetes mellitus (65% vs 37%, P=0.0001), hypertension (87% vs 65%, P=0.0011), previous cardiac event (12.7% vs 3.8%, P=0.0046), congestive heart failure (16.4% vs 1.6%, P<0.0001), dyspnoea (45% vs 3.5%, P<0.0001), positive smoking history (27% vs 14%, P=0.0135), COPD (18% vs 8%, P=0.0173), dialysis requirement (25% vs 2%, P<0.0001), and acute renal failure (18% vs 1.6%, P<0.0001). Thus, each preoperative variable in the ACS-NSQIP was associated with a higher risk for postoperative pneumonia.
Performance of the ACS NSQIP Surgical Risk Calculator
The distribution of the risk classification of the ACS NSQIP calculator for the primary and secondary outcomes are listed in table 3. The sensitivity of both ACS NSQIP and RCRI for detecting MACE was low, 41.7% for ACS NSQIP versus 33.3% for RCRI, while their specificity for MACE was high, 96.4% for ACS NSQIP versus 95.6% for RCRI. The ACS NSQIP had a calculated AUC of 0.93±0.03 for predicting MACE versus the RCRI, which had an AUC of 0.93±0.02 (P=0.98), indicating that both risk prediction tools have excellent discriminative ability and are comparable in predicting MACE (figure 3). Calibration of the ACS NSQIP with respect to MACE was likewise excellent with a Brier score of 0.08.
Distribution of the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) stratification for the primary and secondary outcomes and frequency of actual events
AUC of the (A) ACS NSQIP Risk Calculator and (B) RCRI for predicting MACE. ACS, American College of Surgeons; AUC, area under the receiver operating characteristic curve; MACE, major adverse cardiac events; NSQIP, National Surgical Quality Improvement Program; RCRI, Revised Cardiac Risk Index.
Its performance in predicting the secondary outcomes were likewise assessed using the AUC (table 4). It had excellent predictive ability for pneumonia (AUC 0.93, SD 0.0176) and good predictive ability for all-cause mortality (AUC 0.89, SD 0.0628) and morbidity (AUC 0.88, SD 0.0342). It had fair predictive ability for renal failure and return to the OR. However, for surgical site infection, urinary tract infection and venous thromboembolism, its predictive ability was poor.
Summary of the predictive ability of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator to predict the secondary outcomes
Calibration of the ACS NSQIP was also excellent for all-cause mortality, morbidity, pneumonia, venous thromboembolism and renal failure as indicated by the Brier Score in table 4. Calibration was poor for surgical site infection, urinary tract infection and return to OR, with Brier Scores approaching and exceeding 0.25.
Statistical power
Post-hoc power analysis was done using the OpenEpi Calculator version 3. Using a 95% CI, the study is adequately powered (>80%) for the primary outcome of MACE, mortality, morbidity, pneumonia, renal failure and return to OR. It does not have adequate power (<80%) for surgical site infection, urinary tract infection or venous thromboembolism.
Discussion
For a comprehensive preoperative risk stratification that predicts not only MACE but also important post-surgical in-hospital morbidity, the ACS NSQIP Surgical Risk Calculator can be used. Its calculated AUC for predicting MACE in the present study is 0.93, comparable with that of RCRI. This surpassed the original published AUC of 0.895 for cardiac events during the initial derivation and validation of the calculator.4 The ACS NSQIP, however, has a low sensitivity for detecting MACE at 41.7%. It is worth noting that this low sensitivity is comparable with that of RCRI (33.3%), which is currently the more widely-used preoperative stratification tool. Despite the low sensitivity, the excellent discriminative ability and calibration of the ACS NSQIP Surgical Risk Calculator makes it a powerful tool for preoperative risk stratification. Furthermore, each ACS NSQIP preoperative risk factor was associated with an increased risk for pneumonia, one of the most significant causes of perioperative morbidity as reiterated in this study.
For the secondary outcomes, this study had enough predictive power to make conclusions for all-cause mortality, morbidity, pneumonia, renal failure and return to OR. The calculated AUCs were 0.89 for all-cause mortality and 0.88 for morbidity, thereby reflecting good predictive ability; the AUC for pneumonia was 0.93, also reflecting excellent predictive ability. These results are likewise congruent with the results of the original ACS NSQIP dataset. Thus, this study confirms the usefulness of the ACS NSQIP Surgical Risk Calculator as a valid risk prediction model for MACE, all-cause mortality, morbidity and pneumonia. The calibration for these outcomes were also excellent.
For renal failure and return to OR, the ACS NSQIP Surgical Risk Calculator only had fair discriminative ability with AUCs of 0.76 and 0.78, respectively. In the original dataset, the predictive ability was good to excellent. Despite having good power to make generalisable conclusions, the calculated predictive ability may not be reliable due to the very low events rate for these outcomes. Similarly, the events rate for surgical site infection, urinary tract infection and venous thromboembolism were very low (n <10), contributing to the low power of the study for these outcomes. When the number of actual events is smaller than the number of predictors in a risk model, overfitting can ensue, resulting in the underestimation of the event among those labelled as low risk and overestimation among those labelled as high risk.7 Hence, definitive conclusions regarding the predictive ability of the ACS NSQIP Surgical Risk Calculator for renal failure, return to OR, surgical site infection, urinary that infection and venous thromboembolism could not be made and could only be addressed by increasing the sample size and corresponding events rate of these outcomes.
One limitation of the ACS NSQIP Surgical Risk Calculator is that it can only be accessed through the ACS NSQIP webpage. There is no simple formula that can generate the risk scores because it is ‘procedure-specific’ and has different cut-off values for each unique surgical procedure. Another limitation of this study is the need for more outcome events to reliably evaluate the capability of the tool to predict the other secondary outcomes.
Conclusion
The ACS NSQIP Surgical Risk Calculator is a valid risk model that can be used to predict postoperative MACE, all-cause mortality, morbidity and pneumonia, which can lead to more informed risk–benefit considerations. An extension of this study with more outcome events is necessary to further validate the calculator’s predictive ability for renal failure, return to OR, surgical site infection, urinary tract infection and venous thromboembolism.
Key messages
What is already known about this subject?
The ACS NSQIP Surgical Risk Calculator is an excellent predictor of perioperative complications including mortality, morbidity, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure and return to operating room. It has been validated with good predictive ability using the ACS NSQIP database but has not been externally validated, especially in an Asian population.
What does this study add?
This study externally validated the ACS NSQIP Surgical Risk Calculator and proved that it is a useful tool for preoperative risk stratification in the Philippines setting.
How might this impact on clinical practice?
This study supports the use of the ACS NSQIP Surgical Risk Calculator as a tool to ascertain the risk for perioperative complications before non-cardiac surgery. It is more comprehensive than the widely-used RCRI, predicting not only MACE but also all-cause mortality, morbidity and pneumonia, among others.
Footnotes
Contributors Conception or design of the work: KCY, KFA, LAG-P, EE. Data collection: KCY, KFA, LAG. Data analysis and interpretation: KCY, KFA, LAG. Drafting the article: KCY. Critical revision of the article: KCY. Final approval of the version to be published: KCY, KFA, LAG,EE.
Funding There was no specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval St Luke’s Medical Center Research and Biotechnology Division Institutional Ethics Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement A draft of our paper is available in the archives of the Research and Biotechnology Division of St Luke’s Medical Center as part of the hospital’s policy.