Page 1 of 16

European Journal of Applied Sciences – Vol. 11, No. 1

Publication Date: January 25, 2023

DOI:10.14738/aivp.111.13891. Mangia, C. M. F., Toledo, M. D., Rossi, R., Nakano, E. Y., Carneluti, A., Kopelman, B. I., Carvalho, W. B., & Andrade, M. C. (2023).

Performance of Brazilian Pediatric Risk of Severity Model for Illness (Br PRISM) Compared to Pediatric Index of Mortality and

Pediatric Risk of Mortality 2. European Journal of Applied Sciences, Vol - 11(1). 287-302.

Services for Science and Education – United Kingdom

Performance of Brazilian Pediatric Risk of Severity Model for

Illness (Br PRISM) Compared to Pediatric Index of Mortality and

Pediatric Risk of Mortality 2

Mangia, C. M. F. MD, MSc, MBA, PhD.

Pediatric Critical Care Division, Escola Paulista de Medicina,

Universidade Federal de São Paulo, Brazil

Toledo, M. D., MD.

Pediatric Critical Care Division, Escola Paulista de Medicina,

Universidade Federal de São Paulo, Brazil

Rossi, R., MD.

Pediatric Critical Care Division, Escola Paulista de Medicina,

Universidade Federal de São Paulo, Brazil

Nakano, E. Y.,PhD.

Statistics Department, Universidade de Brasília, Brazil

Carneluti, A., MD.

Faculdade de Medicina, FMABC, Brazil

Kopelman, B. I., PhD.

Pediatrics Department Escola Paulista de Medicina,

Universidade Federal de São Paulo, Brazil

Carvalho, W. B., PhD.

Pediatric Critical Care Division, Universidade de São Paulo

Andrade, M. C., MD, MSc, PhD.

Pediatric Nephrology Division, Escola Paulista de Medicina,

Universidade Federal de São Paulo, Brazil

ABSTRACT

Introduction: The best prognosis score with which to evaluate high-risk patients

upon admission into pediatric intensive care is not well established in resource- limited settings. The objective of study was to formulate a risk-of-illness severity

model for pediatric mortality to be applied upon PICU admission in resource- limited settings. Methods: Our study was designed to develop an illness severity

index and a prognostic model for critically ill children. A prospective, observational

multicenter pilot study, performed between February 1995 and October 1999,

evaluated the variables, methodology and statistical techniques for the

development of a model. A single-center prospective cohort study, performed

Page 2 of 16

288

European Journal of Applied Sciences (EJAS) Vol. 11, Issue 1, January-2023

Services for Science and Education – United Kingdom

between November 1999 and October 2004, collected information from

consecutive admissions into the Pediatric Intensive Care Unit (PICU) at a high- complexity university, teaching, and reference hospital in São Paulo, Brazil. Results:

In the pilot study, 1,459 patients (a PICU mortality rate of 16%) were included, and

in the second study, 1,033 patients (a PICU mortality rate of 13.9% and a hospital

mortality rate of 6.9% after PICU discharge) were included. We used multivariable

regression to determine two probabilistic models; the first addressed survival and

the overall probability of death (hospital plus PICU deaths), and the second was

conditional (i.e., PICU death). An illness severity index stratified these probabilities

into three risk strata: low-, medium- and high-risk patients. In the final step, the

new death probabilities were estimated using a Bayesian adjustment. Conclusions:

The model estimates three probabilities (survival, death in the PICU and death in

the hospital after PICU discharge) stratified into three risk categories. To the best

of our knowledge, this is the first study using a Bayesian adjustment to determine a

prognosis and illness severity, and it should enable us to make therapeutic

adjustments and provide appropriate counseling for high-risk patients in resource- limited settings.

INTRODUCTION

Prognostication efforts are important steps towards understanding the effects of diseases,

medical interventions, and healthcare policies as determinants of outcomes. Mortality risk

models enable the evaluation of the healthcare system, management capacity and quality of

care and facilitate evidence-based decision-making and better resource allocation [1,2,3].

The outcome in intensive care depends on several factors associated with the patient in the first

24 hours after admission and the disease course during the intensive care stay. Severity scores

are usually comprised of two parts: a severity score, which is a number (in general, a high score

reflects a more severe condition), and a probability model, which is an equation that expresses

the probability of death in the hospital or intensive care unit (ICU) [3,4,5].

No consensus about the classification of score systems to be used in the ICU has been reached;

they could be used once or repeatedly over time. There are numerous examples of score

systems, but the main systems are scores based on abnormalities in the physiological variables

measured in the first 24 hours (APACHE, PRISM, PIM) or organ-specific scoring, in which the

main prognostic factors are the number and duration of multiple-organ failures (SOFA, PELOD).

The Brazilian healthcare system is a predominantly public enterprise with universal access for

all citizens. Over the past few years, as part of the Millennium Development Goals for the

reduction of child mortality, new pediatric intensive care units (PICUs) like those in other areas

of the world [6] have been introduced.In 1998, the Brazilian Ministry of Health suggested using

the Pediatric Risk of Mortality (PRISM) [7] score to assess the severity of illnesses and mortality

risks and evaluate PICU performance. Studies have reported that this score is not suited for

critically ill children in resource-limited settings [8], and it is well recognized that performance

scores are variable because the case mix, therapy and selection of patients admitted into the

PICU differ over time. Indeed, PRISM has been outdated for more than 10 years and,

consequently, is obsolete [1,3,4,5].

Page 3 of 16

289

Mangia, C. M. F., Toledo, M. D., Rossi, R., Nakano, E. Y., Carneluti, A., Kopelman, B. I., Carvalho, W. B., & Andrade, M. C. (2023). Performance of

Brazilian Pediatric Risk of Severity Model for Illness (Br PRISM) Compared to Pediatric Index of Mortality and Pediatric Risk of Mortality 2. European

Journal of Applied Sciences, Vol - 11(1). 287-302.

URL: http://dx.doi.org/10.14738/aivp.111.13891

Our main objective was to formulate a risk-of-illness severity model for pediatric mortality to

be applied upon PICU admission in resource-limited settings [3].

METHODS

The Brazilian Pediatric Risk of Severity Model for Illness (Br PRISM) study was developed in

two steps. The first study was a prospective, multicenter, 2 teaching hospitals and 1 private

hospital), cohort study performed between February 1995 and October 1999 and included

1,450 patients. This study evaluated the variables, methodology, and viability of performing a

multicenter study in Brazil and the statistical techniques required for developing a scoring

system and probability model. The second, a validation study, was a single-center, prospective,

observational cohort study performed between November 1999 and October 2004 and

included 1,100 consecutive patients admitted into the Hospital São Paulo from Universidade

Federal de São Paulo, Brazil. Hospital São Paulo, a resource for five million inhabitants, is a high- complexity hospital affiliated with the university teaching medical school. The hospital has 700

beds (80 pediatric) and receives 530,000 emergencies, 32,264 admissions and 163,305

surgeries per year. The pediatric intensive care unit (PICU) has 8 beds and admits medical and

surgical patients between the ages of 0 and 19 years. In the study period, we had 16 pediatric

intensive care residents each year. The medical staff included 2 physicians during the day, 1

physician at night and a total staff of 18 pediatric intensivists (including weekends). In addition,

the nurse-to-patient ratio was 1 nurse to each of 3 beds and one physiotherapist to 8 beds.

Sample Selection

All the consecutive admissions of patients under the age of 19 were analyzed, except the

following: a) patients with a PICU stay of less than 24 hours; b) patients admitted while

receiving continuous cardiopulmonary resuscitation without stable signs for at least 2 hours;

c) brain-dead patients admitted for organ donation. For those patients with multiple PICU

admissions during the same hospital stay, only the data from the first admission were analyzed.

Re-admissions were analyzed if they occurred more than 30 days after PICU discharge. To

determine the outcome, the patients were followed up until they were discharged from the

hospital. Any patients remaining in the hospital after October 31, 2004, were excluded from the

study because their status could not be assessed. The study was approved by the institutional

ethics committee, and parental consent was obtained in all cases.

Variable Selection:

The variables were selected based on our past experience with first study, clinical judgment,

interviews with the intensivists and score review and considered a wide range of citations on

the literature, such as the Pediatric Risk of Mortality II [7] and III [9], the Acute Physiology, Age,

and Chronic Health Evaluation III [10], the Simplified Acute Physiology Score II [11], Mortality

Probability Models II [12], the Pediatric Index of Mortality [13] and the Acute Physiology, Age,

Chronic Health Evaluation II [14]. All the data were collected by the main investigator.

The physiological variables that were eligible for analysis are as follows: the systolic blood

pressure, the heart rate, the respiratory rate, the axillary temperature, any pupillary reactions,

the coma status, diuresis, arterial gasometry, the PaO2/FiO2 ratio, glucose, potassium, sodium,

creatinine, urea, hemoglobin, hematocrit, platelet count, the white blood cell count, and the

prothrombin and activated partial thromboplastin times. The other, non-physiological,

variables are as follows: age, age group, gender, the in-hospital location before PICU admission,

Page 4 of 16

290

European Journal of Applied Sciences (EJAS) Vol. 11, Issue 1, January-2023

Services for Science and Education – United Kingdom

any comorbidities, the clinical or surgical status, the diagnosis by system and etiology (during

the first 24 hours), the use of vasoactive drugs, the use of external oxygen or mechanical

ventilation, the length of stay (LOS) before and after the PICU, the PICU LOS, the total length of

the hospital stay and any outcome data (the vital status in the PICU and upon hospital

discharge) [13,14,15,16,17,18].

The physiological variables were collected upon admission, and, if the laboratory had missed

any biochemical data, we recorded the worst value achieved in the first 24 hours according to

the strict definitions of the previously established variables (Appendix 1). When these values

were age-dependent, we used the range limits of the normal physiological values by age group.

We developed a comprehensive instruction manual, which described all the procedures that

led to the data collection and definition. This manual was based on the evidence in the literature

and included a full description of the study and strict definitions of the variables, their codes,

and, when applicable, their units and normal ranges according to the age group. The age group

was based on the recommendations of the Ministry of Health, which established the following

risk-specific age groups for Brazilian children: less than 12 months, between 12 and 59 months,

between 60 and 119 months, between 120 and 179 months and between 180 and 228 months

[19].

The data were collected in a clinical report form (CRF), the variables were codified, and the

internal quality of the data was checked before keyboarding into the ACCESS® database that

was specially created for this study. The program checked for any out-of-range data using a

logical error system and compiled a report regarding any inconsistent data for each patient.

The quality control of the database included double-keyboarding by two trained and

independent physicians. The first and second sets of keyboarding were compared to the CRF to

determine the reliability of the data from the first and second procedures. The reliability of the

data was compared to that of the CRF and the medical record. To determine the diagnostic

category after PICU admission, we developed a list of the ten major categories of clinical

diseases and nine major categories of surgical interventions. For each major category, we

developed a list of 124 etiology classes according to the age group and the epidemiology of the

pediatric diseases [9,11,12]. The same procedure was used for the categorization of the

comorbidities [15].

Statistical Analysis

The demographic data were represented as absolute numbers and percentages, and the

continuous variables were represented as medians and interquartile ranges. A p-value of less

than 0.05 was significant.

We used multivariable regression to determine two probabilistic models; the first addressed

the probability of death in the PICU, and the second was conditional (i.e., the probability of

death in the hospital after the PICU stay) [20,21,22].

We eliminated variables from the models by backward deletion. These two models produced a

of probabilities for each patient. The first element focuses on hospital survival, the second

focuses on death in the hospital after PICU discharge, and the third probability focuses on the

death probability during the PICU stay. Based on these three probabilities, we created a

severity index that stratifies patients from the worst (PICU death) to best (hospital survival)

Page 5 of 16

291

Mangia, C. M. F., Toledo, M. D., Rossi, R., Nakano, E. Y., Carneluti, A., Kopelman, B. I., Carvalho, W. B., & Andrade, M. C. (2023). Performance of

Brazilian Pediatric Risk of Severity Model for Illness (Br PRISM) Compared to Pediatric Index of Mortality and Pediatric Risk of Mortality 2. European

Journal of Applied Sciences, Vol - 11(1). 287-302.

URL: http://dx.doi.org/10.14738/aivp.111.13891

outcomes according to a previously published method [23,24]. The a priori probabilities of

model were: 10% (death in the PICU), 5% (death in the hospital after PICU stay) and 85%

(hospital survival) [23,24]. As a final step, the Bayesian method was applied to estimate the

new adjustment of probabilities (a posteriori) using the severity index. Statistical analysis was

performed using SPSS (version 11.0) and Excel 2000.

RESULTS

The variables were collected from 1,100 patients, 67 patients were excluded (37 patients were

discharged within 24 hours after PICU admission, 2 patients were still hospitalized at the end

of the study, 17 patients died within the first 24 hours after PICU admission, and 11 patients

were admitted into the PICU for organ donation after brain death). Following the exclusions,

1,033 patients were included in the development of the model. The patients’ characteristics are

summarized in Table 1. Comorbidities were present in 73.9% of the patients. The main

comorbidities were as follows: congenital cardiac disease (21.2%), chronic neurological

disease (10.6%), chronic renal disease (7.6%) and chronic pulmonary disease (6.2%).

Table 2 presents the logarithm of the first regression analysis, and Table 3 presents the log of

the second regression analysis.

The severity index (SI) was calculated using the following equation: SI = (21⁄2 +1)Pr(U) -

(21⁄2)Pr(H), where Pr(U) is the probability of dying in the PICU, and Pr(H) is probability of dying

in the hospital after the PICU stay (table 4). The cutoff value was £ 0.15 for the SI of the

survivors, 0.16 to 0.30 for hospital mortality after the PICU stay, and 3 0.30 for PICU mortality.

Next, we re-adjusted the probabilities using 3 severity classes based on the cutoff points of the

index (high, medium, or low probability of death). The index demonstrated good differentiation

among the 3 severity classes (p<0.001). The area under the ROC curve (AUC) for the survivors

(index £ 0.15) was good (0.821; 95% CI, 0,789 - 0,854). The cutoff point was 0.1564 (sensitivity,

0.738; 1-specificity, 0.265). The Hosmer-Lemeshow goodness-of-fit chi-squared value for the

survivors was 22.154 with 8 degrees of freedom (p=0.005).

The AUC for death in the PICU (index 3 0.30) was good (0.746; 95% CI, 0,676 - 0,817). The

cutoff point was 0.3058 (sensitivity, 0.674; 1-specificity, 0.324). The Hosmer-Lemeshow

goodness-of-fit chi-squared value for the deaths in the PICU was 9.300 with 8 degrees of

freedom (p=0.318). The a posteriori probabilities for each diagnostic category (hospital

survival, patient death in the hospital after the PICU stay and patient death during the PICU

stay) and the risk strata are presented in Table 5.

Br PRISM Compared to PIM and PIM 2

We compared the performance of Br PRISM to two scores with free access in the literature. The

Pediatric Index of Mortality (PIM; versions 1 and 2) met this criterion. The PIM and PIM 2 scores

were collected for 387 patients. The standardized mortality rate (SMR) for the PIM score was

2.464 (95% CI, 1.413 – 3.515), and the odds ratio was 0.56 (95% CI, 0.33 – 0.95); the SMR for

the PIM 2 score was 2.526 (95% CI, 1.366 – 3.687), and the odds ratio was 0.94 (95% CI, 0.57 –

1.57). The area under the ROC curve was 0.882 (95% CI, 0.846 - 0.913) for Br PRISM, 0.736

(95% CI, 0.689 - 0.7790) for PIM and 0.720 (95% CI, 0.672 - 0.764) for PIM 2. The pairwise

comparison of the ROC curves for Br PRISM vs. PIM and PIM2 showed a difference between the

Page 6 of 16

292

European Journal of Applied Sciences (EJAS) Vol. 11, Issue 1, January-2023

Services for Science and Education – United Kingdom

area under ROC curve (0.146; 95% CI, 0.054 - 0.238; p = 0.002). The areas of Br PRISM vs. PIM

2 were different (0.163; 95%, CI 0.066 - 0.260; p = 0.001). The areas of PIM vs. PIM 2 were also

different (0.017; 95% CI 0.044 -0.077; p = 0.591).

DISCUSSION

We undertook this study to develop an illness severity index and a prognostic risk model for

critically ill children that will be useful and relevant to middle-income environments. This

model is based on variables that are easily collected [25] at the bedside and includes well- defined [26] variables that are selected a priori. The investigators collecting the data were

blinded to the study objectives, and continuous, rigorous monitoring was done to eliminate the

possibility of missing [26,27] information to guarantee a high-quality database [28].

The laboratory variables were collected upon admission; however, when the sample was lost

or when technical problems arose during the processing of the samples by the laboratory, the

worst value during the first 24 hours was recorded [9]. We adopted this criterion because, in

our practice, the first sample was susceptible to loss (breakdown of the sample bottle, for

example), or there was a lack of the reagents with which to process the sample immediately or

during sampling [16].

We defined any admission that occurred after thirty days after PICU discharge as a new

admission. This admission would most likely be the result of a new clinical indication and,

therefore, unlikely to be due to an inappropriately early discharge [29]. Additionally, in our

medical practice, mortality in the PICU is an inadequate measure with which to evaluate the

outcomes of a critical disease. The inclusion of the hospital mortality after PICU discharge adds

a new element to the prediction and improves our knowledge regarding outcomes outside the

PICU, which may have a direct bearing on PICU care or the early discharge of unstable patients

[29].

By analyzing the regression models, we found that the overall survival depended more strongly

on the physiological variables. However, the biochemical abnormalities in the conditional

model were determinants of a major risk for dying in the hospital after PICU discharge.

The model supplied a vector of probabilities with three components (survival, PICU death and

hospital death after PICU stay). However, simultaneous interpretations of these data were

deemed to be too complex to explain to families and healthcare providers. Therefore, the

severity index simplifies the information because only one probability is necessary to explain

the gravity of each case [30,31].

The model described in this study don ́t need revised by new validation because the inclusion

of new patients in the database and the modification of its initial information the model will be

auto adjusted. The attraction of the Bayesian model is that it is a dynamic model and superior

to the information provided by the previous risk scores.

Comparing Br PRISM to the PIM and PIM 2 scores showed that the PIM and PIM 2 scores

overestimated the mortality in the high- and very high-risk bands and underestimated the

mortality in mild- and low-risk bands. The ROC curve analysis demonstrated that both had low

sensitivity and specificity in our population [32]. These observations could be explained by the