ISSN - 0973-0958

Pediatric Oncall Journal

Optimal Pathogen - Directed Therapy using the MALDI TOF in Pediatrics 09/03/2018 00:00:00

Optimal Pathogen - Directed Therapy using the MALDI TOF in Pediatrics

Kenice Ferguson-Paul1,2,3, Xinhua Yu4, Bindiya Bagga1,2, Sandra R Arnold1,2.
1Department of Pediatrics,Division of Infectious Diseases, University of Tennessee Health Sciences Center, Memphis, Tennessee, USA,
2Le Bonheur Children’s Hospital, Memphis, Tennessee, USA,
3Department of Infectious Diseases, St. Jude Children’s Research Hospital, Memphis, Tennessee, USA,
4Department of Epidemiology, University of Memphis, Memphis, Tennessee, USA.

Kenice Ferguson-Paul, MBBS; Le Bonheur Children’s Hospital/University of Tennessee Health Sciences Center, 49 N Dunlap St., Rm 157, Memphis, TN 38105, USA.
Background: Our objective was to determine the effect of the MALDI TOF on time to optimal antimicrobial therapy in bloodstream infections (BSI) compared to conventional methods.
Methods: We reviewed all positive blood cultures from February 2012-2014. The control group (CG) had pathogen identification using conventional methods and the intervention group (IG) with MALDI-TOF.
Results: There were 199 cultures (CG, n=121, IG, n=78). The median time to species identification (hours) for the CG was 48, interquartile range (IQR) 37-57 vs. IG=17, IQR 14-23 (median difference 31, 95% Confidence Interval (CI) -38 to -25). Median time to optimal therapy (hours) for the CG was 28, IQR 2-64 vs. IG= 14.5, IQR 3-55 (median difference -13.5, 95% CI -31-13). By Cox regression MALDI TOF was not associated with reduced time to optimal therapy (hazard ratio (HR) 1.2, p=0.64).
Conclusions: Species level identification is faster with MALDI TOF in BSI but does not allow for faster time to optimal antimicrobials.
MALDI TOF, bacteremia, blood culture, children, Vitek, antimicrobials
Matrix-assisted laser desorption/ionization time-of-flight (MALDI-TOF) is a diagnostic modality used in the rapid identification of microorganisms that has been in use since the 1990s. Historically, conventional means of pathogen identification involved the use of the gram stain and biochemical testing (more recently automated using systems such as VITEK), requiring growth of colonies on agar plates. The average turnaround time for these methods traditionally involves at least 24-48 hours of incubation before pathogen identification, thus, creating the need for more rapid means of bacterial species-level identification. Most recently, MALDI TOF has been used to accurately identify the genus and species of microorganisms by direct analysis of positive blood culture broth. (1,2) Among adult patients, this has led to a reduction in length and cost of hospital stay. (1-4)
The objective of the current study was to determine the effect of the MALDI TOF in comparison to Clinical and Laboratory Standards Institute (CLSI)-approved conventional methods on the time to optimal antimicrobial therapy in the management of bloodstream infections in children, in a setting where specific education about the meaning or reliability of the rapid results was not provided to treating medical trainees or physicians. A secondary objective was to determine the impact of the MALDI TOF on the length of hospital stay amongst children with bloodstream infections.

Patients and Methods
This single center, retrospective study was conducted at Le Bonheur Children’s Hospital (LBCH) and was approved by the University of Tennessee Health Science Center Institutional Review Board. LBCH is a tertiary care academic hospital with a level IV neonatal intensive care unit (NICU), as well as pediatric (PICU) and cardiovascular (CVICU) intensive care units. Patients aged 0 to 18 years of age inclusive with positive blood cultures were identified for the period of February 5, 2012 to February 6, 2014. The patients were subdivided into two consecutive time periods: February 5, 2012 to February 5, 2013 during which time blood culture pathogens were identified using conventional methods only (control group) and February 6, 2013- February 6, 2014 during which time validated pathogens were identified by the MALDI TOF in addition to conventional methods (intervention group). Only the first episode of bacteremia was included in the analysis with subsequent episodes of bacteremia during the same admission excluded for simplicity.
Patients were excluded from analysis for the following reasons: cultures drawn at other hospitals prior to presentation at LBCH, patients discharged from the emergency department after a culture was drawn and called back for the positive culture, culture obtained after starting antibiotic therapy, and cultures that grew only a presumed contaminant. Coagulase negative staphylococcus (including Staphylococcus hominis and epidermidis) were usually considered contaminants and excluded from analysis when isolated from blood cultures with the following exceptions: neonatal intensive care unit (NICU) patients, patients with central venous catheters (CVC) or peripherally inserted central catheters (PICC) and patients with other surgically implanted prosthetic materials, devices or hardware. Organisms were further classified as contaminants based on the treating physician’s clinical and laboratory assessments and decision making.

In the LBCH clinical microbiology laboratory, the performance characteristics of the MALDI TOF have been internally validated for identification of common organisms from blood culture broth and bacterial colonies from agar plates. The validation protocol involved the comparison of confidence levels/ Biotyper scores of at least 30 isolates of each bacterial species using both the MALDI TOF and conventional methods. (5) Through this process, MALDI-TOF results are currently reported for 18 bacterial species, 14 of which were reported starting on February 6, 2013 (Enterobacter cloacae complex, Escherichia coli, Klebsiella pneumoniae, Klebsiella oxytoca, Haemophilus influenzae, Moraxella catarrhalis, Pseudomonas aeruginosa, Salmonella species, Staphylococcus aureus, Staphylococcus epidermidis, Staphylococcus hominis, Streptococcus pyogenes, Streptococcus agalactiae and Stenotrophomonas maltophilia) and an additional four starting on September 1, 2013 (Enterococcus fecalis, Serratia marcescens, Proteus mirabilis and Candida albicans). Validation for additional organisms is ongoing.

The time to optimal therapy was defined as the time from the culture being obtained until the patient was on the final antimicrobial regimen deemed optimal by treating physicians due to pathogen susceptibility, patient tolerance and feasibility of transition to oral alternative. Arriving at optimal therapy involved one or more of the following: escalation or de-escalation of therapy to the narrowest spectrum agent possible, adjustments based on patient factors such as allergies or intolerances of prescribed antimicrobials, addition of antimicrobials in the case of polymicrobial infections not due to contaminants and discontinuation of unnecessary antibiotics i.e. stopping gram negative coverage when a gram positive organism is isolated. Those whose initial empiric therapy was already optimal had a “zero time” to optimal therapy. The time to identification of the organism was defined as the time the culture was drawn until the organism was identified by either conventional means or MALDI TOF. Length of stay was defined as the total duration of hospital days, from admission to discharge.
Comparisons were made between patients in the control group and intervention groups. For each patient in both groups the following dates and times were recorded: admission, initiation of first antimicrobials, collection of specimen for culture, pathogen species identification, initiation of empiric and optimal antimicrobials, discontinuation of antimicrobials and discharge. Clinical information that could affect the outcomes such as patient demographics, the presence of comorbidities (e.g. congenital anomalies, surgical conditions) that may affect choice of empiric therapy or duration of antimicrobials and hospitalizations, patient’s diagnosis, type of infection (hospital acquired vs. community acquired, anatomic site) and sites of all cultures taken for the current infection (bloodstream, central venous catheter or PICC line) were collected. Patients were classified as having severe disease if they required any of the following: supplemental oxygen, ICU admission, mechanical ventilation, hemodynamic instability (including hypotension and tachycardia requiring fluid boluses and/or vasopressors), surgical intervention, acute respiratory distress syndrome (ARDS) and death. All others were considered to have non-severe illness.

Statistical analysis
All statistical analyses were performed using SAS 9.3. Descriptive statistics were performed for demographic and clinical characteristics, and the comparisons between the control and intervention groups were performed using the Student’s t-test and median test for continuous variables and Chi-square and Fisher’s exact test for categorical variables. Time-to-event data and length of stay were analyzed using non-parametric methods, specifically, median times are reported with interquartile ranges (IQR) and difference in median times were analyzed using quantile regression and reported with 95% confidence intervals (CI). In addition, time to optimal therapy data were further analyzed using survival analysis including the Kaplan-Meier method with log-rank test statistics, and Cox proportional hazard regression. Those with zero time to optimal therapy were excluded from the survival analysis. In the Kaplan-Meier plots, the event was defined as receiving or switching to the optimal therapy and the time to event curves were interpreted as time to optimal antimicrobial therapy meaning that a shorter time was preferred.
Cox proportional hazards analysis was used to determine which patient factors were associated with reduced time to optimal therapy. The following covariates were assessed: method of pathogen identification (control versus intervention), age (as a continuous variable in years), gender (categorical variable male or female), race and ethnicity (African American, Caucasian or other; Hispanic, Non-Hispanic, unavailable), insurance status (public versus private), comorbid conditions (categorical variable) (Table 1), presence of a CVL or PICC line and patient location (intensive care unit (ICU) vs. non-ICU). Due to the limited sample size, only statistically significant covariates were included in the final multivariate analysis. A hazard ratio with p value <0.05 was considered significant.
There were 1073 positive blood cultures during the study period including episodes of true bacteremia as well as contaminated samples. Once additional exclusion criteria were applied, 199 samples were included in the final analysis (Figure 1). This included samples of both peripheral bacteremia and central line associated bloodstream infections. Among the evaluable patients, 121 were in the control group and 78 were in the intervention group.
There was no difference in the demographic and clinical characteristics between the two groups (Table 1) with the exception of the number of patients with indwelling central lines (control group 42%, intervention group 19%, p=0.0011) and those with any comorbidities (control group 94%, intervention group 75%, p=0.0022). The median age in the control group was 1.2 years (IQR 0-8.1) compared with 1.1 years (IQR 0.1-6.6) in the intervention group. NICU patients represented 20% (39 of 199) and 43% (85 of 199) of the patients were admitted to one of the ICUs (neonatal, pediatric and cardiovascular). There was no difference in overall disease severity between the two groups (p=0.90).
The most common pathogens identified in both groups were Staphylococcus epidermidis, followed closely by Staphylococcus aureus with a predominance of methicillin susceptible Staphylococcus aureus over methicillin resistant Staphylococcus aureus (Table 2). Gram-negative organisms represented 20% (n=66) of isolated pathogens of which 18% were hospital acquired, and 30% occurred in patients with a central venous catheter.
At least one empiric antibiotic was optimal therapy in 45% (54 of 121) of patients in the control group in contrast to 22% (17 of 78) patients in the intervention group (p=0.002). There was no significant difference in the proportion of patients who had changes made to empiric therapy based on microbiologic data [82% vs. 81% (p=0.85)] between the control and intervention groups respectively. Changes made to therapy included the addition of an antibiotic in the case of polymicrobial infections as well as the addition of synergistic antibiotic therapy in the case of endocarditis and osteomyelitis. In contrast, there was a smaller proportion of patients in the control group who had empiric therapy stopped in favor of optimal therapy 55% (67 of 121) vs. 78% (61 of 78) in the intervention group (p=0.002).
Median time to species level identification in the control group was longer (48 hours (IQR 37-57)) than the intervention group (17 hours (IQR 14-23)) for a difference of -31 hours (95% CI-38 to-25). The median time to optimal therapy was 28 hours for the control group (IQR 2-64) vs. 14.5 hours (IQR 3-55) in the intervention group (median difference -13.5, 95% CI -31-13). Kaplan Meier analysis also demonstrated that neither method of species identification was associated with a significantly faster time to optimal therapy (p value by log rank test of difference= 0.02) (Figure 2). Lastly, the median length of hospitalization was not significantly different between the two groups, 11 days in the intervention group (IQR 7-24) vs. 18.5 days in the control group (IQR 8-52), (median difference -7, 95% CI -16-0.6).
Following multivariate analysis by Cox regression, none of the variables was associated with reduced time to optimal therapy: MALDI TOF as the method of blood culture identification (HR 1.2, p=0.64), mechanical ventilation (HR 0.1, p=0.05) insurance status (HR 1.7, p= 0.01), NICU location (HR 0.8, p=0.53) and CVICU location (HR 1.5, p=0.37).

Figure 1. Flowchart of derivation of control and intervention groups for all episodes of bacteremia during the study period
<b>Figure 1.</b> Flowchart of derivation of control and intervention groups for all episodes of bacteremia during the
study period

* Contaminant organisms S. epidermidis (59), S. aureus (4), S. hominis (16) Moraxella catarrhalis (1)

Table 1. Patient characteristics of the Control and Intervention groups
Patient Demographics Control group (Vitek)
(n=121) (%)
Intervention group (MALDI TOF)
(n=78) (%)
Age (years), median (IQR) 1.2 (0-8.1) 1.1 (0.1-6.6) 0.47
    Male 68 (56%) 42 (54%) 0.82
    Black or African American 79 (65%) 48 (61%)  
    Caucasian 26 (21%) 23 (29%) 0.35
    Other 16 (13%) 7 (4%)  
Insurance status      
    Public 92 (76%) 54 (69%) 0.29
    Private 29 (24%) 24 (31%)  
Disease severity      
    Mild illness 69 (57%) 47 (60%)  
    Hemodynamic instability 19 (16%) 11 (14%)  
    Supplemental Oxygen 1 (1%) 1 (1%)  
    Mechanical ventilation 7 (6%) 5 (6%)  
    Surgical intervention 8 (7%) 6 (7%)  
    ICU transfer 8 (7%) 6 (7%)  
    Death 7 (6%) 1 (8%)  
    ARDS 2 (2%) 1 (1%)  
    Yes 114 (94%) 59 (75%)  
    No 7 (6%) 19 (25%) 0.0002
Hemodynamically sig. CHDa 6 (7%) 9 (8%) 0.95
Renal disease 4 (3%) 2 (3%) 0.76
Sickle cell disease 1 (1%) 1 (1%) 0.76
Indwelling catheters
50 (42%) 15 (19%) 0.001
Solid organ transplant 1 (1%) 3 (4%) 0.13
CNS abnormality 5 (4%) 3 (4%) 0.92
Other 110 (92%) 55 (71%) <0.0001
Chronic lung disease of prematurity 7 (8%) 10 (9%) <0.87
Length of stay (median, days) 18.5 (8-52) 11 (7-24) 0.19
Note: *p-value for all levels of disease severity combined
aCongenital heart disease, bCentral venous line, cPeripherally inserted central catheter.

Table 2. Distribution of organisms isolated from blood cultures in all episodes of bacteremia
Organism identified from blood culture Percentage of total
N=332 (%) includes contaminantsa and polymicrobial infectionsb
Staphylococcus epidermidis 112 (34%)
Staphylococcus aureus
85 (26%)
Escherichia coli 31 (9%)
Staphylococcus hominis 22 (7%)
Streptococcus agalactiae 19 (6%)
Other c 15 (5%)
Enterococcus faecalis 14 (4%)
Klebsiella pneumoniae 12 (3%)
Enterobacter cloacae 11 (3%)
Streptococcus pyogenes 7 (2%)
Enterococcus fecium 4 (1%)
a Contaminants included Coagulase negative staphylococcus including Staphylococcus hominis and Staphylococcus epidermidis, Staphylococcus aureus, Enterococcus fecalis, Enterococcus fecium, Streptococcus pyogenes and Moraxella catarrhalis
b Streptococcus agalactiae, E. coli, Staphylococcus aureus, Staphylococcus epidermidis
c Other pathogens include Pseudomonas aeruginosa, Proteus mirabilis, Candida albicans, Hemophilus influenzae, and Klebsiella oxytoca
1. Methicillin resistant Staphylococcus aureus
2. Methicillin susceptible Staphylococcus aureus

Figure 2. Kaplan Meier curves comparing time to optimal therapy between MALDI TOF (M) and conventional methods (C)
<b>Figure 2.</b> Kaplan Meier curves comparing time to optimal therapy between MALDI TOF (M) and conventional
methods (C)

MALDI TOF has been confirmed as an efficient, rapid, diagnostic method in the identification of bacterial and fungal pathogens directly from blood culture media. (2, 5-9) The performance of the MALDI TOF and its clinical impact have been well described in the literature in the adult and mixed pediatric populations. Most recently, Malcolmson et al., published a study outlining the effect of MALDI TOF combined with antibiotic stewardship on the treatment of bloodstream infections in hospitalized children. (10) They saw a significant reduction in time (22.8 hours) to optimal therapy with MALDI TOF plus antibiotic stewardship interventions (from 77 to 54.2 hours, p<0.001). In addition, they noted that time from blood culture collection to organism identification was significantly reduced when comparing the intervention and control groups (18.8 vs. 43.7 hours (p<0.001) (-24.9 h differential). Comparatively, in our current study, time to organism identification in the intervention group was shorter than the aforementioned study, 17 hours vs 18.8. In addition, our time to optimal therapy in the intervention group was also significantly shorter 14.5 hours vs. 54.2 hours in the aforementioned study.
In our study, MALDI TOF resulted in species-level identification of microorganisms 31 hours faster than conventional means of detection. Previous studies have given comparable reduction of identification times of 28.8 hours and greater. (1, 9) These faster turnaround times have led to mixed results as it relates to time to targeted antimicrobials. While we demonstrated a median 13.5-hour time differential to starting optimal therapy in favor of the MALDI TOF, this was not a statistically significant difference.
The shorter time to optimal therapy in our control group may be due to the efficacy of our empiric antibiotics. Standard of care in many tertiary institutions involves initiation of empiric antibiotics based on individual patient historic susceptibility data as well as institutional antibiograms. Previous studies indicate that in areas of low resistance to empiric antimicrobials, empiric therapy tends to be adequate in patients already on therapy at the time of blood culture draw. (1) In our study, 45% of patients in the control group were already on optimal therapy at the time of organism identification indicating that empiric therapy was appropriate. This included cases of methicillin resistant Staphylococcus aureus as well as resistant gram-negative infections. However, we did see a significantly greater proportion of patients in the intervention group having empiric therapy stopped in favor of optimal therapy, which may indicate that the MALDI TOF plays a role in the cessation of unnecessary antibiotics. This assertion will need exploration in future studies.
There have also been mixed results on the impact of MALDI TOF on length of stay. While some studies report reductions in lengths of stay in patients with pathogens identified with the MALDI TOF (11), other studies indicate no significant differences in comparison to conventional pathogen detection techniques. In our study, length of stay was different between the two groups but the difference was not statistically significant. This is likely because the patients with blood stream infections identified during this time period had long stays in hospital indicating complex conditions whose length of stay might not be affected by duration of antibiotic therapy for a single infection. Furthermore, bacteremia, particularly in our vulnerable pediatric population, is often treated entirely on an inpatient basis with parenteral antibiotics and this is independent of method of pathogen identification.
There does exist the possibility that the observed differences in the data favoring the MALDI TOF in our study could have been due to confounding factors for which the data were not adjusted. These include but are not limited to: the efforts of our antibiotic stewardship program (ASP) and infectious diseases consultations. The ASP at our institution was started in 2011 and increased efforts including prospective audit and feedback were initiated around the time of the MALDI TOF. There were, however, no educational ventures by the antibiotic stewardship team specific to MALDI TOF and how to utilize the information produced which could have influenced our outcomes.
Our study is limited in that it is a retrospective analysis and has a fairly small sample size in both the control and intervention groups. Also, MALDI TOF is inherently limited in that it does not provide antimicrobial susceptibility information, which limits the ability of physicians to definitively tailor antibiotics based on preliminary MALDI-TOF results. The use of rapid diagnostics that can identify genetic determinants of resistance of organisms can potentially improve on this limitation. An additional limitation is that our study was the result of the experience of a single institution, which limits generalization of our results to other tertiary institutions.
There are several potential explanations for our inability to detect a statistically significant difference between groups even with our moderately large point estimate. The times to optimal therapy in our control and intervention groups were much shorter than in previous studies as empiric therapy was optimal in a large number of cases. In addition, the range of times-to-event was wide, the reasons for which include not trusting the MALDI TOF results, due to lack of education regarding this methodology and how to utilize the information or because MALDI TOF does not always accurately identify validated organisms, necessitating additional time to identification and more changes in antibiotics. The proportion of polymicrobial infections in our study, albeit small (5%) may have also further inhibited our ability to demonstrate significant differences in time to optimal therapy between the two groups as this would have led to multiple antibiotic changes before optimal therapy was achieved.
Providing training on MALDI TOF, including efficacy of the technique and suggested ways to utilize early results in conjunction with our hospital antibiogram could improve physician response time to microbiologic data and result in clinically significant reductions in the time to prescribing optimal pathogen directed therapy by all providers at our institution.
In conclusion, species level identification is faster with the MALDI TOF in the diagnosis of bloodstream infections in children. However, it did not appear to allow for faster time to administration of optimal antimicrobials in our institution in the absence of specific education surrounding MALDI TOF and how to utilize its results. MALDI TOF did not affect length of stay in patients with bloodstream infections at our institution. Further study is needed to determine how best to utilize this rapid identification technique to optimize prescribing of antimicrobials.
Compliance with Ethical Standards
Funding None
Conflict of Interest None
  1. Vlek AL, Bonten MJ, Boel CH. Direct matrix-assisted laser desorption ionization time-of-flight mass spectrometry improves appropriateness of antibiotic treatment of bacteremia. PLoS One. 2012;7:e32589.  [CrossRef]  [PubMed]
  2. Doern GV, Vautour R, Gaudet M, Levy B. Clinical impact of rapid in vitro susceptibility testing and bacterial identification. J Clin Microbiol. 1994;32:1757-62.  [PubMed]
  3. Huang AM, Newton D, Kunapuli A, Gandhi TN, Washer LL, Isip J, et al. Impact of rapid organism identification via matrix-assisted laser desorption/ionization time-of-flight combined with antimicrobial stewardship team intervention in adult patients with bacteremia and candidemia. Clin Infect Dis. 2013;57:1237-45  [CrossRef]  [PubMed]
  4. Kothari A, Morgan M, Haake DA. Emerging technologies for rapid identification of bloodstream pathogens. Clin Infect Dis. 2014;59:272-8.  [CrossRef]  [PubMed]
  5. Kerremans JJ, Verboom P, Stijnen T, Hakkaart-van Roijen L, Goessens W, Verbrugh HA, et al. Rapid identification and antimicrobial susceptibility testing reduce antibiotic use and accelerate pathogen-directed antibiotic use. J Antimicrob Chemother. 2008;61:428-35.  [CrossRef]  [PubMed]
  6. Drancourt M. Detection of microorganisms in blood specimens using matrix-assisted laser desorption ionization time-of-flight mass spectrometry: a review. Clin Microbiol Infect. 2010;16:1620-5.  [CrossRef]  [PubMed]
  7. Moussaoui W, Jaulhac B, Hoffmann AM, Ludes B, Kostrzewa M, Riegel P, et al. Matrix-assisted laser desorption ionization time-of-flight mass spectrometry identifies 90% of bacteria directly from blood culture vials. Clin Microbiol Infect. 2010;16:1631-8.  [CrossRef]  [PubMed]
  8. Juiz PM, Almela M, Melcion C, Campo I, Esteban C, Pitart C, et al. A comparative study of two different methods of sample preparation for positive blood cultures for the rapid identification of bacteria using MALDI-TOF MS. Eur J Clin Microbiol Infect Dis. 2012;31:1353-8.  [CrossRef]  [PubMed]
  9. La Scola B, Raoult D. Direct identification of bacteria in positive blood culture bottles by matrix-assisted laser desorption ionisation time-of-flight mass spectrometry. PLoS One. 2009;4:e8041.  [CrossRef]  [PubMed]
  10. Malcolmson C, Ng K, Hughes S, Kissoon N, Schina J, Tilley PA, et al. Impact of Matrix-Assisted Laser Desorption and Ionization Time-of-Flight and Antimicrobial Stewardship Intervention on Treatment of Bloodstream Infections in Hospitalized Children. J Pediatric Infect Dis Soc. 2017;6:178-186  [PubMed]
  11. Perez KK, Olsen RJ, Musick WL, Cernoch PL, Davis JR, Land GA, et al. Integrating rapid pathogen identification and antimicrobial stewardship significantly decreases hospital costs. Arch Pathol Lab Med. 2013;137:1247-54.  [CrossRef]  [PubMed]


Cite this article as:
Ferguson-Paul K, Yu X, Bagga B, Arnold S R. Optimal Pathogen - Directed Therapy using the MALDI TOF in Pediatrics. Pediatr Oncall J. 2018;15: 97-102. doi: 10.7199/ped.oncall.2018.38
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
Disclaimer: The information given by is provided by medical and paramedical & Health providers voluntarily for display & is meant only for informational purpose. The site does not guarantee the accuracy or authenticity of the information. Use of any information is solely at the user's own risk. The appearance of advertisement or product information in the various section in the website does not constitute an endorsement or approval by Pediatric Oncall of the quality or value of the said product or of claims made by its manufacturer.
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0