Application of receiver operating characteristic analysis to a remote monitoring model for chronic obstructive pulmonary disease to determine utility and predictive value

Abstract

This is a foundational study that applies Receiver Operating Characteristic (ROC) analysis to the evaluation of a chronic disease model that utilizes Remote Monitoring (RM) devices to identify clinical deterioration in a Chronic Obstructive Pulmonary Disease (COPD) population. Background: RM programmes in Disease Management (DM) are proliferating as one strategy to address management of chronic disease. The need to validate and quantify evidence-based value is acute. There is a need to apply new methods to better evaluate automated RM systems. ROC analysis is an engineering approach that has been widely applied to medical programmes but has not been applied to RM systems. Evaluation of classifiers, determination of thresholds and predictive accuracy for RM systems have not been evaluated using ROC analysis. Objectives: (1) apply ROC analysis to evaluation of a RM system; (2) analyse the performance of the model when applied to patient outcomes for a COPD population; (3) identify predictive classifier(s); (4) identify optimal threshold(s) and the predictive capacity of the classifiers. Methods: Parametric and non-parametric methods are utilized to determine accuracy, sensitivity, specificity and predictive capacity of classifiers Saturated Peripheral Oxygen (SpO2), Blood Pressure (BP), Pulse Rate (PR) based on event-based patient outcomes that include hospitalisation (IP), accident & emergency (A&E) and home visits (HH). Population: Patients identified with a primary diagnosis of COPD, monitored for a minimum of 183 days with at least one episode of in-patient (IP) hospitalisation for COPD in the 12 months preceding the monitoring period. Data Source: A subset of retrospective de-identified patient data from an NHS Direct evaluation of a COPD RM programme. Subsets utilized include classifiers, biometric readings, alerts generated by the system and resource utilisation. Contribution: Validates ROC methodology, identifies classifier performance and optimal threshold settings for the classifier, while making design recommendations and putting forth the next steps for research. The question answered by this research is that ROC analysis can provide additional information on the predictive capacity of RM systems. Justification of benefit: The results can be applied when evaluating health services and planning decisions on the costs and benefits. Methods can be applied to system design, protocol development, work flows and commissioning decisions based on value and benefit. Conclusion: Results validate the use of ROC analysis as a robust methodology for DM programmes that use RM devices to evaluate classifiers, thresholds and identification of the predictive capacity as well as identify areas where additional design may improve the predictive capacity of the model.