Validation of a new reference standard to improve the accuracy of validating and comparing automated respiratory rate counting devices to aid diagnosis of childhood pneumonia
Pneumonia is the leading infectious cause of death in children under five. As per current World Health Organization (WHO) guidelines, community health workers across low-resource settings currently assess a child’s respiratory rate as a proxy sign for pneumonia. However, community health workers often find it difficult to accurately count breaths manually. Automated respiratory rate counters, such as the Child Automated Respiration Monitor (ChARM) developed by Philips, offer a potential solution. To introduce new tools, performance must first be validated against a robust reference. Currently, there is no gold standard. The aim of this study was to build evidence around a new manual video annotation tool as a reference standard for assessing respiratory rate in children under 5 with cough or difficulty breathing.
Malaria Consortium performed an inter-rater study to evaluate agreement between reviewers assessing the respiratory rate of children, using a manual video annotation tool. Video data had been collected in two previous studies conducted by Malaria Consortium. The video annotation tool allowed reviewers to manually indicate certain breaths, uncertain breaths and possible distortions (non-breath movements or other interruptions to normal breathing, such as crying). The tool had functionalities to optimize the annotation quality, like slowing down playback speed, zooming, adjusting brightness, etc. A panel of 10 reviewers were recruited, all medical staff with experience in pediatrics and manually counting respiratory rates. 50 videos were selected for assessment and included children under five with cough and ensuring equal representation of three clinically relevant age groups. Additionally, qualitative data from focus group discussion with panel members was evaluated using thematic analysis to assess usability and acceptability of the video annotation tool of the new tool as a reference standard.
Assessment of manual video annotation as a reference standard for determining respiratory rate in children the objective of this project is to build the evidence base around the accuracy of manual video annotation, as a reference standard for counting the respiratory rate in children under five. This will improve the accuracy of validating and comparing new-to-the-market automated respiratory rate counting devices, which are important and much needed tools to improve diagnosis of childhood pneumonia.
Overall, agreement on continuous respiratory rate between five reviewers was good, with an average respiratory rate of 9.19 beats per minute between the rate counted by the highest and lowest reviewer. There was substantial agreement on the breathing status between the reviewers, corresponding to all five reviewers agreeing on 70% of the videos. Agreement measures differed somewhat between age groups and were generally lower in younger age groups. The findings indicate that especially distortions observed in the videos, for example, distortions caused by restless and crying children, impact the inter-rater reliability of the tool. This was confirmed by qualitative assessments of reviewers' experiences. It was also noted that visual reference standards have inherent limitations due to human subjectivity, as humans do not always mark breaths in the same way, particularly when there is distortion. However, annotators perceived the respiratory rate to be more accurate with the annotation software than using manual counting, as it allowed them to distinguish between normal breaths, uncertain breaths, distortion and movement. They also mentioned that tool functionalities, like slowing down, changing color and adjusting brightness helped them in distinguishing between these different kinds of breaths and movements.
Malaria Consortium concludes that video annotation can be considered a reliable reference standard, if the marking of breaths in the distorted period can be improved. One advantage of video annotation is that it allows assessors to consider new elements when counting respiratory rate. The functionalities of the instrument supported the assessors in highlighting these elements, although it can be time consuming. Malaria Consortium indicates that acceptable limits of inter-rater agreement as standard for respiration rate counting are currently under review by the global community. As next steps, the Consortium concludes that consensus on such acceptable levels of accuracy and reliability, as well as on exactly which elements to include in the calculation of respiratory rate when using video annotation, needs to be reached before final conclusions can be drawn about whether the video annotation methodology fully meets the requirements as a robust reference standard.
When 2019 - 2020
Status Concluded
Where
Ethiopia
Topics
Tags
[1] Presentation of the results at the ASHTM 68th Annual Meeting – November 2019 [2] Peer reviewed publication on the study – Charlotte Ward et al., Determining the agreement between an automated respiratory rate counter and a reference standard for detecting symptoms of pneumonia in children: Protocol for a cross-sectional study in Ethiopia, JMIR Research Protocols 2020 | Vol 9, No 4 (2020): April [3] “What is a breath?” Blog by Charlotte Ward
Relevant links