Article Text

Download PDFPDF

Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria
  1. Bo Remenyi1,2,
  2. Jonathan Carapetis3,
  3. John W Stirling4,
  4. Beatrice Ferreira5,
  5. Krishnan Kumar6,
  6. John Lawrenson7,8,
  7. Eloi Marijon9,
  8. Mariana Mirabel10,
  9. A O Mocumbi11,
  10. Cleonice Mota12,
  11. John Paar13,
  12. Anita Saxena14,
  13. Janet Scheel15,
  14. Satu Viali16,
  15. I B Vijayalakshmi17,
  16. Gavin R Wheaton18,
  17. Liesl Zuhlke19,
  18. Karishma Sidhu2,
  19. Eliazar Dimalapang2,
  20. Thomas L Gentles20,
  21. Nigel J Wilson2,21
  1. 1Menzies School of Health Research, Casuarina, Northern Territory, Australia
  2. 2Green Lane Cardiovascular Services, Auckland City Hospital, Auckland, New Zealand
  3. 3Telethon Kids Institute, University of Western Australia, Subiaco, Western Australia, Australia
  4. 4Paediatric and Congenital Cardiac Services, Starship Children’s Hospital, Auckland, New Zealand
  5. 5Maputo HeartInstitute, Maputo, Mozambique
  6. 6Amrita Institute of Medical Sciences and Research Centre, Kochi, India
  7. 7Paediatrics and Child Health, Stellenbosch University, Cape Town, South Africa
  8. 8Department of Paediatrics and Child Health, Cape Town, South Africa
  9. 9Hop Europeen Georges Pompidou, Paris, France
  10. 10INSERM U970, Paris Cardiovascular Research Center PARCC, Paris, France
  11. 11Inst Coracao, New York City, New York, USA
  12. 12Federal University of Minas Gerais, Belo Horizonte, Brazil
  13. 13Cardiology, Project Health for León, Raleigh, North Carolina, USA
  14. 14All India Institute of Medical Sciences, New Delhi, India
  15. 15Pediatric Cardiology, Children’s National Health System, Washington, District of Columbia, USA
  16. 16Cardiology, Samoa National Hospital, Apia, Samoa
  17. 17Pediatric Cardiology, Sri Jayadeva Institute of Cardiovascular Sciences and Research, Bangalore, Karnataka, India
  18. 18Cardiology, Women’s and Children’s Hospital, Adelaide, South Australia, Australia
  19. 19Groote Schuur Hospital and University of Cape Town, Cape Town, South Africa
  20. 20Paediatric and Congenital Cardiology, Starship Children’s Hospital, Auckland, New Zealand
  21. 21University of Auckland, Auckland, New Zealand
  1. Correspondence to Dr Bo Remenyi, Menzies School of Health Research, Casuarina, NT 0810, Australia; Bo.Remenyi{at}


Objective Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria.

Methods A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen’s and Fleiss’ free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria.

Results Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ.

Conclusions The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.

  • mitral regurgitation
  • aortic valve disease
  • paediatric echocardiography
  • rheumatic fever
View Full Text

Statistics from


  • Contributors JC, NJW, TLG. KS and BR made substantial contributions to the conception and design of the work. BR, JC, JWS, BF, KK, JL, EM, MM, AOM, CM, JP, AS, JS, SV, IBV, GRW, LZ, KS, TLG and NJW made substantial contributions to the acquisition, analysis or interpretation of data for the work. BR prepared draft of manuscript. All authors made substantial contribution to the work or revising it critically for important intellectual content and final approval of the version to be published.

  • Funding Funding was received from the Green Lane Research and Education Fund, Auckland, New Zealand for the development of the study website.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval Ethics approvals were obtained for the study from the Northern X Regional Ethics Committee of the Ministry of Health of New Zealand and from the Human Research Ethics Committee of the Northern Territory Department of Health and Community Services of Australia. Both Ethics Committees waived individual patient consent.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement All data relevant to the study are included in the article or uploaded as online supplementary information.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.