NICE's performance on the Long Covid guideline and the Chronic Pain guideline has been less than great - there were various combinations of a poor selection of reviewers; not listening to the valid concerns of some of the reviewers; not having a proper consultation process; and not listening the valid concerns of stakeholders.
I do hope someone will do it that (apply grade to the two components of the asthma trial separately, to the part of the trial with objective outcomes and to the part of the trial with subjective outcomes).
These, and the associated comments I didn't quote, are great observations. In the industry I used to work in, forestry, there were several competing quality standards for forest management. People see might know the Forest Stewardship Council certification, but there are others. The schemes involve a set of criteria, so everyone knows what is expected. These are developed by committees of the forest managers, people affected by land use, purchasers of forest products, among others. The criteria develop over time as more is known about what makes for sustainable forest management that allows a company to have a social licence to operate.
People in the companies and organisations are trained in evaluation. And there are external assessors. These schemes built on the enormous gains that the ISO certification delivered, but allowed for the particular needs of the sector.
They made an enormous difference to the forestry sector. Of course, there are forest managers who aren't certified, but reputable buyers of forest products have nothing to do with them. And of course there are criminal goings on with assessors being bought off, and documentation faked, but there are processes to pick that up over time, processes aiming to make everything very transparent. Assessment reports are published online for anyone to see.
Something like that should be done for medical research. Compared to the range of things that have to be considered in forestry, I think it would not be hard to come up with a certification scheme for good medical/health research. e.g.
- have stakeholders been consulted? (how are the potential users of the research, the patients and the clinicians involved with the research design? were the needs and rights of indigenous people, of women, of children and others thought about? have ethical standards for involvement of patients been followed?);
- does the methodology for processes fit with defined good practice - the experiment design, the statistical analysis?;
- was the trial design lodged in a public registry?;
- have aspects of privacy and data security been thought about?
- are the outcomes in line with standard outcomes developed in consultation with stakeholders?;
- have aspects of health and safety been thought about (e.g. is it really safe to do that gene-editing coronavirus research? has guidance about use of animals in research been considered? are staff and trial participants kept safe?);
- is there a process for continuous improvement - so what is learned by doing one study is incorporated into the organisation's procedures? what are the processes for selecting peer reviewers? what are the processes for responding to feedback on the research?
- is there a process for making key information public and reviewing the quality of research done internally e.g. making data available; publishing the names of peer reviewers; were all planned outcomes reported?; dissemination of results, including to the participants, to funders, to clinicians, to patient groups. What are the rules for communicating results to the media?
- was the research run in a cost-effective way? Were there processes to ensure that funding was handled appropriately? Were planned milestones met? If not, why not?
- and what is the process for external review all of these things?
Purchasers of research could specify that only researchers or organisations who are certified under the standard can participate in funding rounds.
It might be fun to come up with such a standard. Maybe there is one already, that brings together the bits like the guidance on the use of animals in research.
The stakes of medical research are unquestionably very high, and yet my impression is that the standard of the work done is more often than not pretty low. And the regulating processes to ensure it is done well seem inadequate. If the forest sector can do it, surely a sector full of doctors and universities can manage to?
