Biomedical articles on MEpedia - purposes and pitfalls

[Jonathan Edwards]

Moderator note:
This thread has been split off from the introductory MEpedia thread here.

My hope is that discussions in this forum can be less about debating the content matter or the question of whether X is "true" (where so many conversations on forums end up leading) and focus more on how to improve the page in question by making it more complete, accurate, readable, etc.

For example, one could debate whether non-cytolytic enteroviral infections or a/the cause of ME, but that's a different question than how to improve this page: https://www.me-pedia.org/wiki/Non-cytolytic_enterovirus or what on this page is or is not accurate. Conversations about the former should probably be moved to a different sub-forum.

Put another way, whether something is accurate (i.e., it accurately reflects the source material and its quality) is a different question from whether it is true (we may not have the evidence to know).

@JenB

My reading of @JaimeS's 'objective truth' is pretty much the same as your 'accuracy'. The point seems to be that people should aim not to give a biased (subjective) view. So for the page on non-cytolytic enterovirus my thought would be that right at the start it should be mentioned that the biomedical research community as a whole do not think that the concept of non-cytolytic enterovirus infection is of any clinical significance to anything much.

We have a big problem that has developed in the last twenty years which is that the great majority of review articles in biomedical science are essentially marketing tools for grant sourcing. It is only too easy to find statements that are strictly speaking true or accurate in some sense but which are wildly misleading. The crux of the matter is both the 'quality' of the sources as you put it and how justified extrapolation is.

If MEpedia pages are left to enthusiastic individuals to create I think you are going to end up with something very like the review journals in science now - wall to wall advertising of speculative ideas. I think it is worth thinking hard about whether that is really a useful project.

 

[Andy]

@JenB

My reading of @JaimeS's 'objective truth' is pretty much the same as your 'accuracy'. The point seems to be that people should aim not to give a biased (subjective) view. So for the page on non-cytolytic enterovirus my thought would be that right at the start it should be mentioned that the biomedical research community as a whole do not think that the concept of non-cytolytic enterovirus infection is of any clinical significance to anything much.

We have a big problem that has developed in the last twenty years which is that the great majority of review articles in biomedical science are essentially marketing tools for grant sourcing. It is only too easy to find statements that are strictly speaking true or accurate in some sense but which are wildly misleading. The crux of the matter is both the 'quality' of the sources as you put it and how justified extrapolation is.

If MEpedia pages are left to enthusiastic individuals to create I think you are going to end up with something very like the review journals in science now - wall to wall advertising of speculative ideas. I think it is worth thinking hard about whether that is really a useful project.

So the argument here seems to be that, because there is a bias problem elsewhere, MEpedia shouldn't exist because it might become biased itself? And I'm also puzzled why you think unnamed 'enthusiastic individuals' will cause a problem with this when you have been kind enough to praise other enthusiastic patients in their ability to comprehend and dissect the reams of bad science that, in particular, psych researchers, who at times are deemed experts, have dumped upon us. I don't see why we can't hope that these same worthwhile patients, many, but not exclusively, who have contributed to these forums, won't contribute to MEpedia in an even, considered and accurate way.

 

[Trish]

I think there is a big difference between reading a BPS paper about a trial of GET and CBT using the Chalder Fatigue scale as an outcome measure and reading a biomedical paper about viruses or metabolomics or brain scans and summarising it for an MEpedia article.

Most of us can see that such a BPS paper is junk and make a good stab at explaining why it is junk - we don't need a medical degree to see the obvious. And we have a good range of people in the ME community with sufficient statistical skills and understanding of research to debunk that junk. Even then, it was important that people with expertise like Keith Geraghty, Carolyn Wilshire, Jonathan Edwards, Mark Vink, not all of whom have ME, and others have published peer reviewed papers to refute the junk. It would not be sufficient for us to simply allow anyone who volunteers to produce that analysis from scratch. We needed these experts. It is then possible and useful for MEPedia to collate a list of these BPS papers and the critiques.
And to have articles about the history, key characters in the ME story etc. All good stuff.

But most of us haven't a clue how to assess the quality or validity of biomedical research. It takes many years of very specialized learning and experience to be able to tell whether a biomedical research paper is really telling us something important or has fundamental flaws.

I have no idea whether the individuals in the ME community who are helpfully volunteering to write articles for MEpedia on biomedical science actually know what they are talking about. That is the nature of such a volunteer-run open editorial project. That is why I think there are dangers in simply inviting anyone to try to 'deep dive' into topics in which they are not expert and produce summaries. The result may be a lot of hard work for the individual, but a lot of misinformation and confusion too. This could be counterproductive and end up giving MEpedia a bad name.

I think if MEpedia is going to venture into writing biomedical articles, it needs to restrict them to people with sufficient expertise, or to linking to useful and reputable summaries and explanations provided by others elsewhere.

 

[Jonathan Edwards]

So the argument here seems to be that, because there is a bias problem elsewhere, MEpedia shouldn't exist because it might become biased itself? And I'm also puzzled why you think unnamed 'enthusiastic individuals' will cause a problem with this when you have been kind enough to praise other enthusiastic patients in their ability to comprehend and dissect the reams of bad science that, in particular, psych researchers, who at times are deemed experts, have dumped upon us. I don't see why we can't hope that these same worthwhile patients, many, but not exclusively, who have contributed to these forums, won't contribute to MEpedia in an even, considered and accurate way.

I think it is more subtle than that, @Andy. I may not have expressed myself well but essentially I am agreeing with what both JaimeS and JenB have said about quality control but tried to emphasise just how difficult that may be to achieve without formal editing. I think an MEpedia is a very good idea. There were discussions about it on another place about five years ago and I was all for it.

I have indeed praised patients' abilities to analyse the science. However, picking holes in poor science is a rather different thing in human nature terms than enthusing about a pet theory. When it comes to pet theories we are all likely to get a bit carried away. And it does not help that the review literature consists nowadays of people who have got carried away giving biased accounts of ideas. And glancing through the MEpedia pages there are already lots of examples of over interpretation of the scientific literature.

 

[Andy]

But then who is to say that people who are selected to be the editors won't introduce bias themselves? And I am not convinced that suitably qualified/skilled/knowledgeable editors are going to be available and interested in sufficient numbers to make MEpedia workable. The alternative to the current situation is something more in line to a journal, where only once a submission has been deemed suitable is it then published, which is obviously a different idea to the concept of crowd sourcing knowledge that MEpedia currently employs.

So I'm not trying to deny that content is added to MEpedia by those who have an interest in it, and that, because of their interest they are likely to, despite the guidelines, introduce their own bias to the content. But the idea behind the project is that if someone else notices the bias in the article then they can either edit it themselves, or start a discussion on whether it should be changed or not. So the concept is that it should be an iterative process, in a similar way to how science itself is meant to progress.

I myself add content to MEpedia, when I have both the energy and the interest. If it became something more involved where I had to submit my changes to be approved or not then I'm likely to not bother at all, and I fear that this would be something that would happen with other contributors.

 

[Jonathan Edwards]

But then who is to say that people who are selected to be the editors won't introduce bias themselves? And I am not convinced that suitably qualified/skilled/ knowledgeable editors are going to be available and interested in sufficient numbers to make MEpedia workable. The alternative to the current situation is something more in line to a journal, where only once a submission has been deemed suitable is it then published, which is obviously a different idea to the concept of crowd sourcing knowledge that MEpedia currently employs.

Indeed. It is not easy. But the traditional model is that a group of editors bring together material on a wide range of topics and that works because the editors are unlikely to have bees in their bonnet about all the different topics. That system worked until about 1995 and would still work without the commercial pressures of modern journals.

You only need 'sufficient numbers' if the idea is to 'cover everything that moves'. Organisations that put out information for patients normally use amore cautious approach. Rather than trying to cover everything they cover a few things they think they are reasonably confident they can get right. That may well be bit more like an old style journal or textbook. I think it might be a better idea. I constantly have in the back of my mind the terrified parent with a child with ME who goes to the net to find out information and who ends up with the misleading idea that ME is 'a multi-system disease with immune, metabolic and so on abnormalities' and gets conned by every quack in the system. ME is a serious health problem that needs to be handled very carefully.

But the idea behind the project is that if someone else notices the bias in the article then they can either edit it themselves, or start a discussion on whether it should be changed or not. So the concept is that it should be an iterative process, in a similar way to how science itself is meant to progress.

I agree that one would hope that would happen. But that other forum crashed because certain tensions proved unresolvable. And the roles people took were not easy to predict. There's nowt as queer as folk as they say. And science might have progressed as it was meant a while back but I have little faith it does now. It is driven by fads and politics.

I don't want to stop people trying to get things right but I think it is worth flagging up caution based on forty years experience of trying to get medical information right. I keep coming back to the idea that the entire content of an MEpedia should probably be that ME is a bloody awful illness that makes people severely disabled for years and we cannot really find anything wrong and have no idea what is going on but there must be something badly wrong so we need better research to begin to understand. Plus a few actual facts like symptoms and demographics.

 

[JenB]

We want to make the literature accessible to students, scientists, and the public by summarizing, collating, and interlinking it. Whether or not you think the literature has ever found anything useful, there are 6000+ studies, not to mention the literatures of adjacent fields. Most of the people working within the field let alone outside of it have barely scratched the surface of what is there. If I were entering this field, I'd love to know where it's already been before deciding where I want to go. I hope, in many years' time, MEpedia will make this easier.

But we definitely want to avoid arguing (in the undergraduate term paper sense) the literature. Any page that does this you can mark for "Cleanup" using the cleanup template and specifying the reason. This both marks the page to other contributors/editors as "as needs improvement" (https://www.me-pedia.org/wiki/Category:All_articles_needing_cleanup) and helps act as a caution to anyone reading it by framing its potential shortcomings.

However, there are also pages that are intentionally meant to be more speculative. These are the "medical hypotheses" pages (https://www.me-pedia.org/wiki/Category:Medical_hypotheses). We've made them a different color as well. Contrast the Vagus nerve infection hypothesis page (https://www.me-pedia.org/wiki/Vagus_nerve_infection_hypothesis – the current template design is pretty ugly as is, we're working to improve) and the actual vagus nerve page (https://www.me-pedia.org/wiki/Vagus_nerve). It may be the solution is that some pages need to be split because they are struggling to straddle what is known and unknown. This can be discussed on the discussion page of that page or in conjunction with the Cleanup template.

My MUCH bigger concern than any of this are the pages that give direct treatment advice including information on dosing, etc. of supplements and medications. This should be deleted wherever it's seen or again, marked for Cleanup. I think it's a small number of contributors doing this, so another option is to look at the revision history, go to that person's profile, and leave a message pointing to our editorial guidelines: https://www.me-pedia.org/wiki/Editorial_Guidelines Actually, I don't see something specifically about this, so this is something that is probably worth adding @JaimeS

As for sticking to facts and fact sheets that charities traditionally put out, we have those too, in PDF format. The purpose and possibilities of a wiki are very different.

 

[JenB]

I proposed a wiki six years ago (I'm sure I wasn't the first) and the heated discussion about whether it should exist was such that I abandoned the effort entirely for many years. MEpedia does exist and this space is really about contributing to it and making it better.

So I appreciate your comment @Jonathan Edwards re: not trying discourage people trying to get it right. And yes, of course there will be challenges as it grows beyond the three or four people who have written 90% of it over the last two years, but those are the kinds of problems I hope for and welcome

 

[Adrian]

So the argument here seems to be that, because there is a bias problem elsewhere, MEpedia shouldn't exist because it might become biased itself? And I'm also puzzled why you think unnamed 'enthusiastic individuals' will cause a problem with this when you have been kind enough to praise other enthusiastic patients in their ability to comprehend and dissect the reams of bad science that, in particular, psych researchers, who at times are deemed experts, have dumped upon us. I don't see why we can't hope that these same worthwhile patients, many, but not exclusively, who have contributed to these forums, won't contribute to MEpedia in an even, considered and accurate way.

I thought the point was to try to not make articles opinion articles or theories but to reflect the range of different views based on the primary evidence. As you get one person writing an article in a difficult area it is very hard for it not to become an article pushing particular opinions. So I think care really needs to be taken to ensure accuracy and a fit to the evidence.

 

[Ravn]

@JenB @Jonathan Edwards

Would the following idea help overcome some of the potential bias issues you're discussing? And if so, would it be technically feasible, some sort of pop up window when hovering maybe?

For each study cited there should also be a brief list of some key points that would allow readers (and potential editors looking for weak points to correct) to quickly scan for likely quality and relevance.

A few ideas for what the list could include, just off the top of my head:

• Size of study.
• Human, animal or cell(type).
• Type of study (pilot, clinical, randomised, double-blind, open label, etc).
• Outcome measures (self-reported, objective).
• Finance source if it constitutes a possible conflict of interest. Other conflicts of interest.
• Highlight major controversies and other studies with conflicting results (where that hasn't been done in the text itself already).
• Highlight if, for small or poor quality studies, they are the only ones available to date (again, if that hasn't been done in the text itself already).

 

[Barry]

But most of us haven't a clue how to assess the quality or validity of biomedical research. It takes many years of very specialized learning and experience to be able to tell whether a biomedical research paper is really telling us something important or has fundamental flaws.

I have no idea whether the individuals in the ME community who are helpfully volunteering to write articles for MEpedia on biomedical science actually know what they are talking about. That is the nature of such a volunteer-run open editorial project. That is why I think there are dangers in simply inviting anyone to try to 'deep dive' into topics in which they are not expert and produce summaries. The result may be a lot of hard work for the individual, but a lot of misinformation and confusion too. This could be counterproductive and end up giving MEpedia a bad name.

I think if MEpedia is going to venture into writing biomedical articles, it needs to restrict them to people with sufficient expertise, or to linking to useful and reputable summaries and explanations provided by others elsewhere.

This is my very real worry, and why I queried in the other thread whether there is any process to avoid chaos. MEpedia inevitably has a lot of credibility in many people's eyes, and its content just has to be worthy of the trust people place in it. How do you go about ensuring that?

 

[Barry]

I myself add content to MEpedia, when I have both the energy and the interest. If it became something more involved where I had to submit my changes to be approved or not then I'm likely to not bother at all, and I fear that this would be something that would happen with other contributors.

I agree very much with the overall aims here. But what if someone similar to you in many ways, with bags of enthusiasm and dedication, also provides material - except they have one big difference from you ... they don't have the diligence for accuracy and objectivity that you have, but instead are more focused on their own interpretations. Surely you would find it very difficult to see such content being published. I'm not saying I know the answer, because I don't.

 

[JenB]

I thought the point was to try to not make articles opinion articles or theories but to reflect the range of different views based on the primary evidence. As you get one person writing an article in a difficult area it is very hard for it not to become an article pushing particular opinions. So I think care really needs to be taken to ensure accuracy and a fit to the evidence.

Yes, but for that we need contributors.

@JenB @Jonathan Edwards

Would the following idea help overcome some of the potential bias issues you're discussing? And if so, would it be technically feasible, some sort of pop up window when hovering maybe?

For each study cited there should also be a brief list of some key points that would allow readers (and potential editors looking for weak points to correct) to quickly scan for likely quality and relevance.

A few ideas for what the list could include, just off the top of my head:

• Size of study.
• Human, animal or cell(type).
• Type of study (pilot, clinical, randomised, double-blind, open label, etc).
• Outcome measures (self-reported, objective).
• Finance source if it constitutes a possible conflict of interest. Other conflicts of interest.
• Highlight major controversies and other studies with conflicting results (where that hasn't been done in the text itself already).
• Highlight if, for small or poor quality studies, they are the only ones available to date (again, if that hasn't been done in the text itself already).

This is a really nice idea but we don't really have the capacity to implement this for every study we cite. I think where it's relevant, at this stage this type of careful qualification just needs to be incorporated directly into the prose of the page. Many pages already do this, especially for findings that haven't been replicated.

Much of the time, it's going to be case-by-case/contextual. Wrestling with these questions in the discussion pages attached to each article is going to be important.

Global issues are best addressed by suggesting changes to the science guidelines: https://www.me-pedia.org/wiki/Science_Guidelines. @JaimeS the Editorial Guidelines talk about different types of source quality https://www.me-pedia.org/wiki/Editorial_Guidelines but the Science Guidelines only talk about human v. animal studies. Perhaps the "Types of Studies" section of the Science Guidelines could be expanded?

 

[JenB]

I agree very much with the overall aims here. But what if someone similar to you in many ways, with bags of enthusiasm and dedication, also provides material - except they have one big difference from you ... they don't have the diligence for accuracy and objectivity that you have, but instead are more focused on their own interpretations. Surely you would find it very difficult to see such content being published. I'm not saying I know the answer, because I don't.

You hit "edit."

 

[JenB]

Or you mark the page as "Needs Cleanup."

We have copied many of these templates from Wikipedia (which deals with all these same issues and questions) but can always create custom templates/disclaimers for our use. I think a page that is problematic / incomplete with the right disclaimer is better than it not existing at all, because then there's something there to contend and discuss and work with. Wikis are really about collaboration and incremental improvement, but it means a lot of pages will be rough drafts / works in progress for a long time.

You could also try to recruit someone who might have the interest/expertise to contribute a different perspective to the page.

 

[Barry]

You hit "edit."

Who does? What if that person simply has different but just as misguided opinions of what should be written? Could you not just end up with a succession of edited content, none of which is anything other than biased opinion? Some might be good, but others not. I don't see the mechanism whereby content will tend towards good quality; feels like in some cases it simply get stuck in a very misleading and poor quality. And no, I'm very well aware that I am way short of the expertise needed to provide such content, so am not presuming to speak from that perspective.

 

[JenB]

Who does? What if that person simply has different but just as misguided opinions of what should be written? Could you not just end up with a succession of edited content, none of which is anything other than biased opinion? Some might be good, but others not. I don't see the mechanism whereby content will tend towards good quality; feels like in some cases it simply get stuck in a very misleading and poor quality. And no, I'm very well aware that I am way short of the expertise needed to provide such content, so am not presuming to speak from that perspective.

It's a risk, but those risks are the risks of wikis. And the internet. And of science and human society. I don't think we should put all of that on MEpedia. My aim is to work as hard as possible and bring in as many dedicated, interested and/or expert people as possible to learn and have fun together, while producing a concrete output that can be useful to our community and the world at large. No, there are no guarantees but there never are.

We aren't Wikipedia, but here is Jimmy Wales talking about very similar issues:



And no, we don't have this level of participation yet, but neither did Wikipedia when it started. But I think over time we can get there, scaled to the size and nature of this project.

 

[JES]

Who does? What if that person simply has different but just as misguided opinions of what should be written? Could you not just end up with a succession of edited content, none of which is anything other than biased opinion? Some might be good, but others not. I don't see the mechanism whereby content will tend towards good quality; feels like in some cases it simply get stuck in a very misleading and poor quality. And no, I'm very well aware that I am way short of the expertise needed to provide such content, so am not presuming to speak from that perspective.

Isn't that a general issue that applies to regular Wiki pages as well and not specifically to MEpedia? Regular Wiki pages are up to all sorts of discussion, edits, vandalism etc., but usually they end up rather fine.

 

[JenB]

This is also interesting: https://en.wikipedia.org/wiki/Wikipedia:Wikipedians#Number_of_editors

Wikpedia has 5,694,003 pages and 120,415 users who have made an edit in the last 30 days. (Or 47 pages for every active user.)
Roughly 3395 have contributed more than 100 edits in the last month, or 2.8% of active users.

MEpedia has 2093 pages and 30 users who have made an edit in the last 30 days. (Or 70 pages for every active user.)
4 users have contributed more than 100 edits in the last month, or 13% of active users.

It would definitely do us well to increase the number of active users! But the number of pages we have is just always going to be much, much smaller. I think if we could grow to 100 active users, and maintain that same ratio of super users, we'd have a good chance of starting to replicate the dynamics that make Wikipedia work.

Age also matters. The older the project is, the more "mature" pages there will be, and the less maintenance and editing they will require.

 

[Alvin]

It's a risk, but those risks are the risks of wikis. And the internet. And of science and human society. I don't think we should put all of that on MEpedia. My aim is to work as hard as possible and bring in as many dedicated, interested and/or expert people as possible to learn and have fun together, while producing a concrete output that can be useful to our community and the world at large. No, there are no guarantees but there never are.

Basically this.
Someone once said half of what you learn in medical school will later be proven wrong (and i wish someone had told me which half).



@Jonathan Edwards If we apply your standard of only whats been conclusively proven or a disease mechanism (and how do we set this standard, a dozen replications) then human progress would be centuries behind where it is today. Who will pay for constant replications and what exact standard of proof is acceptable? At what point can we finally talk about results?

I would like to see only accurate data on the news, coming from politicians, Wikipedia and MEpedia, but if we go beyond the standards of the rest of medicine and society and only put down what is replicated many times over or stands up decades later then it will take millennia to get anywhere. Not only this but those who act in bad faith will not hold themselves to your standard, their goal is to spread their heroic lies, they will take advantage of us holding back legitimate research and fill the void with their garbage. It then becomes the only game in town...

A great example of real life failure; fats causing heart disease. We would love if this never happened, but it did and is still infecting medicine. But if someone sat on it it would not have become "common wisdom" which is good but then it could still be on the books barely known because there would have been no incentive to test and prove or ultimately disprove it leading to hopefully a better theory that is correct (btw my money is on Vitamin K2, worth googling and needs more research money). Toiling in obscurity has a certain romanticism to it but progress is built on previously disseminated discoveries, good and bad. I wish we had the ability to root out only the bad but we do not and throwing out the baby with the bath water does nobody any good. Our best bet is to try and screen as well as we can and acknowledge our limitations in inference of results.

What if Louis Pasteur sat on his germ theory because it was too different and controversial? Who would know to test it, replicate it, use it, build on it, learn from it? What if sanitation practices took decades longer to be invented and percolated because the germ theory was unknown hence could not be built upon preventing disease?