Evaluation of research applications for funding

Moderator note: This post has been copied, and posts discussing it moved from this thread:
Patient Representative Reports from Dr Karl Morten's collaborative group, Oxford, UK

The MRC do not work like the American NIH. The NIH earmark a certain amount of money for each category of "illness", and that is the amount on offer that year. (Although, in reality, chunks of the money set aside for ME/CFS have been "lost", or spent on non-ME stuff).

The MRC does not have an overall quota (or in my terms, it has no overall strategy). It simply allocates funds to any study that is "good enough". In theory that could mean that in 2020 all of its budget could go to a brilliant collection of studies on ingrowing toenails.

Back in 2011, 123 MPs signed an Early Day Motion complaining about the MRC's lack of funding for biomedical research into ME, and as a result of that, the MRC finally earmarked £1.6 million for that. It wasn't much, but, we thought, it was at least a start. But it wasn't a "start" because, apart from funding a small follow-up to one of the studies, no more biomedical research studies were funded until very recently.

The argument about the lack of quality of applications into research into ME, like all excuses, will have some truth behind it. If you were a brilliant potential researcher at university and were thinking of researching ME, I'm sure your tutors would point out the fact that no money is put into that research, and would direct you to another area, or advise you to take a psychological approach. But the Gibson report named three experienced biomedical researchers who had had applications turned down, including one who was a member of the MRC.

The argument about the "poor" quality of applications would be more convincing if the quality of the majority of accepted applications was high, but that doesn't appear to be the case. I'm not arguing that poor quality ME studies should be funded. I am arguing that, given the varied quality of accepted applications (think the PACE trial), the judgement process is flawed and allows prejudice against biomedical research into ME to have an effect.

At school, before being allowed to grade coursework or mark papers, teachers have to undergo specific training so that they are clear about what is wanted, and so that they will mark work consistently, with grades matching those of experienced examiners. If my department had marked coursework 9, 8, and 3, I would have been very worried. At A-level statistics, if the two of us (independently) marking their coursework did not agree completely with the grade (which seldom happened) the reason for the disagreement had to be pinned down, or an external assessor would have been needed. The process in the medical world is unprofessional and sloppy, both in assessing the worth of applications for grants and in the peer review process deciding upon whether a study should be published.

I think it reflects the arrogance of the medical system that their members are highly educated so have no need of such training. We are stuck in the days of the Victorian gentleman scientist.

It is only very recently that Wellcome have funded any research into ME, and hasn't their more recent foray into the ME world included funding Esther Crawley's work?

In the reply to my complaint of a few years ago, the MRC said that the overall success rate was a quarter: from what I could make out from the information I had, the success rate for applications to study ME was around half of that (that's from memory – if you want to quote it, let me know and I'll see if I can sort it out again).

 

A fair bit of money in the NIH is earmarked for specific illnesses, but the vast majority isn't earmarked. I believe grants are just given according to the quality of the grant, and perhaps other factors (basically like the MRC). The full process is a bit complicated, and there's more relevant info on these pages: https://grants.nih.gov/grants/peer-review.htm and https://nexus.od.nih.gov/all/2011/02/15/paylines-percentiles-success-rates/

 

True, but for many years they earmarked $6 million per year for ME/CFS, even though it didn't get spent on the sort of stuff we would like. I very much doubt whether the total state funded research into biomedical research on ME/CFS over the last 40 years much exceeds $6 million.

Like Andy, I doubt whether Carol could reverse the decision, and, to be honest, we don't know whether the application was too ambitious. But to my mind, if she is to push for a bigger emphasis on biomedical research, a summary of the history of such spending would help her.

 

This is true but I don't think it has anything to do with the variations of 9,8,3,2. What those numbers mean is either that 9 and 8 understand the idea and 2 and 3 do not, or vice versa. It is purely a matter of the limits of intellect. My most important theoretical paper was turned down by the Lancet in the grounds that 'we already have an adequate theory'. It was published elsewhere and is now one of my most cited papers. I know the referee at the Lancet didn't understand because he had been my boss.

The problem is that being professional and unsloppy does not affect limits of intellect. Some people will never understand why one theory is better than another. And these people often do well climbing up the science ladder because they stick to the fashionable theories and get grants.

My reference to Wellcome was in the context of the progress of a project from preliminary data to programme grant. Wellcome have always been keen to 'pump prime' by funding early projects in a way MRC have always said they did not. Committee complements turn over usually on a 3-5 year term so history is never a reliable guide to what a charity will fund.

 

I understand your comment, but I hope you will understand that, as teachers, we often ended up marking work from students with particular knowledge or skills way beyond ours. Probably the most challenging for me was a piece of coursework which involved programming in a language unknown to me, but the one by a mature student on biopsies was pretty challenging. In such cases you need to have very clear criteria (somehow I doubt that the MRC has such criteria), based on the experience of many skilled assessors, together with the honesty to ask around for help in understanding such things, or to pass the work over to someone who does. Always, a number of our assessments would be reviewed by other experienced assessors.

It takes a considerable amount of time to assess such work: going from many of the comments that I have seen from the peer review process, I am not convinced that all of them do spend that amount of time. As teachers we insisted on seeing all the data and the analysis, and would expect to check through them. I doubt very much whether many (any?) reviewers ask for the data/spreadsheets, and suspect that very few of them have gone through any sort of rigorous training.

I'm not knocking all reviewers, researchers and specialists, of course. There are brilliant ones. I'm criticizing the system that allows sloppiness to flourish.

I have come across a number of medical specialists in my time, and only five of them, in my opinion, were intelligent enough to have the humility to understand and accept that they didn't have all the answers. I suspect that researchers are similar: the whole structure of medical learning seems to be to revere the words of the specialist, whereas in science progress is expected to emerge from the younger members challenging established views.

Reviewing and assessing need to be seen as a high-level skills, ones that need time and attention, and need to be paid positions. The pay need not be amazing, because the roles should carry respect and status.

 

Yes, I think we have similar thoughts about the basic problem. What I think may be different specifically in the area of studying disease mechanisms is that there is no established arbiter of who to pass things to who understands.

That might just be 'sloppiness' and there is certainly a lot of that, but I guess that whereas there is a definitive answer to questions about computing languages that somebody should know, for pathophysiology it is all much less certain.

I often quote as an example the reason why people with heart failure have swollen feet. All textbooks say this is due to an imbalance in Starling forces. But in reality it cannot be. It has to be due to a problem that Starling recognised but not an imbalance of his forces.

And that is an easy one. Actually proving any mechanism in pathophysiology is extraordinarily difficult. Proving mechanisms in normal physiology is not too bad but disease poses quite different constraints. I published a paper predicting that rituximab would be effective in RA, and went on to show it was but 95% of my colleagues still assume that the reason it works is a different theory, the flaws in which led me to make the prediction. So I retired feeling pleased that I had developed a highly effective treatment but sad that none of my colleagues understood the science.

SO yes, there is sloppiness in medicine and failure to find the person who knows how to interpret the problem, but there is the additional problem that the great majority of people in the field do not know how to recognise who that is. Maybe medical science has always been like that but having had the opportunity to talk at length to Andrew Huxley - who epitomised the zenith of UK biomedical science in the early 50s with Hodgkin and Crick and Watson - it seems to me that in the sixties and seventies the arbiters in biomedical science did know what they were talking about. Maybe in further generations those people went to work for Goldman Sachs.

And this is not a UK problem. In the 1970s US medical textbooks were faultlessly evidence based. Nowadays I don't even know if there are such textbooks. Students are tube fed a torrent of electronic drivel as far as I can see.

Maybe I am getting old, but maybe it isn't just that.

 

Yes, @Jonathan Edwards , I am sure that we agree on most of this. The one thing that I am uncomfortable about though is this idea that you need to find someone who is able to understand the whole picture. Maths coursework, like the statistics coursework, was much more open-ended in the early days that people imagine. But work on, say, quantum theory back in its early days certainly was a big unknown, with no real experts. Good criteria should encompass that. After all, research is about testing hypotheses, drugs, other treatments, etc.: the results of the study determine whether it is worthwhile. Criteria should be focused on the quality of the actual methodology etc. (which it clearly wasn't with PACE). It is true that you don't want hundreds of good quality trials on stupid hypotheses, but equally, good scientists cannot turn their backs on new and controversial ideas.

You example about swollen feet is a good one. Those who really are on the ball, should recognize this as a speculative explanation: one that satisfies a number of people, but which doesn't actually fit the facts. It's rather like the dual wave/particle theory that answers the question, but which we know is only a fudge.

However I must take you to task on one of your speculations: "maybe" you are getting old. Maybe? Really? When you reach our age, is there really that much doubt?

 

Ah, but here I have to refer you to the Radical Pragmatic theory of Charles Travis, linguistic philosopher. Words such as maybe mean different things in every different sentence. 'Maybe I am getting old but...' is normally interpreted as 'Yes I know I am getting old, but ....

Moreover, I am not sure that we should take this as fact. I have for many years now maintained that after a certain age, maybe about 57 and a half, one actually starts to get younger again. My mother, who is 97 has certainly corroborated this hypothesis. She quotes Pooh Bear endlessly and sings hymns selected for girls at boarding school. This also fits with the fact that we are getting shorter. I have also gone back to eating marmalade sandwiches at tea time.

Even moreover, as a follower of Leibniz I believe that all souls are immortal and nothing ever grows old. Quantum mechanics says the same thing. And Bach is playing on the radio and he never grows old.

So I rest my case, having argued it out;
I think there's plenty of doubt about.

 

But it’s soooo tempting

 

Biscuit, anyone?

 

Ok they you go then...but only because you made me

 

Not even for a ... ?