Ways to improve evidence synthesis for systematic reviews - discussion thread

[Midnattsol]

Split from the Garner thread

Perhaps this says a lot about the subject of evidence synthesis - I'm not sure they ever really talk about robust methodology just how to tabulate things in other papers and add up the numbers. They don't even seem to check that the stuff they are adding up has the same intervention and hence can be combined.

That would be my experience of how these things are taught at medical faculties (from someone who has had methodology courses with MD students).

 

[Adrian]

That would be my experience of how these things are taught at medical faculties (from someone who has had methodology courses with MD students).

I have thought it would be interesting to take a more formal approach to describing methodology (i.e. a formal specification of the methodology) and then having model checkers running on this to validate certain properties are achieved or identify potential flaws. However, formal specifications are hard so no one would go that route. However, I wonder if an LLM could be used to analyze methodologies and find potential flaws etc - probably need a agentic multi-stage system that would go through a reasoning chain about potential issues etc.

 

[Utsikt]

I have thought it would be interesting to take a more formal approach to describing methodology (i.e. a formal specification of the methodology) and then having model checkers running on this to validate certain properties are achieved or identify potential flaws. However, formal specifications are hard so no one would go that route. However, I wonder if an LLM could be used to analyze methodologies and find potential flaws etc - probably need a agentic multi-stage system that would go through a reasoning chain about potential issues etc.

What would a «formal specification of methodology» look like?

 

[Adrian]

What would a «formal specification of methodology» look like?

Not sure. But in a paper they often describe the process that they go through in running a trial - things like selection, treatment, assessment etc. But in english and in ways that can be ambigious. So I imagine a more formal description codifying this back to a set of privatives and logic statements that can be reasoned over (does a property hold, or how does a property hold, compare to a risk model of known issues etc). To do such a thing would be a research task (its done in other areas). Ideally the description would also allow for the generation and testing of the stats code for processing results. (another issue I think with some trials is the lack of a code testing methodology - I've never seen anyone talking about this).


It may be that the various steps are really independent and there is no need to do complex analysis (for example, does the way subjects are selected have an interaction with treatment choices and hence outcomes).


Looking at thinks like GET trials they all have slightly different versions of GET which makes it really hard to compare. So having a way at least to capture the process they do and critical aspects feels important for any meta analysis.