How to read a funnel plot

Started 13 April 2025 · 24 replies · 267 hearts · read 4.4k times

PatrickLimerick, IE · 13 April 2025

1.3 years ago

Something I have been sitting with: How to read a funnel plot Not looking for advice so much as other people's experience.

I have read the paper and I am not sure I have understood it.

I would like to understand the limits of this properly rather than the headline, because the headline is the part everybody already knows.

I would like to know what the best single piece of human evidence is, because I suspect the honest answer is thinner than the discussion suggests.

I will summarise whatever the thread concludes and add it to my notes, in case it is useful to somebody later.

TheaBergen, NO

What I have taken from this group is mostly humility about how much is genuinely settled.

10me too 4

SunilFacilitatorBirmingham, UK

I want to say what this circle owes anybody who decides against, says so, and gives their reasoning: respect, and no argument about it. That is the tone I want here and it is not negotiable.

NilsUppsala, SE

My reading has got slower and better. I get through a quarter of what I used to and I understand considerably more of it.

TiborBratislava, SK

What Nils described happened to me too, in a slightly different order.

Something worth knowing: the difference between a trial and an observational study changes what you can conclude, and I did not know that two years ago.

XanBerlin, DE

Nothing to add except that I am glad this thread exists.

StellanMalmö, SE

Xan — yes, and the bit that took me longest to accept was the next part.

One paper a fortnight, read properly, has taught me more than three years of reading summaries of papers.

5me too 8

NadjaBelgrade, RS

Building on what Stellan wrote rather than answering the original question.

The session where we read a paper about something none of us takes was the most useful, because nobody was defending a position.

GilTel Aviv, IL

Nothing here is medical advice, and a paper is not a recommendation for any individual person including the ones in it.

PrunellaBury St Edmunds, UK

Nadja — yes, and the bit that took me longest to accept was the next part.

Something practical: I write three sentences about each paper afterwards and I now have two years of notes I actually refer back to.

HattieExeter, UK

#10

My experience is that the group is much better at this than I am alone, because somebody always spots the thing I skimmed.

AmmiKarachi, PK

#11

The registry entry is worth checking against the published paper, and on one occasion the outcomes we were reading were not the ones originally listed.

BeaDublin, IE

#12

I found the discussion sections the least reliable part of most papers and the results tables the most.

GwendaEbbw Vale, UK

#13

Bea said something I want to sit with for a second.

What surprised me was how readable most of these are once you stop being intimidated by the first page.

PerrineStrasbourg, FR

#14

The paper we read last month had a population nothing like most of this circle, which does not make it useless and does limit what it tells us.

14me too 2

NkemPort Harcourt, NG

#15

Somebody here asked what would change our minds, at the start of a session, and it improved the discussion enormously.

EphraimBrooklyn, NY

#16

Agreed, and I have nothing to add to it.

PriyaReading, UK

#17

What I would flag is that a study in a different population may not transfer, and that is a limitation rather than a criticism.

8me too 6

SaoirseKilkenny, IE

#18

What Priya described happened to me too, in a slightly different order.

I would encourage anybody nervous about joining to come and say nothing for a few sessions. That is what I did for four months.

FabianBasel, CH

#19

Saoirse, thank you for that. It gives me somewhere to start from.

Coming back to say the fortnightly rhythm has held for two years and it is the thing on this site I would least like to lose.

JunoBrighton, UK

#20

The thing I look at first now is who was actually in the study, because that is usually where the answer to my question lives.

11me too 8

TarqWindsor, UK

#21

Juno said something I want to sit with for a second.

What I found is that the abstract and the results section quite often tell slightly different stories, and the difference is the interesting part.

TildaCambridge, UK

#22

I had to learn that an effect being statistically significant and it mattering to a person are two entirely separate questions.

SunilFacilitatorBirmingham, UK

#23

Facilitator note: if the last few replies have touched something for you, there is no need to say so here. Our support resources page lists what to look for, quietly and without any drama.

GwilymCaernarfon, UK

#24

Something practical: I read the methods before the conclusions now, which somebody here suggested and which changed how much I trust what I read.