Interlude 1: Evidence

leigh from the path forward made a  big point on part 1,

i think one of the most important skills you can convey as a physician is how to evaluate the quality/credibility of a source. the general public is quite open to a wide variety of definitions of “expert” while we in the scientific community are far more particular

Absolutely correct, so we’ll start with a short discussion on evidence, credibility will come in another post later.

When I look at articles, or anything else cited as evidence, I’m evaluating a lot of things, including experimental design, data analysis, and statistics. But don’t fret! You don’t need to take a class in statistics or get a Doctorate degree to understand levels of evidence.

In evaluating evidence, physicians often follow something like what’s shown on this page:

If that all looks like greek to you, don’t worry. It basically is. The ranking system there essentially ranks types of evidence presented in a journal article from 1 (best) to 5 (worst).It’s not actually rocket science, there are a few general guidelines. I’m not going to go number by number, but just give general guidelines based on this system that parents can follow. *HUGE DISCLAIMER: I’m not a scientist, I’m a doctor. I do not have masters, or a doctorate of any sort. When real scientists comment below that I’ve oversimplified, and have missed important things, they’re absolutely right. The guidelines actually make intuitive sense, once you get the basic question.


The basic question that underlies evidence based medicine is this: If I were going to tell you something amazing, that flies in the face of the world as you know it, what would it take to convince you? 

Well, what if I called my friend, an expert on the subject and he said “Well, I haven’t really done anything to back this up, but this is what I think?” Not that convincing right? That’s the lowest level of evidence, expert opinion does not good evidence make.

What If I tracked a few patients with our disease in question over time? I cherry picked these patients based on what fits into my study. Well it’s better than just telling you right? Still not great evidence though. 

How about this then: I just looked up all the cases of a particular disease in our community, and then I compared them to similar people without the condition. This is a case control study. As you can imagine, since I didn’t just cherry pick a few cases, and since I’m comparing them to people without the condition (a control group!) this is a better study type. This kind of study was among the first to show the link between tobacco and lung cancer, so we’re getting into more respectable evidence here. The big problem is to have any value, these studies need to follow groups over long periods of time, and follow a large number of people. One step up from this kind of evidence: let’s take several case-control studies on the same thing, and review them together, see if they agree. If they do, even better evidence!

A weakness of the studies mentioned above, is that they all require us to have already diagnosed a disease before we make any comparisons.  So, next we have a cohort study. A cohort study takes a group of people with something in common (let’s say, people who smoke cigarettes), and compares them to a similar group, without that one thing in common (e.g. people in the same community who don’t smoke cigarettes). Then we watch these people, and eventually, we count how many had a particular outcome (say, those who get lung cancer), if those who smoked were more likely to get lung cancer, BINGO, evidence. These studies are expensive, and take foreeever.

Ok, so these are studies that let us know about things that happened already (even in the cohort study, you don’t know if smoking causes cancer until after people get cancer). Those studies aren’t as great for evaluating treatments. For evaluating treatments, the awe-inspiring gold standard is the randomized controlled trial (RCT).  Here, we have different treatments and/or a placebo, and we randomize people to get one treatment or another. There are all sorts of complications with this one, what you need to know is that it’s the best kind of individual evidence. For further information, I recommend the Wikipedia article, and its references I know I know, Wikipedia is a cop out, but I want to get Part 2 out soon and go over evidence!

So what’s better than a RCT? Lots of RCTs, so we pool lots of RCT have and get a systematic review of randomized controlled trials. This is great evidence, especially when the trials all agree closely.

Now you have a basic hierarchy of evidence, so when someone tells you that they’re personal expert (Like say, the DAN! Doctors, who are taught everything they need to know to cure autism in a one day seminar) assure you that vaccines cause autism, you know what to tell them.  

Explore posts in the same categories: Medicine

5 Comments on “Interlude 1: Evidence”

  1. This was totally just going to be part of Hard Conversations, Part 2, but it kept getting bigger!
    Hopefully it’s a useful resource for those who’s careers aren’t science related to get a grip on evidence based medicine.

  2. Avi Steiner Says:

    Try surrounding quotes with “blockquote” and “/blockquote” (except in the form text )

    I think it should work:


  3. Michael Says:

    Short coat, that Wikipedia reference is certainly one of their worst articles, which leads me to another issues with Wikipedia. You can find what other Wikipedia articles internally link to that article. If a lay person was trying to understand why double-blind, randomized clinical trial is critical to understanding why vaccines are safe, that article would not help. However, I link to Wikipedia more than I should, because it’s easy. I’m on a personal one-man mission to point out the flaws in Wikipedia, and even I rely upon them too much.

    • It’s certainly not an amazing education article, but for an article written at a lay person level its not bad. To be honest, if you understand RCTs at the level of that article, you’re doing better than alot of community physicians I’ve worked with.

      But yea, I link to, and use wikipedia far too often.

      I’m not requesting parents to understand scientific articles anywhere near as well as I’d want a physician to understand them so I felt most other explanations on the web were inappropriate.
      The goal with this article was just to give basic levels of evidence, most of the time I see pro-mocktroversy advocates quote a phone survey, or an ‘expert’ opinion, and then write off the standardized trials. This is to explain the problem with that, rather than to really detail how important it is to use double blind RCTs (which I agree, are the cornerstone of evidence based medicine)

      As for RCT’s in vaccine safety, I guess I’m hoping this article will support the understanding I try to give with the next part, which I’m being far too slow in supplying

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: