How to Evaluate a Scientific Study

The advent of the Internet has brought us many things, good and bad. But one think that is particularly troubling to me is the internet’s treatment of science. The Internet has given a voice to alternate scientific theories that they probably would not have had otherwise. It’s great that we can disseminate information more quickly, but it’s not so great that some of that information is total bunkum. In fact, Popular Science chose to close their comments section because they felt that certain comments which make alternate theories seem as legitimate as well-studied one were undermining their mission. You know that commercial where it says everything on the internet it true? “They can’t put anything on the internet that isn’t true.” Well, sometimes I feel like we are living in that world. I don’t think our understanding of science/scientific literacy is very high and I do think that it is hurting us.

So today I present how to evaluate a scientific study. I am going to try and use examples that are mostly made up and our completely ridiculous sounding so you don’t get bogged down in the politics of some of the psuedo-science that is out there and instead focus on the point of what I am saying and in hopes that I don’t offend anyone. I hope that when you finish reading this post, you will be armed with more tools to help you understand science so that you can make better informed decisions.

Correlation/Causation

This is one of the biggest ones to tackle and one of the least understood ones. To help illustrate the difference, let me show you this graph I found.

This graph was made by someone practicing with a graphing program using real data. These are real numbers and this graph is real. Now, you could take this data and say “Oh no! Organic food causes autism! If you eat organic food you will get autism.” But that would be assuming causation. That would be assuming, basically, that because they both show a climb at roughly the same time, that one must make the other one happen.

Which is where correlation comes in. The definition of correlation is, “the degree to which two or more attributes or measurements on the same group of elements show a tendency to vary together.” Okay, now you might be asking yourself what does that mean? Basically it means that two things show up together but that does not necessarily mean that one is caused by the other. Many studies that come out are correlational because it is very hard to prove causation concretely, but many people take those studies as one thing causing the other. This could be dangerous for many reasons. Just look back to the graph above. Just because it’s a pretty graph and it looks nice and it shows two trends growing at the same time does not mean one is caused by the other. Do you honestly think that organic food is causing autism? I hope not.

So when you see a study try and ask yourself, is x really causing y or are they just two correlated things? Perhaps there is a third variable that has been overlooked that is causing both of them instead of one causing the other. And it’s also to remember, just because someone can put it in a pretty chart doesn’t make them related either. I hope I am making sense in this section, but if you’ve read it and you still don’t understand, I’m happy to answer any questions in the comments.

Biases

Another problem is biases. Often times, there can be biases involved in the research. There are many different kinds of biases, but I’ll try and cover two of the bigger ones.

Confirmation bias is the first one. This is what happens when you already believe something and then you do research and it backs it up. It’s called confirmation bias because when you already believe something is the case, sometimes you unconsciously (or consciously) manipulate the data to confirm what you already know. It’s like if I think penguins are the best bird and then I do a study comparing them to other birds and find that indeed, they are the best bird, I could have been only looking at specific things so that I reach the specific conclusion I was already inclined towards.

The second has to do with having a financial bias. You don’t want a study done by someone who stands to benefit financially. Money is a powerful motivator for people and the love and want of money can lead people to do less than credible science. For example, if I was the president of the National Association of Snozzberry Farmers and I put out a study saying that snozzberries were the best possible fruit you could eat, there could be some financial bias interfering with that study, since a study saying they are super-healthy for you has the potential to lead more people to buy them, which has the potential to put more money in my pocket.

These biases are what make it so important that you not only understand what the science says but who was doing (or funding) the science. You will never be able to eliminate bias one hundred percent, but you can definitely weed out the ones that are more biased than others.

It’s also important to be aware that you might be personally biased to accept studies that fit in with your worldview no matter how poorly done they are. You probably can’t remove your bias totally, but I think it helps if you can be aware of it. If you are aware of it, you can help yourself to be more open to looking at studies that present conflicting evidence or of taking a closer look at the ones that agree with you.

Peer Reviewed

Good science is able to be looked at by other scientists. Those other scientists will be able to say, yes this science was conducted in a manner that meets scientific standards. This is called peer-review. Those studies are then usually published in a (wait for it) peer-reviewed journal. Being published in one of those is a sign that it has been looked at by other scientists. Unfortunately, these may be difficult to read online since many you have to may for, but some libraries (especially academic libraries like at a college or a university) may have access to them if you really want to read more than the abstract.

Now certainly, the peer-review process is not perfect since it is run by humans, it has it’s critics and they have valid points, but at least there are some checks and balances on the process. There is a big difference between something like the journal Analytical Chemistry and a website like ilikechemistry.com. As long as you have the money, you can have a .com, it’s that simple. I mean, if I can buy a .com anybody can buy a .com. And anybody can write anything they want on the internet. If I had gotten my website and I started writing that my name was Rosie Braunenhauser and I had a Ph.D. in underwater basket weaving, I could have done that because there is nobody stopping me from just making up whatever I want to make up. Peer reviewed journals might not be perfect, but they are at least a starting point.

Sample Size

Generally, the bigger the sample the more accurate the results of the study are likely to be. If you look at something among 100 people, you are going to get a part of the picture, but not as much of the picture as if you looked at 10,000 people. And who the sample is matters also – was the sample all men or all women or all white people or all fill-in-the-blank here? It matters because it helps inform the picture. Say I did another snozzberry study and I fed snozzberries to 100 men and they liked them and then I went and said “People like snozzberries.” That wouldn’t be accurate, because I didn’t include groups of people like women and children. It would be more accurate to say “Men like snozzberries” and it would hold even more weight if I had gotten 10,000 men to eat snozzberries and give them their feelings on them. Basically, what I’m trying to say is the larger and more diverse the group is, the more you can apply those results to the general public. Sometimes studies are done on such a small and select group of people that they can be applied, but only within narrow confines and not necessarily to everyone.

This is not to say that small studies never have merit – sometimes whatever is being studied is rare and so you are only able to have a limited set of data. It is just to say that the more people we are able to study, the clearer and more accurate picture we are able to have.

Methodology

How the study is conducted matters, it matters a lot. It matters because of the factors of bias that I mentioned before. The sort of gold-standard in studies is randomized double-blind controlled studies. What this means is that some people are given a real drug and some people are given a placebo (one that looks like the real thing but without the active ingredients) in the control group (which is the group that is used to measure against the test group to make sure that the results aren’t just random and couldn’t come about by just anything) and who gets what is decided by random and neither the person administering the test nor the person getting the test knows what they are receiving. This can eliminate a lot of bias (like confirmation bias where people expect the drug they are getting to work and so it does).

However, just because a study is not randomized double-blind doesn’t mean it has no merit. Randomized double-blind studies are not always ethical in certain situations or on certain populations. In these cases, more observational or data studies are used where we look at numbers and facts after the fact to try and determine if we see a pattern. An example of this would be a lot of the data on what is safe in pregnancy. It would be unethical to just give pregnant women drugs to see if they caused harm to her or the baby, however sometimes women need to be on certain drugs because they have a medical condition that is more important to treat than to worry about the risks and after the fact we can look at the data and see if it could generally be considered safe or unsafe based on our observations of people who had to take it.

The Danger of Anecdotes

I’ve covered everything that I have to cover on science, so lastly I want to talk a little bit about anecdotes. Anecdotes are often designed to make us feel highly emotional and the person saying them usually truly believes the cause and effect to be true, but anecdotes are not science and they don’t make the cause and effect true. And you can pretty much have an anecdote for everything and many, many things that are not related or that are correlated get turned into cause and effect when you come to anecdotes. Plus, anecdotes can swing in both ways. And people are especially susceptible to believing the cause and effect in anecodotes when it already agrees with their world view. 

To give you an example, a few weeks ago I got sick. I went to bed and then I woke up later and threw up. I could say now that I am never sleeping again because since I went to sleep right before I through up, clearly the sleeping caused the throwing up. Or for another example – at nineteen when I had my car accident I was listening to a country music CD. I could say now I am never listening to country music in the car again because it caused my accident and I don’t want to have another accident. These are obviously silly examples, but do you understand what I mean? You can often times tie two things that aren’t related to be together  in an anecdote and with enough emotion, you can believe them. It’s important to be aware that they are out there and that does not make them true or related things.

Along these same lines are those statements “I did x and I’m fine.” Just because you did something (or didn’t do something) and are fine does not make whatever you did or didn’t do safe or right. Most things don’t happen to every person – there are always outliers and anomalies – and  just because you managed to do something and turn out okay doesn’t mean that it’s a good idea to keep doing that thing. I’ll give you a silly example: say I got into a car accident and lived (which I did), does that mean I should say “Oh I got into a car accident and lived, now I can just crash my car because I know that car accidents are survivable.” It would be insane logic to purposely crash my car because I lived through an accident previously and ignores all the other people who have died from car accidents. If you did something that we now consider unsafe and you were fine, it doesn’t necessarily mean it’s fine to do that thing, it more means that you got lucky. You may have survived but not everyone has. I really dislike that argument, but I’ve probably used it at some point because we tend to want to use our experiences to justify/validate things.

————————————————————————————————————————————–

All right, I hope I have explained myself well and I hope you have a better understanding of scientific studies after this! If there’s anything you don’t understand, please let me know so I can try and clarify. And if there’s anything I forgot or left out, let me know that too so I can add it in. I know it all made sense in my brain, but I want it to make sense in other people’s brains and these are things I’ve known for a long time and so sometimes I act like everyone knows them, but that’s not true. I would love to make this as clear as possible, so if anything isn’t clear, please feel free to let me know so I can make it clear.

Related Posts Plugin for WordPress, Blogger...

Post a comment

CommentLuv badge