Do your own research. But do it right. (Patreon)

[This is a transcript with links to references.]

I’ve noticed that a lot of scientists on social media make jokes about people who do their own research. This has been going on ever since COVID and it’s only been getting worse. We’re now at the point where they’re claiming you’re only allowed to talk about a topic if you’re an expert yourself. Don’t you dare commenting on anything you don’t have a PhD in!

I think this nonsense has been going on for long enough. So I’m here to say: It’s of course completely okay if you do your own research, provided you do it right. But how do you do that? That’s what we’ll talk about today.

This video is not a guide for writing a PhD thesis, setting up a placebo-controlled drug trial, or building a particle collider. I’m not talking about how to do a scientific research project. I’m talking about how to research resources that you can for the most part find online. It’s this context in which the phrase ‘doing your own research’ has become common.

As with most problems on social media, the idea that you shouldn’t talk about topics you don’t have a degree in started from a good place and then went horribly wrong.

It began in the early days of the COVID pandemic when few people knew what was going on, but they were difficult to hear amidst the noise of non-experts who generously handed out uninformed opinions. People who “did their own research” became a running joke for good reasons, and quite a few of them ended up on emergency wards.

Unfortunately, the joke continued after the reasons were gone. And so here we are today, where you’re not allowed to quote from governmental reports on, say, gender affirming care, because that counts as “doing your own research”.

And yes, I am talking about this because it personally affects me. I frequently get people complaining, both here and on other social media, that I’m talking about topics I’m not an expert in. Marijuana, climate change, obesity, bees, ai, neurodiversity. I don’t know much about any of those, or at least I didn’t until I had to figure out how to pronounce tetrahydrocannabinol, yet that hasn’t stopped me. Some people really really don’t like that. You’ve probably seen them in the comments here saying I should “stick to physics”.

Indeed, why doesn’t Sabine stick to physics? I talk about other topics not because I like to annoy people. Though, now that I think about it, that’s definitely a bonus. But mostly it’s because I like to be well-informed about what’s going on in science, engineering, and technology, broadly speaking, not just my own niche in physics. And I’ve found it difficult to find good information, that is (a) comprehensible (b) doesn’t dumb things down, is (c) politically neutral and (d) brief. So I decided to do it myself.

Unlikely as that might have seemed three years ago, I’m now the owner of a science news business. When I talk on this channel, I don’t talk as a physicist, I talk as a science communicator with 20 years’ experience backed up by a team of script writers and fact checkers. Of course, I talk about topics I’m not an expert in myself. Do you think the guy in the evening news is an expert on child soldiers in Congo?

But I’ve found it difficult to find people to work for me who are good at doing their own research. And so I was thinking, I’ll do a video and explain how to do it.

Before we talk about doing your own research, let’s talk about when *not to do it. That’s when there’s nothing to research, when there’s no information online, other than that coming from people who know more than you do.

The early days of the COVID pandemic were an example of that. If you didn’t know your way around with virology and epidemiology you couldn’t make sense of the little data that were available. Another case may be when information is classified. Nothing to research, so all you find online is just noise. In such cases, the best thing you can do is shut up listen to the experts. This is why I don’t talk about UFOs.

This brings up the question of how to recognize an expert. That’s a good question but it would lead us somewhat astray. If you’re interested in it, let me know and I’ll do another video on it.

A big part of doing your own research begins before you research anything. Before you start, spend a moment to think about what you can reasonably expect. It takes about 10 years to do a PhD. It takes another 5 to 10 years to get a good overview on a research area. You’ll not reach the same level of understanding by spending a few hours on Google. No, you won’t.

You’ll have to draw a line somewhere, usually that’s at a point where it just doesn’t matter for your purposes. For example, when I was talking about whether to legalize marijuana, I didn’t really need to know the biochemical pathways of THC. When I was talking about climate models, I didn’t really need to know how to numerically integrate the Navier Stokes equation.

I find that it helps to make a list of concrete questions I want answers to before I even start. That prevents me from going down rabbit holes. Sometimes.

Besides having reasonable expectations, I also have to constantly remind myself to be honest with myself and acknowledge what I don’t understand. I know from past experience that if I start glossing over incomprehensible sentences, that much increases the risk that I make mistakes.

A good example is the blunder in my video on the greenhouse effect, where I said that the gravitational force decreases with !!altitude and that’s why air density decreases.

Now it’s right that the gravitational forces decrease with altitude and it’s right that the air density also decreases with altitude, but the former isn’t the reason for the latter. Obviously not. If you think about it for more than a millisecond that doesn’t make any sense, the density would decrease even if the force didn’t depend on altitude.

And why did I get this wrong? Because the combination of words sounded kind of plausible and I didn’t think about it. I left the video up the way it is because it doesn’t matter for anything else I said there, but I’m still embarrassed about it.

Another example is the story about how the core of Earth supposedly started spinning in the other direction, remember that? Someone from my team put this into the script that I was supposed to read. I couldn’t see how that was possible. Where did all that angular momentum go? How could the core suddenly start going the other way when it’s coupled to the mantel through a liquid layer in between. Just like that?

Wouldn’t you at least expect a very good explanation for how this can possibly happen? I didn’t understand what my script writer had written, so I had a close look at the paper. And sure enough, it turned out that the paper wasn’t about the absolute rotation of the core, but the rotation relative to that of the mantel. The core of earth used to spin a little bit faster than the mantel, now it seems to spin a little slower. And yes, that actually does have a tiny effect on the length of the day. Suddenly it all made sense. Now I understood it.

Why did I not just read what my script writer had written and make the same mistake as many others? Because I didn’t understand it and I didn’t want to say something I didn’t understand.

So when you do your own research, keep track of what you do and don’t understand.

Something else to get out of the way before you start is acknowledging your own biases. If you’re doing research on a topic that you have previously put forward strong opinions about, you might be reluctant to admit that you were wrong. The best way I find to deal with this is to remind myself that admitting to being wrong is the hallmark of a good scientist. Yeah, I still don’t like it. But it does help a little. I find it difficult especially with topics that I have worked on myself, such as specific dark matter models.

Another factor that can bring in bias is if you or someone you know is personally affected. Suppose you had an expensive and painful procedure that a doctor said would help you, but new research finds that this intervention doesn’t have any clear benefits. You’re likely to insist that it worked for you anyway, because that makes a good psychological protection. And that’s fine if it’s just your personal way of dealing with it. Not so fine if you tell others the study is wrong because *you don’t like the result.

So we have reasonable expectations, honesty with yourself, acknowledging biases. And that puts us in a good position to start with the actual research.

I suggest you start your research by looking for peer-reviewed review articles, governmental reports, lecture notes, and textbooks. Then. You read them.

Yes, this is going to take time. No one said it would be easy! But you don’t have to read all of this in great detail. Those sources usually begin with an overview, and most of the details that follow later won’t be so important for you. The important thing to take away is what the key questions are that you need to pay attention to.

A review article is basically an overview on the status of research, and some scientific journals specialize in them. The nature group publishes a lot of them, and if you can find one on the topic you’re interested in, that’s usually pure gold.

You find governmental reports either on the governmental websites or that of the ministry of science and education or whatever it’s called in your country. Other good resources are specialized national institutions such as the National Academy of Sciences in the USA or the Fraunhofer Institute in Germany.

After you know what you need to pay attention to, you can move on to meta-reviews, if they exist in the field you’re looking at. A meta-review is a summary of previous research studies. Meta reviews can be great and save you a lot of work, but enjoy them with some caution. A summary of crappy studies does not make a good study. You should only trust meta-reviews if they assess the quality of the studies they summarise, and not just lump together different data.

And after that you can move on to the most recent studies. Then you should have a pretty good impression of what is going on in the field and be well prepared to put new stuff into context.

Okay, now let’s talk a bit about how to search for scientific literature and how to understand what you’re looking at.

First of all, if you’re searching for scientific literature, I suggest you don’t use Google’s main interface, but you use Google Scholar. Indeed, in many cases Google will suggest that you switch to scholar. Google scholar shows you quickly whether a paper has been published in a scientific journal, and it has an extremely useful feature which is that it lets you search for articles that have referred to an earlier article.

This is great if you have a fairly old reference, and you want to know what’s happened recently. All you have to do is search for all papers that have cited the old one, and sort them by date. I did this a lot when I was working on the video on collective intelligence. That’s because only thing I knew about the topic were some of the classical studies in the field, and looking for recent papers that cite the classics is a good way to find reviews.

Next thing you need to do is figure out whether an article has been peer reviewed. At present, almost all peer review is done at submission to a scientific journal. If the paper passes peer review, it will be accepted for publication. So you have to check whether the paper has a journal reference, and if it’s a decent journal. It can be difficult to figure out what’s with the journal, but if you don’t know the journal, at the very least check if it’s on the list of predatory journals because such journals will basically publish anything for money.

Also make sure that what you’re looking at is not a pre-print server. Pre-print servers should really be called pre-peer review servers. It’s where scientists upload papers before they submit them to a journal. You should stay away from pre-prints unless you can consult an expert. The most widely used pre-print servers are the arXiv in physics and related disciplines, and the bioRxiv in the life science, though there are a few others.

Also be very careful with reports or surveys from non-profit organizations or corporations because these are rarely peer reviewed. It’s not that they’re necessarily wrong, but that they often don’t disclose how they arrived at their results, and there’s a risk they have omitted relevant information.

Conference proceedings can be difficult to understand because their purpose strongly depends on the field. In some areas, like particle physics, conference proceedings are where you dump preliminary results or summaries of your own work that wouldn’t get published in a journal anyway, not because they’re wrong, but because they’re not new or not interesting enough.

In other fields, like engineering or computer science, conference proceedings are the final product. They do their research, present it at a conference, write it up and publish it in the proceedings. And that’s that. They’re usually peer reviewed, or at least they’re supposed to be, so don’t dismiss conference proceedings. In some fields they’re vital.

Let’s then talk about some things you should not do.

The number one mistake that I see among people who “do their own research” is that they just pick one or two papers that kind of say what they want to say and leave it at that. This is not how research works.

The second most important “don’t” is to never trust a second-hand reference. Don’t trust websites referring to other websites, press releases referring to papers, newspapers referring to reports, or magazines quoting scientists. Don’t trust books referring to other books, don’t trust Wikipedia, and don’t trust people talking on YouTube. You have to track every piece of information down to its source, and then check how reliable that source is.

That is not to say that second-hand references are useless. They are pretty good for pointing you into the right direction. Wikipedia in particular is a good starting point to launch into the scientific literature. Just don’t make it the end point.

My final advice for today is that if there’s data, look at the data, don’t trust the text. I see this all the time that data in a paper don’t support what the abstract or the conclusion says. For this you’ll have to know what p-values and R-coefficients are and stuff like that, and that would make for another video entirely. So for today let me just say that while outright scientific fraud is rare, it happens all the time that written text is sloppy or doesn’t properly represent the data.

Ok, in summary. Before you set out to do your own research, clarify for yourself which questions you are trying to answer and if you have a reasonable chance of answering them in the first place. Acknowledge the limits of your understanding and possible biases. Once you’ve done that, start with the basics and try to get an overview by help of reviews and reports. Then work your way up to more recent publications. Don’t cherry pick studies, don’t trust text if there’s data, don’t trust second hand sources, and don’t trust me either. Did I forget anything important? Let me know in the comments.