Every so often a piece of academic research related to law enforcement is discovered by the popular media, summarized and used to promote a pre-existing editorial point of view.
If you’re like me, your blood pressure can soar just from reading the headlines this generates, as you know where the commentary is headed.
Instead of being frustrated at how research is used to support various points of view, follow these six tips to interpret the findings.
1. Recognize your own bias
I had a student who was reporting on his study of domestic violence. He quoted the outcome of a research study and then said, “But I don’t really believe that.”
Data that contradicts what we think we know is hard to digest, but the point of research is to challenge our assumptions. Usually research focuses on answering a very narrow question and won’t, by itself, create major paradigm shifts.
2. Recognize bias in the research
Every researcher should know that bias is impossible to avoid. Here are some questions you should ask:
- Who funded the research?
- How was the research question phrased?
- What political issues may have spurred the interest in conducting the research?
- What assumptions are acknowledged by the researchers?
- Is the report full of words that express emotion or political bias? For example, abortion can be phrased as women’s health care, murder, pro-choice, pro-life, eugenics and so on.
Researchers can also claim that “nearly half of those surveyed agreed with X” or lament that “fewer than half of those surveyed agreed with X,” using the same data to make contradicting claims.
3. Read the research paper, not just the abstract or media presentation
Summaries and translations into the vernacular can result in loss of vital clues to the substance of the research. This can be quite a burden, but there’s no other way to fully evaluate a study. If you’re a real research dog, look up some of the key research papers cited in the research you’re reading!
4. Understand the methodology
Most of us make our decisions and conclusions based on our own experience. Your anecdotal evidence could be that small, brown dogs are more likely to bite you based on your experience of having been bitten twice by small, brown dogs. We often perpetuate our beliefs within our expectations. For example, you may be more nervous around small, brown dogs and somehow incite their biting tendencies, solidifying your beliefs about small, brown dogs.
Objective research is peer-reviewed, compares apples to apples, has control groups to compare to the groups studied and uses a sample size that makes sense mathematically to ensure the results can be generalized to a larger group.
5. Understand the peer-reviewed journal
Scientific journals are the garden (and graveyard) of scholarly research. In recent times, the desire to see one’s name in print has given rise to bogus “professional” journals where just about anybody can get published (sometimes for a fee). So having a high falutin’ title is no guarantee the publication is legitimate.
A true peer-reviewed journal means a panel of specialists has reviewed the research report and verified that appropriate research and ethical standards have been met. Research that hasn’t been peer-reviewed, or can’t be duplicated, is less credible.
6. Read the limitations and analyze the need for further research
A standard component of research reports is the authors’ admissions of what questions were not answered and what remains to be discovered.
Most research focuses on a narrow topic that might not apply more broadly than to the very specific hypothesis (research question) that it was designed to explore. If we find that the brown-spotted tsetse fly has a breeding season of four hours, does that necessarily mean that the red-striped tsetse fly has a four-hour breeding season?
Many policies are influenced by academic studies. Our distaste for ivory tower opinions must not blind us to the reality of the impact these studies have on our daily work.
