Features Research
7 questions to ask a researcher

From physical and mental health to building codes and pandemic planning, research plays an important role in determining policy and practice in both the fire sector and society in general. The views of researchers are often held in high esteem and the products of research can translate into new technologies, methods, or treatments.

However, seldom will a single piece of research be able to prove something definitively. In fact, the better the research and the manner in which it is presented; the clearer its limitations, caveats, alternate explanations, possibilities, and cautions. In other words, good research always has limitations. The power of research lies in the bodies of knowledge that accumulate, not in a single study. Understanding this is essential for leaders in the fire sector. It is also essential to maximize the value of research and practice collaborations.

Below are some questions that might be helpful in your conversations with researchers when you are assisting in collaborations on a single study or analyzing an article. It is not exhaustive and there are certainly many others. However, the word “conversation” is the key. Your experience in practice is as valuable as, and essential to, research as the research is to practice.

1. Is the research qualitative or quantitative?
Research is generally divided into qualitative and quantitative research methods. Qualitative research uses non-statistical and systematically applied methods to analyze “data” from interviews, focus groups, document analysis, or researcher experiences. It is generally used to understand experiences, meanings, and develop theories that can later be tested. Quantitative research is used when knowledge is more advanced and allows for more precise research questions. However, don’t be fooled by the use of numbers. In quantitative research, numbers are used to conduct statistical analysis, not simply to create counts or averages. It is entirely possible to have a qualitative study with numbers in it. This leads us to the next question.

2. Are the findings statistically significant and (or) practically significant?
In quantitative research, statistics are often used to understand the extent to which a relationship between two variables may be attributed to something other than chance. If it is just chance, it is not statistically significantly. However, even if a finding is not “statistically significant”, it doesn’t mean that the finding isn’t important and if it is, it doesn’t mean that it matters practically. Similarly, be cautious that some researchers may use the expression “there is no evidence to suggest that”. This means just what it says — there is no evidence — not that something is not true. It may just mean no one has studied it.

3. What was the goal or your research question?
Understanding the research goal in qualitative research or the question in quantitative research is important as it will allow you to trace the methods, results and conclusions back to a starting point around which a good design would be based. A result or conclusion related to something other than the original question may be interesting, but needs to be considered very carefully.

4. Is there reason to be believe this is a causal relationship?
It is often tempting, particularly in quantitative research, to conclude that something causes something else. This is seldom possible. For example, if we find that perceptions of stress in organizations go up with mental health issues, we may say there is a relationship, but this doesn’t mean one causes, the other. It might, but it may also be that the more mental health issues, the more negatively the environment is rated. Unless we set up tests to understand this, we simply don’t know. This shouldn’t create a paralysis by analysis, the proposed conclusion it is may be possible, and one can always improve organizational culture. However, be cautious there may be other factors at play.

5. Are there reasons to believe this research is generalizable?
Particularly in public safety, there is a tendency to conduct studies with convenience samples. This means that the sample of people or responses studied may be those that can be accessed and not necessarily be representative of the general population. Consider, for example, a study done on public safety personnel work environments, then consider that 85 per cent of firefighters work in volunteer departments. Also consider the differences between the structure, resources and culture of fire departments compared to federal public safety personnel or police departments. Are they all the same? They may be, but unless due consideration was given to this, one should not begin with that assumption.

6.What cautions and limitations should I use in interpreting your research?
Some researchers, such as the teams at FireWell and CIPSRT and the NFID database, are very concerned about how lay people interpret research. In the lay summaries of the FireWell research, there is often a section on “cautions”. If a researcher isn’t sharing the cautions to the article interpretation, don’t hesitate to ask!

Perhaps the most important question to ask a researcher — No. 7 — is not about a single study, but about his or her body of work. In other words, what would he or she like you to know, and/or what advice might he or she have to help inform your decision making. Remember, most research is intended to apply to a broad population. Strategic decision making is about leveraging the particulars of your own situation. In conclusion, always consider the possibility, but don’t hesitate to have conversations with researchers. As much as they have knowledge to share, they need your practical insights and experiences as well.

Tina Saryeddine, PhD, MHA, CHE, is the Executive Director of the Canadian Association of Fire Chiefs. You can learn more about the CAFC at www.cafc.ca.