\u201cA study showed that\u2026\u201d<\/p>\n
This is a sentence after which a discussion sometimes seems to stop. A study showed it. Therefore, it must be proven.<\/p>\n
In fact, this is precisely where the most interesting part should begin.<\/p>\n
What exactly did the study show? Who or what was studied? What was it compared with? What was measured? And perhaps most importantly: is the conclusion we draw from it actually the conclusion the results allow us to draw?<\/p>\n
This does not mean that we should distrust studies. Quite the opposite. Research becomes much more useful when we understand which questions it can answer and which it cannot. One idea from a Judy Morgan video has stayed with me: a study can be interpreted or presented in a way that supports what we already want to prove. For a long time, I thought this happened only in conventional medicine. Over time, however, I have realised that exactly the same thing can happen in integrative veterinary medicine. A single statement is taken from a study, then a huge leap is made to a conclusion that the research does not actually support. For example, finding cancer-related cellular or biological effects of sulforaphane is not the same as saying that \u201csulforaphane prevents or treats cancer\u201d.<\/p>\n
As human beings, we tend to ignore evidence when we very much want to believe a particular claim. I did the same.<\/strong><\/p>\n Veterinary medicine uses the principles of evidence-based veterinary medicine (EBVM) to address this. The idea is not to find one \u201ccorrect\u201d study, but to combine the best available scientific evidence with clinical experience and the circumstances of the individual animal.<\/p>\n A study generally describes what happened, on average, in the group being studied. But the animal living in your home is not an \u201caverage dog\u201d or an \u201caverage cat\u201d. Age, health, medication, medical history and many other factors can affect how an individual responds to the same intervention.<\/p>\n Even if most dogs in a study tolerated a product well, that does not automatically mean that every single dog will tolerate it.<\/p>\n You do not need to be a scientist or statistician to interpret research sensibly. Five simple questions can already tell you a great deal.<\/p>\n This is the first thing worth checking, even before reading the results.<\/p>\n Was the research carried out in dogs? Mice? Humans? Cell cultures? Were the subjects healthy adult dogs or patients with a particular illness? If dogs were studied, which dogs: young or old, one breed or several, family pets visiting a clinic or animals kept in laboratory conditions?<\/p>\n These details are not decoration in a scientific paper. They determine how well the findings can be applied to the dog actually standing in front of us.<\/p>\n If a substance affects a cellular pathway associated with inflammation in a laboratory, that may be a very interesting finding. It may justify the next study. But we cannot yet conclude that the same substance reduces pain in a dog with arthritis.<\/p>\n There are many questions between those two claims. Is the substance absorbed? Does it reach the relevant tissue? At what dose? Is that dose safe for dogs? And does a change in a biological marker ultimately mean that the dog moves better or feels less pain?<\/p>\n Likewise, findings in one highly specific group of dogs do not necessarily apply to every dog.<\/p>\n Evidence-based veterinary medicine recommends defining the population, intervention, comparison and desired outcome as precisely as possible when formulating a question. This is known as the PICO framework. Its purpose is straightforward: before looking for an answer, we need to know which question we are actually trying to answer.<\/p>\n So the first question is: are the animals or model in the study sufficiently similar to the situation about which the claim is being made?<\/strong><\/p>\n Not all studies answer the same kinds of questions.<\/p>\n A randomised controlled trial (RCT) can be very suitable for comparing the effects of two treatments. But this does not mean that an RCT is always, automatically, the best type of study for every question.<\/p>\n If we want to know a disease’s prognosis or the possible long-term effect of a risk factor, a cohort study may be more appropriate. Evaluating a diagnostic test calls for a study that compares its results with a suitable reference method. A cross-sectional study may be the right tool for estimating disease prevalence.<\/p>\n So it makes little sense to ask only, \u201cHow high up the evidence pyramid is this study?\u201d<\/p>\n A better question is: does the study design match the question it is trying to answer?<\/strong><\/p>\n A systematic review or meta-analysis does not magically turn weak original data into strong evidence. If the studies being combined are small, highly diverse or at high risk of bias, the final conclusions must take that into account.<\/p>\n And conversely, a study may belong to a supposedly \u201chigh-level\u201d category, but if it was poorly conducted, the label RCT will not rescue it.<\/p>\n Study quality is influenced by sample size, randomisation, blinding, possible biases, measurement methods, statistical power and how comparable the groups actually were, among other things.<\/p>\n The evidence pyramid is a useful map. It is not an elevator that automatically takes every study to the correct answer.<\/p>\n Suppose ten dogs with joint problems receive a new supplement, and eight of them move better a month later. It sounds promising.<\/p>\n But we still do not know what would have happened without the supplement.<\/p>\n Might their symptoms have improved over time anyway? Were the dogs also receiving pain medication? Did they lose weight? Did their exercise routine change? Did the owners know that their dogs were receiving a \u201cgreat new product\u201d and therefore evaluate their condition more optimistically?<\/p>\n This is why a comparison group matters so much in research.<\/p>\n In a sound comparison, the groups should be as similar as possible and differ primarily in the factor being studied. Randomisation helps distribute known and unknown differences between the groups, while blinding helps reduce the possibility that a researcher\u2019s or owner\u2019s expectations influence the assessment.<\/p>\n Not every study can or should be conducted as a double-blind, placebo-controlled trial. For some questions, this is neither practical nor ethical.<\/p>\n Nevertheless, the question remains the same:<\/p>\n What are we comparing the outcome with, and does that comparison allow us to say that the observed difference was caused by the intervention being studied?<\/strong><\/p>\n \u201cIt got better afterwards\u201d is an observation. \u201cIt got better because of this\u201d is a causal claim.<\/p>\n An entire study design lies between those two statements.<\/p>\n This question is especially important for health and supplement claims.<\/p>\n A study may demonstrate a very clear change in blood results, cells, enzyme activity, an inflammatory marker or another laboratory value. But was the question really about a laboratory marker, or did we want to know whether the dog actually feels better?<\/p>\n Those are not always the same thing.<\/p>\n A biological mechanism can be entirely real without producing any noticeable clinical benefit. Conversely, a clinically important effect may exist even when we cannot yet describe its entire mechanism precisely.<\/p>\n That is why we should look at what the study\u2019s \u201coutcome\u201d, or outcome measure, actually assessed.<\/p>\n Did it measure:<\/p>\n Veterinary medicine also uses validated owner-completed questionnaires, because dogs cannot tell us their pain level on a scale. This means an owner\u2019s assessment is not automatically \u201cunscientific\u201d, but the measurement tool should be appropriate and as reliable as possible.<\/p>\n The difference between statistical and clinical significance is also important.<\/p>\n A result can be statistically significant but so small that it makes no practical difference in the dog\u2019s daily life. EBVM therefore emphasises that a statistically significant difference is not automatically a clinically meaningful one.<\/p>\n So ask: did the study measure the outcome that the marketing copy, social post or health claim is now talking about?<\/strong><\/p>\n If the study measured an inflammatory marker, that should not quietly become \u201creduces pain\u201d.<\/p>\n If it demonstrated a change in the composition of gut bacteria, that should not automatically become \u201ctreats allergies\u201d.<\/p>\n The steps in between also need evidence.<\/p>\n This is where a study can turn into a much bigger story than the study itself.<\/p>\n A study may find an association. In the headline, it becomes a cause.<\/p>\n A study may demonstrate a possible mechanism. In an advertisement, it becomes a treatment effect.<\/p>\n A small pilot study may produce an interesting result. On social media, that turns into \u201cscience has proven it\u201d.<\/p>\n To avoid this, consider how strong the wording justified by the study really is. \u201cWas associated with\u201d is not the same as \u201ccaused\u201d. \u201cMay affect\u201d is not the same as \u201cimproves\u201d. \u201cWas observed in this study\u201d is not the same as \u201cworks for every dog\u201d.<\/p>\n And a statistically significant result alone does not tell us how large the effect was, how precisely it was estimated or how well it can be replicated.<\/p>\n With any individual study, it is worth asking how it fits into the wider body of knowledge.<\/p>\n Have other studies found the same thing? Are the findings contradictory? Was the sample large enough to give the researchers a reasonable chance of detecting a meaningful difference? How large was the effect? How uncertain is that estimate?<\/p>\n These questions do not turn a good result into a bad one. They put the finding into proper perspective.<\/p>\n For me, there is an important boundary in critical thinking. The goal is not to reach a point where we can believe nothing because every study has limitations. Perfect studies practically do not exist.<\/p>\n Nor does evidence-based veterinary medicine mean basing a decision on a single scientific paper. The best available scientific information must be combined with clinical experience, the individual patient\u2019s circumstances and the owner\u2019s possibilities and values.<\/p>\n Likewise, a gap in knowledge should not mean that practical or integrative experience is simply discarded. Experience may be a valuable signal and suggest an excellent research question.<\/p>\n But experience should be called experience. A biological mechanism should be called a mechanism. An association should be called an association. And a clinically demonstrated effect should be called a clinically demonstrated effect.<\/p>\n The same standard should apply to a medicine, supplement, medicinal plant, dog food or veterinary treatment.<\/p>\n If you do not have an hour to take a study apart, start with these five:<\/p>\n Who or what was studied?<\/strong> Dogs, humans, mice or cells? How similar were they to your dog? Were there other important differences or accompanying factors that could have influenced the result?<\/p>\n What type of study was it?<\/strong> Did its design fit the question in the first place?<\/p>\n What was it compared with?<\/strong> Was there an appropriate control, or could something else explain the change?<\/p>\n What was actually measured?<\/strong> A laboratory marker or a health outcome that genuinely matters to the dog?<\/p>\n What conclusions does the result support?<\/strong> Are we talking about an association, a possible effect or a fairly well-established clinical outcome?<\/p>\n These five questions will not always provide an immediate answer. But they help us recognise when an answer is strong and when the story has run several steps ahead of the research.<\/p>\n And sometimes the most accurate conclusion is: This is interesting. But first, let\u2019s look at what the study actually showed.<\/strong><\/p>\n RCVS Knowledge. EBVM Toolkit: formulating a clinical question, understanding study designs, critical appraisal and applying evidence.<\/a><\/p>\n Veterinary Evidence. Guidance for writing the clinical bottom line: matching study design to the clinical question and weighing research quality and the overall strength of evidence.<\/a><\/p>\n RCVS Knowledge. EBVM Learning: principles of evidence-based veterinary medicine and applying evidence in clinical practice.<\/a><\/p>\n1. Who or what was actually studied?<\/h2>\n
2. What type of study was it?<\/h2>\n
3. What was the result compared with?<\/h2>\n
4. What was actually measured?<\/h2>\n
\n
5. What can we actually conclude from the result?<\/h2>\n
\u201cA study showed\u201d is not the final answer<\/h2>\n
Five questions to ask about the next health claim<\/h2>\n
Sources and further reading<\/h2>\n