(a) "the quality of being important" and
(b) "the quality of being statistically significant".
These two short statements, one following right after the other, reflect the origins of a widespread and profound misunder-standing, which many people are not sufficiently aware of. In a nutshell: we tend to equate practical significance with statistical significance, and vice versa. This, however, is at best flawed.
We can often hear people say that some research has come up with significant results. It is then not immediately clear whether they are referring to statistical or practical significance. Yet, more often than not, they in fact seem to quote statistical significance but implicitly mean practical importance, assuming that the former would automatically ensure the latter.
While we would certainly hope to always arrive at research results that are at the same time statistically significant (the overriding concern of many academic researchers) and of practical relevance (the main concern of managers), the two are by no means perfect substitutes of each other. In fact, we can easily come across research findings that are statistically significant, but otherwise devoid of practical importance (and vice versa).
For example, a new production process might reduce the failure rate by 10%. This reduction might prove to be statistically significant. However, whether it is also of practical relevance depends primarily on whether, in the given context, we consider such reduction as big enough to warrant the additional costs that the implementation of the new process might entail.
To complicate matters further, the opposite can also happen: Practically significant improvements due to the introduction of a new policy might be dismissed prematurely because of a lack of statistical significance, which in turn might be simply a result of a poorly conceived research design.
So, it is never statistics that can tell us what effect we should consider as "important" or "negligible" - this has to be established by us and before we do statistical analyses. Furthermore, a valid research design is crucial for ensuring that statistical significance can be established or ruled out confidently.
It is for the above reasons that we at PracSig pay particular attention to the initial phase of the consulting process, i.e. the proper understanding and framing of the client's concerns and aspirations so as to devise an adequate research methodology and strategy. For more details on how we work, click here.
