Post hoc analysis is when researchers go looking for patterns in data. (Post hoc is Latin for after this.) Testing for statistically significant associations is not by itself a way to sort out the true from the false. (More about that here.) Still, many treat it as though it is - especially when they havent been able to find a significant association, and turn to the bathwater to look for unexpected babies. Even when researchers know the scientific rules and limitations, funny things happen along the way to a final research report. Its the problem of researchers degrees of freedom: theres a lot of opportunity for picking and choosing, and changing horses mid-race. Researchers can succumb to the temptation of over-interpreting the value of what theyre analyzing, with convincing self-justification. (See the moving goalposts over time here, for example, as trialists are faced with results that didnt quite match their original expectations.) And even if the researchers dont read too much into their own data, someone else will. That interpretation can quickly turn a statistical artifact into a fact for many people. Lets look more closely at Significus pet hate: post hoc analyses. There are dangers inherent in multiple testing when you dont have solid reasons for looking for a specific association. The more often you randomly dip into data without a well-founded target, the higher your chances of pulling out a result that will later prove to be a dud. Its a little like fishing in a pond where there are random old shoes among the fish. The more often you throw your fishing line into the water, the greater your chances of snagging a shoe instead of a fish. - The trap of assuming causal relationships from correlations in cases where only coincidence is involved has been one of the most persistent and difficult errant cognitive habits for humans to absolve ourselves from. The ability to quickly recognize such patterns is incredibly usefully, particularly in cases where a causal relationship actually does exist, but also leads to a lot of erroneous beliefs when not tested and re-examined more rigorously. I would guess that most superstitions are a product of this type of thinking, as well as a lot of other faulty conclusions (sometimes even in science). Via Got Proof? statistically-funny.blogspot.co.uk/2014/03/if-at-first-you-dont-succeed.html
Posted on: Sun, 02 Nov 2014 22:59:50 +0000