An Overview of Significance Testing

By Glenn Iwata and Stephanie Apostol

Your boss says he'd like to see regression analysis or even a complete market segmentation applied to a project. Your mind is swimming ... regression ... what does my childhood have to do with this?

Well your good friends at WestGroup are here to help. WestGroup analysts compiled a list of "the most frequently used statistical analyses" that our clients have used in the past few years. This guide can significantly add to your understanding of your data and the confidence to make useful conclusions and recommendations. Today, we will review the most common technique used at WestGroup. Are you ready ... ?

The NUMBER ONE most requested statistical technique is ...

Significance testing — Are men really from Mars and women from Venus? When comparing survey responses from two demographic groups, the most obvious question is, "Do these numbers mean that the two groups are really different?" Just because 70% of men responding to the survey answered "yes" to a question compare to only 40% of women, these differences might or might not actually be "different." Significance testing answers the question.

Nearly all studies with large enough sample sizes involve significance testing. Typically, we will always test for differences between demographic groups (i.e. men vs. women, young vs. old, high income vs. low income,etc.). We will also look at differences unique to each study. For instance, we may compare the responses of PC owners to non-PC owners, or breakdown and compare responses by geographical location. We often compare Valley cities to one another for government work. Are Scottsdale residents "significantly" more likely than other Valley residents to drive SUV's? It is not valid to assume there are real differences between groups unless the differences are "statistically significant."

In a recent study about elder care, we asked respondents about their concern over affordable housing for the elderly. Surprisingly, residents over 55 were not significantly more likely to express concern than younger residents. They were "slightly" more likely to be concerned, but due to sample size, margin of error and levels of confidence, this could not be extrapolated to mean there was an actual difference in the opinions of younger and older residents on this issue. Are you getting it?

So in our reports, when we say women are significantly more likely than men to be right, we're saying it with statistical methods to back it up! (Really, we're not making this stuff up!)