- Introduce statistical significance
- Learn how to perform the Chi-square test for significance.
- Learn how sample-size and standard deviation affect the level of significance.
- Our studies are typically based on samples for reasons of economy.
- All of the public opinion studies used in our labs and assignments thus far represent only a sample of a population.
- Generally, we use sample findings as a basis for inferences about some population.
- The sample (n) consists only of those individuals who were directly surveyed for the study.
- The population (N) consists of the wider collection of individuals or cases about which we want to generalize.
- In order to determine whether a sample is representative of a broader population, we measure statistical significance.
- In calculating significance we ask whether the result obtained in the sample is representative of the population, or due to chance.
- Inferential statistics such as Chi-Square and Anova address sampling error only, not errors in question construction or errors in the coding and weighting of the data.
- Although based upon different sampling distributions, Chi-Square and Anova use the same standards for interpretation.
- This lab will discuss using Chi-square; Lab 12 will take up Anova. Lab 13 addressing T-Tests will not be covered in class but you may work through it on your own.
- To determine whether a relationship in a cross tabulation is significant (not due to sampling error) use the Chi-square (Χ2) test.
- Χ2 works by comparing observed frequencies in a table with what we would expect to find if there is no association between the variables (as indicated by differences across the columns). The greater the difference between what we observe and what we expect (no association), the less likely it is due to sampling error.
- Cells should not have fewer than 5 cases.
- Χ2 indicates only whether an observed relationship may be due to sampling error. As such, Χ2 is not a measure of the strength of a relationship. Nor does it indicate the direction of a relationship.
- If the probability level is less than .05, we conventionally infer that the observed relationship may be generalized to the population.
- Statistical significance depends in part on sample size so as the number of cases (n) grows statistical significance is easier to achieve. As the sample size shrinks, significance is more difficult to find.
- A significant Chi-square tells us only that there is a significant difference in the table, not specifically where that difference is.
- As we will in the next lab, we can determine which specific categories on the independent variable have significantly different values on the dependent variable by using ANOVA .
CONVENTIONAL STANDARDS FOR STATISTICAL SIGNIFICANCE
|probability level||Description||Acceptability||GENERAL COMMENTS||Chi-square INTERPRETATION|
|.00 to .01||Highly Significant||GENERALLY ACCEPTABLE||The difference in the sample is very likely representative of the population, not due to chance.||The differences in the dependent variable across the categories of the independent variable are very unlikely to be due to chance.|
|.01 to .05||Significant||CONVENTIONALLY ACCEPTABLE||The difference in the sample is likely representative of the population||The differences in the dependent variable across the categories of the independent variable are unlikely to be due to chance.|
|.05 to 0.1||Marginally significant||Acceptable in some circumstances.||The difference may well be due to chance||The differences in the dependent variable across the categories of the independent variable may be due to chance.|
|.1 to 1.00||Non-significant||NOT ACCEPTABLE||The measurement in the sample is likely due to chance||The differences in the dependent variable across the categories of the independent variable are likely due to chance.|
- Independent Variable:
- Party ID
- Dependent Variable:
- Egalitarian Attitudes
- Hypothesis Arrow Diagram:
- Party ID → Economic Egalitarian Policy Preferences
STEP ONE: CHI-SQUARE TEST FOR SIGNIFICANCE
*Identifying EconEq Index Items*. weight by weight_full. missing values cses_govtact (-9 thru -6). recode cses_govtact (1=1) (2=.75) (3= .5) (4= .25) (5=0) into eceq1. missing values ineqinc_ineqreduc (-9 thru -6). recode ineqinc_ineqreduc (1=1) (2=0) (3= .5) into eceq3. missing values guarpr_self (-9 thru -2). recode guarpr_self (1=1) (2=.832) (3= .666) (4= .5) (5= .332) (6= .166) (7=0) into eceq5. *Constructing the Index*. compute RawIndex = eceq1 + eceq3 + eceq5. *Recoding the Index*. recode RawIndex (.00 thru 1.00 =1) (1.01 thru 1.85 =2) (1.86 thru 3 = 3) into IEcEq3. *Creating an IV*. missing values pid_self (-9 thru 0, 5). missing values pid_x (-2). recode pid_self (1=1) (3 = .5) (2=0) into pid. value labels pid 1 'Dem' .5 'Ind' 0 'Rep'. *Adding chisq to assess significance in a crosstab*. crosstabs tables = IEcEq3 by pid /cells = column count /statistics = phi btau chisq.
- Note that data have been weighted
- Syntax has been edited to exclude reliability and frequency analyses completed in previous labs
- The DV has been renamed IEcEq3 to indicate it is an index recoded into 3 values
- Crosstabulation is a necessary prelude to Chi-Square
- The chisq specification is added to the /statistics subcommand following the measure of association.
Crosstabulation of support for action toward Income Equality by Partisan Identity
IEcEq3 Partisan Identification Repub Ind Democ Support for
low 66.3% 36.7% 14.9% med 20.2% 34.3% 37.1% hi 13.5% 29.0% 48.0% N = (1403) (1629) (1746)
Cramer’s V =.311
Taub = .377
Pearson Chi-sq = 927.1 with 4 df, p = .000
- Glancing at the crosstab, we can see that at least some of the cell-percentages differ as we scan across the rows. This suggests that there may be a relationship between the two variables.
- The value of Chi-square (927.1) and degrees of freedom (4) are essential for calculating statistical significance but not for its interpretation.
- In interpreting the results the most important figure in the output is the significance (reported by SPSS as Sig 2-sided), which is .000. This means that there is a very slight chance (less than .0005%) that the relationship between PID and IEcEq3 in the population is due to sampling error.
- We can conclude that there is a statistically significant relationship between party identification and egalitarian attitudes.
- We do not know, however, which columns differ significantly (beyond what one would expect due to chance).
- We can push our analysis further using ANOVA to identify which specific differences are significant. This will be discussed in Lab 12.
- Begin by selecting one of the available datasets and hypothesize a relationship between two variables. Either variable can be measured at the nominal, ordinal, or interval level.
- Identify missing values and essential recodes using Frequency runs.
- Prepare a crosstab analysis a usual.
- Include chisq on the /statistics subcommand along with the appropriate measure of association.
- Determine whether you can infer that a relationship between the two variables exists in the population based on what you observe in the sample by referring to the Significance for the Pearson’s Chi-square. Use the same guidelines as in the previous labs.
- Repeat the steps above until you find a pair of variables that yield a significant relationship for the Chi-Square test.
QUESTIONS FOR REFLECTION
- Even though the relationship in the cross tabulation clearly may be significant, is it possible for the variables not to be strongly related?
- Since the level of significance level improves as we increase the sample size, why do surveys usually limit sample-size?
- The significance of Chi-square applies to the table as a whole, but do we know specific columns differ significantly from one another?
- What further analyses can we conduct which will enable us to do so?
- Statistical significance and strength (or predictability) of association are two different things. One can have a weak relationship that is statistically significant or a strong relationship that is not significant. Notice too that sample size affects Chi-square.
- After a certain sample-size, adding more cases does not much improve the significance level. At such a point, the marginal benefit of increasing the sample-size has to be considered against the cost associated with gathering more data. The tipping point is somewhere between 1500 and 2000 cases in most surveys.
- A Chi-square analysis does not tell us which columns differ significantly unless, of course, the independent variable has only two columns.
- To determine which specific columns in a multi-column crosstabulation differ from one another requires another approach.
- An inefficient approach would be to construct a series of two column tables.
- A more efficient way to proceed is to turn to Analysis of Variance, the topic of Lab 12.