Lab 8

POL242 LAB MANUAL:  Lab 8

The Reliability of an Index

PURPOSE

  • To determine whether a combination of related variables are suitable to construct an index.
  • To learn how to measure the reliability of an index.
  • To learn how to form a conceptual definition of a combination of related variables.

MAIN POINTS

  • Some concepts are complex and cannot be measured adequately by a single question or indicator.  Several indicators can often be combined to form an index.  Indices are generally useful in data analysis because they more effectively measure a concept than any single indicator.
  • To determine whether a combination of questionnaire items is suitable for an index, we use the Reliability procedure in SPSS. The resulting analysis enables us to determine the extent to which a number of indicators measure the same thing.
  • In past exercises, when the relationship between two variables was too strong (>.45) you were to discard the findings because the two variables probably just measured the same concept.  But such variables are perfectly suited for constructing an index.
  • Technically, Reliability analysis estimates the extent to which several indicators go together using Cronbach’s Alpha and other measures. The higher the alpha score, the more unified the movement of the indicators and hence the better suited they are to form an index.
  • The threshold value of Cronbach’s alpha for our work is approximately .60.

EXAMPLE

  • Dataset:
    • CES2011
  • Concept:
    • Attitude toward reducing inequality
  • Indicators:
  • PES11_41: How much should be done to reduce gap between the rich and the poor in Canada?
  • mbs11_k2 – the government SHOULD ACT/SHOULD NOT ACT to reduce differences in income and wealth;
  • mbs11_b3 – the govt should see to it that everyone has a decent standard of living/leave people to get ahead on their own.
  • pes11_52b – an NDP government would really hurt the Canadian economy.

Syntax

*Preparing 4 indicators of Attitudes re Inequality*

*declare missing values on pes11_41*.
missing values pes11_41 (8,9).
*reverse scoring on pes11_41 and make it range from 0-1*.
recode PES11_41 (1=1) (2=.75) (3=.5) (4= .25) (5=0) into undogap.
value labels undogap 0 'muchless' .25 'someless' .5 'asnow' .75 'somemore' 1 'muchmore'.

*rescale mbs11_k2 from 0-10 to 0-1 and reverse its scoring*.
missing values mbs11_k2 (-99).
compute govact = (((mbs11_k2 * -1) +10)/10).
value labels govact 0'not act' 1 'gov act'.

*recode and re-label mbs11_b3 and pes11_52b*.
recode mbs11_b3 (1=1) (2=0) into goveqch.
value labels goveqch 1 'decent living' 0 'leave alone'.
recode pes11_52b (1=0) (3= .33) (5= .66) (7= 1)into NDPnohurt.
value labels NDPnohurt 0 'strdisagree' .33 'disagree' .66 'agree' 1 'stragree'.

Reliability Variables = undogap govact goveqch ndpnohurt
     /scale (fixineq) = undogap govact goveqch ndpnohurt
     /summary = all.

 

  • Syntax Legend
    • Missing values and Recodes are entered manually using syntax;
    • The Reliabiltiy procedure calculates various index statistics including Alpha. Indicators for possible inclusion in the analysis are listed immediately following the Reliability command. The (indented) /scale subcommand contains the indicators actually used in the analysis. In this case it contains all the previously listed indicators. Additional (indented) /scale subcommands may be added to permit comparisons among several possible indexes composed of different subsets of indicators. See below for an example. Additional statistics are available by specifying the (indented) subcommand /statistics=all.
  • Output
Reliability Statistics
Cronbach’s Alpha Cronbach’s Alpha Based on Standardized Items N of Items
.692 .718 4

 

Item-Total Statistics
Indicators included in Analysis ScaleMean if Item
Deleted		 ScaleVariance
if Item
Deleted		 Corrected
Item-Total
Correlation		 Squared
Multiple
Correlation		 Cronbach’sAlpha if Item
Deleted
undogap 1.9572 .593 .525 .309 .618
govact 2.1459 .553 .557 .341 .591
goveqch 1.9072 .441 .486 .249 .634
NDPnohurt 2.1574 .503 .417 .177 .673

 

  • Output Legend
    • While the Alpha score in the first output table is perhaps most important, it is only one of many useful statistics included in the output. It is sometimes tempting to report the standardized alpha but this should be done only when the index will be computed using standardized indicators. See Lab 9 for details,
    • In the second output table labeled Item-total statistics, the column farthest to the right shows the value of alpha if a particular item is deleted from the index. It is very useful in deciding which indicators, if any, to drop from your index, as well as which may be indispensable for the index. For example, if the govact question were to be deleted, alpha would decrease to .591, suggesting that this item is essential for the index. However, if the NDPnohurt indicator is removed from the index, the alpha would drop only marginally from .69 to .67.
    • The ‘alpha if item deleted’ value is not the only way to decide which items to keep in the index. For example, the figures in the item-total correlation column suggest that, of the four items, NDPnohurt is the least central of the four.
    • It is often necessary to try different combinations of the questions in an iterative trial-and-error process to discover the combination that will yield the highest alpha value. There may be particular combinations that will provide a higher alpha value that you will not realize if you pay attention to only the ‘alpha if item deleted’ values.
  • Interpretation
    • Cronbach’s alpha is .69.  Since it is > .60, we may conclude that the questions are related enough to combine into an index.
    • Looking at the column showing the value of alpha if item deleted, we can see that removing the last item yields an only slightly lower alpha, whereas removing the first item yields a substantially lower alpha
    • Generally longer indexes have higher alpha scores, however this does not rule out the possibility that fewer items will yield an alpha as high or higher.  But this must be determined by trial-and-error.
    • All of the questions relate to attitudes toward inequality. Each question taps a particular mode or dimension of attention.
    • All of the items are scored so that a high score indicates greater attention to the election. Hence, when combined to create an index, we might call it FixUneq.
    • In the next exercise we use SPSS syntax to construct an index that functions much like any other variable.

INSTRUCTIONS

  1. From a data set of interest, select at least 3 questions that appear to be measuring the same concept with which to form an index.
  2. Before running a reliability analysis, you must take account of the missing variables and any recodes. It is generally advisable to recode questions so they have a comparable range of scores. So open a syntax window and enter the appropriate missing value and recode (or compute) commands.
  3. Now enter the syntax command for a Reliability analysis making sure to include the /scale and /summary commands. Use the syntax included in the sample above as an example.
  4. If the Cronbach’s Alphavalue is below .60 after having taken account of the recodes and the missing values, then the questions you have selected may not be sufficiently related to one another to form an index. You should return to the list of questions to select another combination.
  5. In attempting to reach the.60 threshold, you can experiment by including or excluding items to try to attain a higher alpha. Use the ‘Alpha if Item Deleted’ column in the output to guide your work.
  6. Once you find an adequate combination of questions you should formulate a conceptual definitionof the index that reflects the content of all the questions in the index.

 

QUESTIONS FOR REFLECTION

  • Sometimes an indicator may increase the alpha value of the index, but it does not quite reflect the desired concept.  Should it be included? Consider, for example, item # pes11_52b, “an NDP government would really hurt the Canadian economy.” Should it be retained or discarded?
  • How many indicators should one begin with in a reliability analysis?

DISCUSSION

  • The conceptual definition of the index depends on the selection of indicators that are included.  When the combination of questions in the index changes, then the index may reflect a different concept.
  • You should, of course, be prepared to jettison any indicator that on the face of it may seem to measure your concept but drastically reduces the alpha-value.
  • You should also consider dropping indicators that may not reflect your concept, even if they increase the alpha. If you include such an indicator, then you may have to reformulate the concept, which will affect the conclusions you may draw.
  • The indicator asking whether an NDP government would hurt the economy introduces partisan preferences into the index.
  • If partisanship is likely to be a relevant Independent variable, the item should perhaps be excluded.
  • This concern, along with those raised earlier about this indicator, suggests perhaps excluding this item.
  • It is generally a good idea to begin a reliability analysis with more than three indicators in order to allow room for deletions on either technical or theoreti