UCSC Lab 8

The Reliability of an Index

PURPOSE

  • To determine whether a combination of related variables are suitable to construct an index.
  • To learn how to measure the reliability of an index.
  • To learn how to form a conceptual definition of a combination of related variables.

MAIN POINTS

  • Some concepts are complex and cannot be measured adequately by a single question or indicator.  Several indicators can often be combined to form an index.  Indices are generally useful in data analysis because they more effectively measure a concept than any single indicator.
  • To determine whether a combination of questionnaire items is suitable for an index, we use the Reliability procedure in SPSS. The resulting analysis enables us to determine the extent to which a number of indicators measure the same thing.
  • In past exercises, when the relationship between two variables was too strong (>.45) you were to discard the findings because the two variables probably just measured the same concept.  But such variables are perfectly suited for constructing an index.
  • Technically, Reliability analysis estimates the extent to which several indicators go together using Cronbach’s Alpha and other measures. The higher the alpha score, the more unified the movement of the indicators and hence the better suited they are to form an index.
  • The threshold value of Cronbach’s alpha for our work is approximately .60.

EXAMPLE

  • Dataset:
    • ANES 2012
  • Concept:
    • Attitude toward reducing economic inequality
  • Indicators:
  •  cses_govtact: Please say to what extent you agree or disagree with the following statement: ‘The government should take measures to reduce differences in income levels.’
  • ineq_incgap_x: Do you think the difference in incomes between rich people and poor people in the United States today is LARGER, SMALLER, or ABOUT THE SAME as it was 20 years ago?
  • ineqinc_ineqreduc: Do you favor, oppose, or neither favor nor oppose the government trying to make this income difference smaller?
  • ineqinc_ineqgb: The difference in incomes has grown in the past few decades…Is it good, bad, or neither good nor bad that the DIFFERENCE between the top and the bottom incomes has changed this way?
  • guarpr_self: Guaranteed jobs and living std placement–Where would you place YOURSELF.

Syntax

*Identifying EconEq Index Items*.

missing values cses_govtact (-9 thru -6).
fre var=cses_govtact
   /statistics stdev skew kurtosis.
recode cses_govtact (1=1) (2=.75) (3= .5) (4= .25) (5=0) into eceq1.
fre var eceq1.

missing values ineq_incgap_x (-9, -8).
freq var ineq_incgap_x.
recode ineq_incgap_x (1=1) (2= .75) (3= .5) (4= .25) (5=0) into eceq2.
fre var eceq2.

missing values ineqinc_ineqreduc (-9 thru -6).
fre var ineqic_ineqreduc.
recode ineqinc_ineqreduc (1=1) (2=0) (3= .5) into eceq3.
fre var eceq3.

missing values ineqinc_ineqgb (-9 thru -6).
freq var ineqinc_ineqreduc ineqinc_ineqgb.
recode ineqinc_ineqgb (1=0) (3= .5) (2=1) into eceq4.
fre var eceq4.

missing values guarpr_self (-9 thru -2).
recode guarpr_self (1=1) (2=.832) (3= .666) (4= .5) (5= .332) (6= .166) 
   (7=0) into eceq5.
fre var guarpr_self eceq5.

*Conducting Reliabiility Analysis*.
reliability
   /variables= eceq1 eceq2 eceq3 eceq4 eceq5
   /scale(EcEq1) all
   /summary = all.

reliability
   /variables= eceq1 eceq2 eceq3 eceq4 eceq5
   /scale (EcEq2) eceq1 eceq3 eceq4 eceq5
   /summary = all.

reliability
   /variables= eceq1 eceq2 eceq3 eceq4 eceq5
   /scale (EcEq3) eceq1 eceq3 eceq5
   /summary = all.
  • Syntax Legend
    • Missing values and Recodes are entered manually using syntax;
    • The Reliability procedure calculates various index statistics including Alpha. Indicators for possible inclusion in the analysis are listed immediately following the Reliability command. The (indented) /scale subcommand contains the indicators actually used in the analysis.  Additional statistics are available by specifying the (indented) subcommand /statistics=all.
    • The first Reliability command contains all the previously listed indicators. The second and third reliability commands use only a subset of the indicators.
    • The output presented below is shown in stages.Output from first Reliability command
Reliability Statistics
Cronbach’s Alpha Cronbach’s Alpha Based on Standardized Items N of Items
.653 .642 5

 

Item-Total Statistics
Indicators included in Analysis ScaleMean if Item
Deleted		 ScaleVariance
if Item
Deleted		 Corrected
Item-Total
Correlation		 Squared
Multiple
Correlation		 Cronbach’sAlpha if Item
Deleted
eceq1 2.5175 .768 .486 .356 .562
eceq2 2.1086 .973 .272 .090 .653
eceq3 2.4280 .594 .591 .370 .493
eceq4 2.2187 .862 .305 .172 .646
eceq5 2.4681 .844 .398 .252 .605
  • Output Legend
    • While the Alpha score in the first output table is perhaps most important measure reported, it is only one of many useful statistics included in the output. It is sometimes tempting to report the standardized alpha but this should be done only when the index will be computed using standardized indicators. See Lab 9 for details,
    • In the second output table, labeled Item-total statistics, the column farthest to the right shows the value of alpha if a particular item is deleted from the index. It is very useful in deciding which indicators, if any, to drop from your index, as well as which may be indispensable for the index. For example, if the eceq3 item were to be deleted, alpha would decrease to .493, suggesting that this item is essential for the index. However, if the eceq2 indicator is removed from the index, the alpha would remain the same, suggesting this item is unnecessary for the index
    • The ‘alpha if item deleted’ value is not the only way to decide which items to keep in the index. For example, the figures in the item-total correlation column suggest that, of the four items, eceq2 item is the least central of the four.

      Interpretation

    • Cronbach’s alpha is .65.  Since this is > .60, we may conclude that the questions are related enough to combine into an index.
    • Generally longer indexes have higher alpha scores, however this does not rule out the possibility that fewer items will yield an alpha as high or higher.  But this must be determined by trial-and-error.
    • The second and third Reliability analyses produced by the syntax shown above demonstrate that alpha can be increased to .687 by eliminating both eceq2 and eceq4.
    • No further improvements in alpha can be achieved by further eliminating of indicators.
    • It is often necessary to try different combinations of the questions in an iterative trial-and-error process to discover the combination that will yield the highest alpha value.
    • Nevertheless, there may be particular combinations of indicators that will provide a higher alpha value that you will not see if you look only at the ‘alpha if item deleted’ values.
    • Reliability Statistics
      Cronbach’s Alpha Cronbach’s Alpha Based on Standardized Items N of Items
      .653 .649 4
    • Item-Total Statistics
      Indicators included in Analysis ScaleMean if Item
      Deleted      		 ScaleVariance
      if Item
      Deleted      		 Corrected
      Item-Total
      Correlation      		 Squared
      Multiple
      Correlation      		 Cronbach’sAlpha if Item
      Deleted
      eceq1 1.6909 .599 .511 .356 .535
      eceq3 1.6014 .457 .582 .356 .466
      eceq4 1.3921 .711 .268 .151 .687
      eceq5 1.6415 .674 .408 .250 .605
    • Reliability Statistics
      Cronbach’s Alpha Cronbach’s Alpha Based on Standardized Items N of Items
      .687 .701 3
    • Item-Total Statistics
      Indicators included in Analysis ScaleMean if Item
      Deleted      		 ScaleVariance
      if Item
      Deleted      		 Corrected
      Item-Total
      Correlation      		 Squared
      Multiple
      Correlation      		 Cronbach’sAlpha if Item
      Deleted
      eceq1 .9744 .364 .591 .353 .490
      eceq3 .8849 .303 .485 .253 .651
      eceq5 .9250 .429 .468 .249 .643
  • Next Steps
    • Note that all of the items are scored so that a high score indicates greater concern over economic inequality. Hence, when combined to create an index, we might call it FixUneq. or OpposeInEq.
    • In the next Lab we use SPSS syntax to construct an index that functions much like any other variable.
    • INSTRUCTIONS
  • From a data set of interest, select indicators that appear to be measuring the same concept with which to form an index.
  1. From a data set of interest, select at least 3 questions that appear to be measuring the same concept with which to form an index.
  2. Before running a reliability analysis, you must take account of the missing variables and any recodes. It is generally advisable to recode questions so they have a comparable range of scores. So open a syntax window and enter the appropriate missing value and recode (or compute) commands.
  3. Now enter the syntax command for a Reliability analysis making sure to include the /scale and /summary commands. Use the syntax included in the sample above as an example.
  4. If the Cronbach’s Alpha value is below .60 after having taken account of the recodes and the missing values, then the questions you have selected may not be sufficiently related to one another to form an index. You should return to the list of questions to select another combination.
  5. In attempting to reach the.60 threshold, you can experiment by including or excluding items to try to attain a higher alpha. Use the ‘Alpha if Item Deleted’ column in the output to guide your work.
  6. Once you find an adequate combination of questions you should formulate a conceptual definition of the index that reflects the content of all the questions in the index.

QUESTIONS FOR REFLECTION

  • Sometimes an indicator may increase the alpha value of the index, but it does not quite reflect the desired concept.  Should it be retained or discarded?
  • How many indicators should one begin with in a reliability analysis?

DISCUSSION

  • The conceptual definition of the index depends on the selection of indicators that are included.  When the combination of questions in the index changes, then the index may reflect a different concept.
  • You should, of course, be prepared to jettison any indicator that on the face of it may seem to measure your concept but drastically reduces the alpha-value.
  • You should also consider dropping indicators that may not reflect your concept, even if they increase the alpha. If you include such an indicator, then you may have to reformulate the concept, which will affect the conclusions you may draw.
  • The indicators eceq2 and eceq4 do not relate directly to public policy preferences whereas the other indicators so so. This supports the statistical analysis suggesting an index composes only of eceq1, eceq3 & eceq5.
  • It is generally a good idea to begin a reliability analysis with more than three indicators in order to allow room for deletions on either technical or theoreti