new Lab 8

The Reliability of an Index

PURPOSE

  • To determine whether a combination of related variables are suitable to construct an index.
  • To learn how to measure the reliability of an index.
  • To learn how to form a conceptual definition of a combination of related variables.

MAIN POINTS

  • Some concepts are complex and cannot be measured adequately by a single question or indicator.  Several indicators can often be combined to form an index.  Indices are generally useful in data analysis because they more effectively measure a concept than any single indicator.
  • To determine whether a combination of questionnaire items is suitable for an index, we use the Reliability procedure in SPSS. The resulting analysis enables us to determine the extent to which a number of indicators measure the same thing.
  • In past exercises, when the relationship between two variables was too strong (>.45) you were to discard the findings because the two variables probably just measured the same concept.  But such variables are perfectly suited for constructing an index.
  • Technically, Reliability analysis estimates the extent to which several indicators go together using Cronbach’s Alpha and other measures. The higher the alpha score, the more unified the movement of the indicators and hence the better suited they are to form an index.
  • The threshold value of Cronbach’s alpha for our work is approximately .60.

EXAMPLE

  • Dataset:

    • PPIC Oct 2016
  • Concept:
    • Attitude toward recreation use of marijuana
  • Indicators:
    •  Q21. “Proposition 64 is called the ‘Marijuana Legalization. Initiative Statute’ … If the election were held today, would you vote yes or no on Proposition 64?”
    • Q22 “How important to you is the outcome of the vote on Proposition 64—is it very important, somewhat important, not too important, or not at all important?”
    • Q36 “Next, in general, do you think the use of marijuana should be legal, or not?”
    • Q36a “Keeping in mind that all of your answers in the survey are confidential, have you ever tried marijuana? (IF YES, ASK: have you used marijuana in the last 12 months?)”
  • Syntax
*Identifying RecrMJ Index Items*.

recode q21 (1=1) (2=0) into MJPropD.
value labels MJPropD 1 'yes' 0 'no'.
fre var = MJPropD.

recode q22 (1=1) (2=.66) (3=.33) (4=0) into MJImp.
value labels MJImp 1 'very' .66 'somewhat'
  .33 'not too' 0 'notatall'.
fre var = MJIMP.

recode q36 (1=1) (2=0) into MJLegalD.
value labels MJLegalD 1 'yes' 0 'no'.
fre var = MJLegalD.

recode q36a (1=1) (2=.5) (3=.0) into MJTry.
value labels MJTry 1 'recent' .5 'not recent'
  0 'no'.
fre var = MJTry.
*Conducting Reliability Analysis*.
reliability /variables=MJPropD MJImp MJLegalD MJTry
  /scale('RecMJ4') all
  /statistics=descriptive
  /summary=total.

reliability /variables=MJPropD MJImp MJLegalD MJTry
  /scale('RecMJ3') MJPropD MJLegalD MJTry
  /statistics=descriptive
  /summary=total.

reliability /variables=MJPropD MJImp MJLegalD MJTry
  /scale('RecMJ2') MJPropD MJLegalD
  /statistics=descriptive
  /summary=total.

Syntax Legend

  • Missing values and Recodes are entered manually using syntax, with a frequency checking each;
  • The Reliability procedure calculates various index statistics including Alpha. Indicators for possible inclusion in the analysis are listed immediately following the Reliability command. The (indented) /scale subcommand contains the indicators actually used in the analysis.  Additional statistics are available by specifying the (indented) subcommand /statistics=all.
  • The first Reliability command contains all the previously listed indicators. The second and third reliability commands use only a subset of the indicators.
  • The output presented below is shown in stages.

    Output from first Reliability command

Reliability Statistics

Cronbach’s Alpha

N of Items

.646

4

Item-Total Statistics

Scale Mean if Item Deleted

Scale Variance if Item Deleted

Corrected Item-Total Correlation

Cronbach’s Alpha if Item Deleted

MJPropD

1.6517

.606

.686

.346

MJImp

1.4386

1.305

-.008

.777

MJLegalD

1.6484

.605

.689

.343

MJTry

1.8734

.967

.392

.602

Output Legend

  • While the Alpha score in the first output table is perhaps most important measure reported, it is only one of many useful statistics included in the output.
  • In the second output table, labeled Item-total statistics, the column farthest to the right shows the value of alpha if a particular item is deleted from the index. It is very useful in deciding which indicators, if any, to drop from your index, as well as which may be indispensable for the index. For example, if the MJPropD item were to be deleted, alpha would decrease to .346, suggesting that this item is essential for the index. However, if the MJImp indicator is removed from the index, the alpha would remain the same, suggesting this item is unnecessary for the index
  • The ‘alpha if item deleted’ value is not the only way to decide which items to keep in the index. For example, the figures in the item-total correlation column suggest that, of the four items, MJImp item is the least central of the four. In fact, it is uncorrelated with the total..

Interpretation

    • Cronbach’s alpha is rounded to .65.  Since this is > .60, we may conclude that the questions are related enough to combine into an index.
    • Generally longer indexes have higher alpha scores, however this does not rule out the possibility that fewer items will yield an alpha as high or higher.  But this must be determined by trial-and-error.
    • The second and third Reliability analyses produced by the syntax shown above demonstrate that alpha can be increased to .777 by eliminating MJImp and perhaps further increased by eliminating MJTry. Although dropping MJTry from the index will improve the alpha score, it is likely preferable to have three indicators in the index. Such choices should also be based on theoretical considerations.
    • No additional improvements in alpha can be achieved by further eliminating of indicators.
    • It is often necessary to try different combinations of the questions in an iterative trial-and-error process to discover the combination that will yield the highest alpha value.
    • Nevertheless, there may be particular combinations of indicators that will provide a higher alpha value that you will not see if you look only at the ‘alpha if item deleted’ values.
  • Second Run Output

    • Reliability Statistics

      Cronbach’s Alpha

      N of Items

      .777

      3

    • Item-Total Statistics

      Scale Mean if Item Deleted

      Scale Variance if Item Deleted

      Corrected Item-Total Correlation

      Cronbach’s Alpha if Item Deleted

      MJPropD

      .8863

      .521

      .748

      .535

      MJLegalD

      .8830

      .505

      .784

      .486

      MJTry

      1.1080

      .909

      .374

      .912
  • Third Run Output

    • Reliability Statistics

      Cronbach’s Alpha

      N of Items

      .912

      2

    • Item-Total Statistics

      Scale Mean if Item Deleted

      Scale Variance if Item Deleted

      Corrected Item-Total Correlation

      Cronbach’s Alpha if Item Deleted

      MJPropD

      .5556

      .247

      .838

      .

      MJLegalD

      .5524

      .247

      .838

      .
  • Next Steps
    • Note that all of the items are scored so that a high score indicates support for the use of recreational marijuana. Hence, when combined to create an index, we might call it Support Recreational MJ or more simply RecMJ
    • In the next Lab we use SPSS syntax to construct an index that functions much like any other variable.
  • INSTRUCTIONS
  1. From a data set of interest, select at least 3 questions that appear to be measuring the same concept with which to form an index.
  2. Before running a reliability analysis, you must take account of the missing variables and any recodes. It is generally advisable to recode questions so they have a comparable range of scores. So open a syntax window and enter the appropriate missing value and recode (or compute) commands.
  3. Now enter the syntax command for a Reliability analysis making sure to include the /scale and /summary commands. Use the syntax included in the sample above as an example.
  4. If the Cronbach’s Alpha value is below .60 after having taken account of the recodes and the missing values, then the questions you have selected may not be sufficiently related to one another to form an index. You should return to the list of questions to select another combination.
  5. In attempting to reach the.60 threshold, you can experiment by including or excluding items to try to attain a higher alpha. Use the ‘Alpha if Item Deleted’ column in the output to guide your work.
  6. Once you find an adequate combination of questions you should formulate a conceptual definition of the index that reflects the content of all the questions in the index.

QUESTIONS FOR REFLECTION

  • Sometimes an indicator may increase the alpha value of the index, but it does not quite reflect the desired concept.  Should it be retained or discarded?
  • How many indicators should one begin with in a reliability analysis?

DISCUSSION

  • The conceptual definition of the index depends on the selection of indicators that are included.  When the combination of questions in the index changes, then the index may reflect a different concept.
  • You should, of course, be prepared to jettison any indicator that on the face of it may seem to measure your concept but drastically reduces the alpha-value.
  • You should also consider dropping indicators that may not reflect your concept, even if they increase the alpha. If you include such an indicator, then you may have to reformulate the concept, which will affect the conclusions you may draw.
  • It is generally a good idea to begin a reliability analysis with more than three indicators in order to allow room for deletions on either technical or theoretical.

Advanced Exercises

1. Combining q21 and q22 in the PPIC October 2016 data.

Use the following syntax to keep q22 in the analysis.

if (q21 =1) and (q22 =1) StrMJ = 1.
if (q21 =1) and (q22 =2) StrMJ = .86.
if (q21 =1) and (q22 =3) StrMJ = .71.
if (q21 =1) and (q22 =4) StrMJ = .57.
if (q21 =2) and (q22 =4) StrMJ = .43.
if (q21 =2) and (q22 =3) StrMJ = .29.
if (q21 =2) and (q22 =2) StrMJ = .14.
if (q21 =2) and (q22 =1) StrMJ = 0.

2. Build an attitude toward inequality measure using ANES 2016 data.

The following items were identified using the ANES variable list pdf & codebook as possible indicators of attitudes toward inequality. They are: V161137, V161138x (combining V161138a, V161138b), V161189 and V162148.

As an exercise see whether they form an index. A lab constructing a similar index using the ANES 2012 is available here. LINK

V161137

Label: PRE: Income gap today more or less than 20 years ago

Item name: INEQ_INCGAP

Question: Do you think the difference in incomes between rich people and poor people

in the United States today is larger, smaller, or about the same as it was 20

years ago?

  1. Larger
  2. Smaller
  3. About the same

-8. Don’t know

-9. Refused

V161138a

Label: PRE: How much larger is income gap today

Item name: INEQ_GAPMORE

Question: IF INCOME GAP TODAY IS LARGER THAN IT WAS 20 YEARS AGO:

(Would you say the difference in incomes is) much larger or somewhat larger?

For Web administration the parentheses indicating optional text were omitted.

V161138b

Label: PRE: How much smaller is income gap today

Item name: INEQ_GAPLESS

Question: IF INCOME GAP TODAY IS SMALLER THAN IT WAS 20 YEARS AGO:

(Would you say the difference in incomes is) much smaller or somewhat

smaller? For Web administration the parentheses indicating optional text

were omitted.

V161138x

Label: PRE: SUMMARY – larger/smaller income gap today

Item name: Not applicable; administrative or derived variable

Question: Not applicable

  1. Much larger
  2. Somewhat larger
  3. About the same
  4. Somewhat smaller
  5. Much smaller

-1. Inapplicable

V161189

Label: PRE: 7pt scale guaranteed job-income scale: self-placement

Item name: GUARPR_GUARSELF

Question: Where would you place yourself on this scale, or haven’t you thought much

about this? For Web administration, DO NOT PROBE was omitted for the

option ‘Haven’t thought much about this’.

  1. Govt should see to jobs and standard of living

2

3

4

5

6

  1. Govt should let each person get ahead on own
  2. Haven’t thought much about this

-8. Don’t know

-9. Refused

V162148

Label: POST: Does R favor or oppose govt reducing income ineqality

Item name: INEQINC_INEQRED

Question: Next, do you favor, oppose, or neither favor nor oppose the government

trying to reduce the difference in incomes between the richest and poorest

households?

  1. Favor
  2. Oppose
  3. Neither favor nor oppose

-6. No post-election interview

-7. No post data, incomplete

-8. Don’t know

-9. Refused