Систематическая ошибка отбора

4.2.7.2 Systematic Error

It is a mistake to believe that systematic errors produce a constant bias or offset between the true value and the measured value. Bias can be inconsistent, especially when there are multiple sources of it.

2.2 Conjunctive Probabilities

Systematic errors in subjective probabilities of conjunctive events have also been identified by behavioral decision researchers (Tversky and Kahneman 1983). In another story, Tversky and Kahneman told participants about a woman named Linda who was described as 31 years old, single, outspoken, and very bright. She majored in philosophy and cared deeply about issues of discrimination and social justice. She also participated in anti-nuclear demonstrations. Then participants were asked to rank the likelihood of various statements, including ‘Linda is a bank teller’ and ‘Linda is a bank teller and a feminist.’ Participants report that the statement, ‘Linda is a bank teller and a feminist’ is more probable than ‘Linda is a bank teller.’ Tversky and Kahneman argued that these responses were violations of the conjunction rule, according to which the judged probability of the intersection of two events cannot exceed the judged probability of either single event. They claimed that people base their beliefs on the similarity of the target description to the category prototype, a strategy known as representativeness.

Gigerenzer and his collaborators have challenged this claim and have further explored the use of frequency representations to reduce, and even eliminate, conjunction effects. (Gigerenzer and Hoffrage 1995). They have had remarkable success at reducing base rate neglect and conjunction errors with frequency formats. This topic is yet another area of considerable controversy within the field.

Bias in psychological assessment

Bias refers to systematic error in the estimation of a value. For researchers, test bias is a deviation from examinees real level of performance. Bias goes by many names and has many characteristics but it always involves scores that are too low or too high to represent or predict an individual’s characteristics. Estimates of scores are required to reveal bias. Types of test bias include social desirability bias (self-enhancement and impression management; Paulhus & Trapnell, 2008), acquiescence, and cultural bias. A newly discovered type of bias is examining process bias (Baldini, Parker, Nelson, & Siegel, 2014).

Discussion

We analysed the systematic errors that healthy human subjects make when estimating the direction of gravity by setting a visual line (SVV) during constant ‘quasi-static’ velocity whole-body rotations about the naso-occipital axis.¹ The majority of subjects showed an instability zone around the upside-down position between 90° left-ear and 90° right-ear down where rapid switches between tilt underestimation (A-effect) and tilt overestimation (E-effect) occurred. When comparing SVV deviations measured during CCW and CW turntable rotations, we found that at identical whole-body positions the A-effect was larger when passing through upside-down than by ‘direct’ rotation from upright (peaks in the side positions). Because rotations were performed ‘quasi-statically,’ these results demonstrate that there is hysteresis for SVV estimation that is independent of roll velocity. Moreover, this static SVV hysteresis is already present at roll angles below 90°, where the A-effect dominates. We speculate that the hysteresis found by Kaptein and Van Giesbergen (2004, 2005) for the switching between A- and E-effects around 135° is a direct result of static hysteresis for the A-effect. In the instability zone, the direction-dependent magnitude of the A-effect may influence the roll position for switching between A- and E-effects.

Bias

Bias is the result of systematic error in the design and conduct of the study, such that the observed results in the sample will be different from the true results. Bias occurs due to flaws in the method of selection of study participants or in the process of gathering information regarding exposure and disease. This systematic error is different from random error due to sampling variability, which results from the use of a sample to estimate parameters for the reference population. We will discuss two broad categories of bias: selection bias and information bias.

Bias, or Systematic Error

Turning to bias, or systematic error, there is also a sampling component. First, the sample frame (i.e., the list or enumeration of elements in the population) may either omit or double count units. For example, the U.S. Census both misses people (especially African-Americans and immigrants) and counts others twice (especially people with more than one residence), and samples based on the census reflect these limitations. Second, certain housing units, such as new dwellings, secondary units (e.g., basement apartments in what appears to be a single-family dwelling), and remote dwellings, tend to be missed in the field. Likewise, within housing units, certain individuals, such as boarders, tend to be underrepresented and some respondent selection methods fail to work in an unbiased manner (e.g., the last/next birthday method overrepresents those who answer the sample-screening questions). Third, various statistical sampling errors occur. Routinely, the power of samples is overestimated because design effects are not taken into consideration. Also, systematic sampling can turn out to be correlated with various attributes of the target population. For example, in one study, both the experimental form and respondent selection were linked by systematic sampling in such a way that older household members were disproportionately assigned to one experimental version of the questionnaire, thus failing to randomize respondents to both experimental forms.

Nonsampling error comes from both nonobservational and observational errors. The first type of nonobservational error is coverage error, in which a distinct segment of the target population is not included in sample. For example, in the United States, preelection random-digit-dialing (RDD) polls want to generalize to the voting population, but systematically exclude all voters not living in households with telephones. Likewise, samples of businesses often underrepresent smaller firms. The second type of nonobservational error consists of nonresponse (units are included in the sample, but are not successfully interviewed). Nonresponse has three main causes: refusal to participate, failure to contact because people are away from home (e.g., working or on vacation), and all other reasons (such as illness and mental and/or physical handicaps).

Observational error includes collection, processing, and analysis errors. As with variable error, collection error is related to mode, instrument, interviewer, and respondent. Mode affects population coverage. Underrepresentation of the deaf and poor occurs in telephone surveys, and of the blind and illiterate, in mail surveys. Mode also affects the volume and quality of information gathered. Open-ended questions get shorter, less complete answers on telephone surveys, compared to in-person interviews. Bias also is associated with the instrument. Content, or the range of information covered, obviously determines what is collected. One example of content error is when questions presenting only one side of an issue are included, such as is commonly done in what is known as advocacy polling. A second example is specification error, in which one or more essential variable is omitted so that models cannot be adequately constructed and are therefore misspecified.

Various problematic aspects of question wordings can distort questions. These include questions that are too long and complex, are double-barreled, include double negatives, use loaded terms, and contain words that are not widely understood. For example, the following item on the Holocaust is both complex and uses a double negative: “As you know, the term ‘holocaust’ usually refers to the killing of millions of Jews in Nazi death camps during World War II. Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” After being presented with this statement in a national U.S. RDD poll in 1992, 22% of respondents said it was possible that the Holocaust never happened, 65% said that it was impossible that it never happened, and 12% were unsure. Subsequent research, however, demonstrated that many people had been confused by the wording and that Holocaust doubters were actually about 2% of the population, not 22%. Error from question wording also occurs when terms are not understood in a consistent manner.

The response scales offered also create problems. Some formats, such as magnitude measurement scaling, are difficult to follow, leaving many, especially the least educated, unable to express an opinion. Even widely used and simple scales can cause error. The 10-point scalometer has no clear midpoint and many people wrongly select point 5 on the 1–10 scale in a failed attempt to place themselves in the middle. Context, or the order of items in a survey, also influences responses in a number of quite different ways. Prior questions may activate certain topics and make them more accessible (and thus more influential) when later questions are asked. Or they may create a contrast effect under which the prior content is excluded from later consideration under a nonrepetition rule. A norm of evenhandedness may be created that makes people answer later questions in a manner consistent with earlier questions. For example, during the Cold War, Americans, after being asked if American reporters should be allowed to report the news in Russia, were much more likely to say that Russian reporters should be allowed to cover stories in the United States, compared to when the questions about Russian reporters were asked first. Even survey introductions can influence the data quality of the subsequent questions.

Although social science scholars hope that interviewers merely collect information, in actuality, interviewers also affect what information is reported. First, the mere presence of an interviewer usually magnifies social desirability effects, so that there is more underreporting of sensitive behaviors to interviewers than when self- completion is used. Second, basic characteristics of interviewers influence responses. For example, Whites express more support for racial equality and integration when interviewed by Blacks than when interviewed by Whites. Third, interviewers may have points of view that they convey to respondents, leading interviewers to interpret responses, especially to open-ended questions, in light of their beliefs.

Much collection error originates from respondents. Some problems are cognitive. Even given the best of intentions, people are fallible sources. Reports of past behaviors may be distorted due to forgetting the incidents or misdating them. Minor events will often be forgotten, and major events will frequently be recalled as occurring more recently than was actually the case. Of course, respondents do not always have the best of intentions. People tend to underreport behaviors that reflect badly on themselves (e.g., drug use and criminal records) and to overreport positive behaviors (e.g., voting and giving to charities).

Systematic error occurs during the processing of data. One source of error relates to the different ways in which data may be coded. A study of social change in Detroit initially found large changes in respondents’ answers to the same open-ended question asked and coded several decades apart. However, when the original open-ended responses from the earlier survey were recoded by the same coders who coded the latter survey, the differences virtually disappeared, indicating that the change had been in coding protocols and execution, not in the attitudes of Detroiters. Although data-entry errors are more often random, they can seriously bias results. For example, at one point in time, no residents of Hartford, Connecticut were being called for jury duty; it was discovered that the new database of residents had been formatted such that the “d” in “Hartford” fell in a field indicating that the listee was dead. Errors can also occur when data are transferred. Examples include incorrect recoding, misnamed variables, and misspecified data field locations. Sometimes loss can occur without any error being introduced. For example, 20 vocabulary items were asked on a Gallup survey in the 1950s and a summary scale was created. The summary scale data still survive, but the 20 individual variables have been lost. Later surveys included 10 of the vocabulary items, but they cannot be compared to the 20-item summary scale.

Wrong or incomplete documentation can lead to error. For example, documentation on the 1967 Political Participation Study (PPS) indicated that one of the group memberships asked about was “church-affiliated groups.” Therefore, when the group membership battery was later used in the General Social Surveys (GSSs), religious groups were one of the 16 groups presented to respondents. However, it was later discovered that church-affiliated groups had not been explicitly asked about on the earlier survey, but that the designation had been pulled out of an “other-specify” item. Because the GSS explicitly asked about religious groups, it got many more mentions than had appeared in the PPS; this was merely an artifact of different data collection procedures that resulted from unclear documentation.

Most discussions of total survey error stop at the data-processing stage. But data do not speak for themselves. Data “speak” when they are analyzed, and the analysis is reported by researchers. Considerable error is often introduced at this final stage. Models may be misspecified, not only by leaving crucial variables out of the survey, but also by omitting such variables from the analysis, even when they are collected. All sorts of statistical and computational errors occur during analysis. For example, in one analysis of a model explaining levels of gun violence, a 1 percentage point increase from a base incidence level of about 1% was misdescribed as a 1% increase, rather than as a 100% increase. Even when a quantitative analysis is done impeccably, distortion can occur in the write-up. Common problems include the use of jargon, unclear writing, the overemphasis and exaggeration of results, inaccurate descriptions, and incomplete documentation. Although each of the many sources of total survey error can be discussed individually, they constantly interact with one another in complex ways. For example, poorly trained interviewers are more likely to make mistakes with complex questionnaires, the race of the interviewer can interact with the race of respondents to create response effects, long, burdensome questionnaires are more likely to create fatigue among elderly respondents, and response scales using full rankings are harder to do over the phone than in person. In fact, no stage of a survey is really separate from the other stages, and most survey error results from, or is shaped by, interactions between the various components of a survey.

Assessing measurement invariance

Bias is a technical term that addresses systematic errors that lead to differential interpretation of scores. In order to evaluate bias, we must determine whether knowledge of an examinee’s group membership influences the examinee’s score on the measured variance (e.g., an item, subdomain, or test), given the examinee’s status on the latent variance of interest (Millsap, 2011). Consequently, for a test to be fair (from a psychometric perspective) it should exhibit measurement invariance across all distinctive subgroups being evaluated. The degree to which the construct measured by a test is consistent across subgroups is known as construct equivalence (CE). CE is of special concern in cross-cultural research whereby constructs, such as intelligence and morality can be culturally affected (Van de Vijver & Poortinga, 2005). It is also associated with test adaptations or computer-based testing whereby the conditions of test administration are altered. The degree to which such alterations affect the construct is unknown.

CE can be evaluated statistically by Confirmatory Factor Analysis (CFA) and weighted multidimensional scaling (MDS), as they are able to analyze the structure of data from multiple groups simultaneously. CFA evaluated the hypothesized test structure, whereas MDS is an exploratory analysis that fits dimensions to best account for the data in all groups. In CFA the degree to which the hypothetical structure adequately fits the data for multiple groups can be analyzed using descriptive statistics, such as root mean square error of approximation, standardized root mean square residual, and adjusted goodness-of-fit statistic.

There are numerous statistical approaches for assessing measurement in variance. These methods can be classified into three groups: (1) linear measurements models, (2) nonlinear measurements models, and (3) observed score methods (Millsap, 2011). These approaches can be broken down into methods that estimate invariance at the scale and item levels (Zumbo, 2003). Scale-level analyses focus on the degree of invariance observed within common factor analytic models across groups. In contrast, item-level analyses and differential item functioning (DIF) examine invariance separately for each item. DIF examines the situation in which examiners who have equal standing on the target construct but those who come from different groups (e.g., ethnicity, age) have different probabilities of responding to the item (Holland & Thayer, 1988). “DIF represents a statistical interaction between group membership and item performance after matching examinees across groups on some criterion (usually total test score)” (Sireci & Sukin, 2013).

3 Heuristics

The study of heuristics tends to focus on systematic errors in human decision making and these heuristics often help to understand anomalies of inferring expectations from evidence (see Heuristics in Social Cognition).

Three heuristics that deal with probabilistic thinking have received considerable attention: (a) availability, (b) representativeness, and (c) anchoring and adjustment. The availability heuristic refers to the tendency to assess the probability of an event based on the ease with which instances of that event come to mind. This heuristic has been investigated in a variety of domains and relates probability estimates to memory access. Generally people overestimate the probability of an event if concrete instances of that event are easily accessible in memory. Generally, ease of recall and frequency of occurrence are correlated. A number of factors that affect memory are, however, unrelated to probability. For example, vivid images are easier to recall than pallid ones. Thus, having been involved in a serious car accident is likely to be better remembered than annual statistics about the frequency of (types of) traffic accidents. The former is likely to have more impact on probability estimates than the latter. Dawes (1994) argued that the salience of negative and relatively extreme exemplars of drug addicts can bias policy-makers’ perceptions of the entire group and result in negative attitudes toward programs such as the provision of clean needles to prevent a further spread of the AIDS virus.

The representativeness heuristic refers to the tendency to assess the probability that a stimulus belongs to a particular class by judging the degree to which that event corresponds to an appropriate mental model. Kahneman and Tversky (1973) reported a well-known example of how ignoring prior probabilities can affect judgment. In their study, respondents were provided with brief personality sketches, supposedly of engineers and lawyers. They were asked to assess the probability that each sketch described a member of one profession or the other. Half the respondents were told the population from which the sketches were drawn consisted of 30 engineers and 70 lawyers, the remaining respondents were told that there were 70 engineers and 30 lawyers. Findings showed that the prior probabilities were essentially ignored, and that respondents estimated the probability of class membership by judging how similar each personality sketch was to their mental model of an engineer or a lawyer.

Anchoring and adjustment refers to a general judgment process in which an initially given or generated response serves as an anchor, and other information is insufficiently used to adjust that response. The anchoring and adjustment heuristic is based on the assumption that people often start their judgmental process by focusing on some initial value that serves as an anchor. The biases related to this heuristic stem from two distinct aspects. First, one could use irrelevant anchors, second one could insufficiently adjust up or down from an original starting value or anchor.

‘Experimenter artifacts’ and ‘subject artifacts’ refer to systematic errors that can be attributed to uncontrolled aspects of the interaction between the researcher and the research participants or to other potentially bias-prone aspects of a research situation, such as the analysis, interpretation, and reporting of the research results. Particular experimenter artifacts that have been studied include observer, interpreter, intentional, biosocial, psychosocial, situational, modeling, and experimenter-expectancy effects. Subject artifacts that have been primarily investigated are generally concerned with the motivation of the research participants and their sensitivity and compliance with task-orienting cues. Discussed in this article are the history of artifacts in the social and behavioral sciences, the specific nature and control of experimenter and subject artifacts, and the delicate balance between ethical accountability, and the avoidance of artifacts in research with human participants.

10.03.6 The Problem of Definition

The definition of test bias has produced considerable continuing debate among measurement and assessment experts (Angoff, 1976; Bass, 1976; Bernal, 1975; Bond, 1981; Cleary et al., 1975; Cole & Moss, 1989; Cronbach, 1976; Dana, 1993; Darlington, 1978; Einhorn & Bass, 1970; Flaugher, 1978; Gordon, 1984; Gross & Su, 1975; Helms, 1992; Humphreys, 1973; Hunter & Schmidt, 1976, 1978; Jackson, 1980; Linn, 1976; McNemar, 1975; Moreland, 1996; Novick & Petersen, 1976; Padilla, 1988; Petersen & Novick, 1976; Reschly, 1980; Reynolds, 1978; 1982b, 1995; Reynolds & Brown, 1984; Sawyer, Cole, & Cole, 1976; Schmidt & Hunter, 1974; Thorndike, 1971). Although the resulting debate has generated a number of selection models with which to examine bias, selection models focus on the decision-making system and not on the test itself. The various selection models are discussed at some length in Schmidt and Hunter (1974), Hunter, Schmidt, and Rauschenberger (1984), Jensen (1980), Petersen and Novick (1976), and Ramsey (1979). The choice of a decision-making system (especially a system for educational decision-making) must ultimately be a societal one; as such, it will depend to a large extent on the value system and goals of the society. Thus, before a model for test use in selection (whether ultimately selection is for a treatment program, a job, a college, etc.) can be chosen, it must be decided whether the ultimate goal of selection is equality of opportunity, equality of outcome, or representative equality (these concepts are discussed in more detail in Nichols, 1978).

“Equality of opportunity” is a competitive model wherein selection is based on ability. As more eloquently stated by Lewontin (1970), under equality of opportunity, “true merit … will be the criterion of men’s earthly reward” (p. 92). “Equality of outcome” is a selection model based on ability deficits. Schooling provides a good model to illustrate these concepts that are also applicable to mental health. Compensatory and remedial programs are typically constructed on the basis of the equality-of-outcome model. Children of low ability or children believed to be a high risk for academic failure are selected for remedial, compensatory, or other special educational programs. Adults vying for jobs may be placed in specialized job training programs. In a strictly predictive sense, tests are used in a similar manner under both of these models. However, under equality of opportunity, selection is based on the prediction of a high level of criterion performance; under equality of outcome, selection is determined by the prediction of “failure” or a preselected low level of criterion performance. Interestingly, it is the failure of compensatory and remedial education programs to bring the disadvantaged learner to “average” levels of performance that resulted in the charges of test bias now in vogue.

The model of “representative equality” also relies on selection, but selection that is proportionate to numerical representation of subgroups in the population under consideration. Representative equality is typically thought to be independent of the level of ability within each group; however, models can be constructed that select from each subgroup the desired proportion of individuals (i) according to relative ability level of the group, (ii) independent of group ability, or (iii) according to some decision rule between these two positions. Even under the conditions of representative equality, it is imperative to employ a selection device (test) that will rank order individuals within groups in a reliable and valid manner. The best way to ensure fair selection under any of these models is to employ tests whose scores are equally reliable and equally valid for all groups concerned. The tests employed should also yield the most reliable and most valid scores for all groups under consideration. The question of test bias per se then becomes a question of test validity. Test use (i.e., fairness) may be defined as biased or nonbiased only by the societal value system; at present, this value system within the USA is leaning strongly toward some variant of the representative-equality selection model. In other sociopolitical structures, other models may be more appropriate. As noted above, all models are facilitated by the use of a nonbiased test. That is, the use of a test with equivalent cross-group validities makes for the most parsimonious selection model, greatly simplifying the creation and application of the selection model that has been chosen.

There are three types of validity as traditionally conceived: content, construct, and predictive (or criterion-related). Test bias may exist under any or all of these categories of validity. Though no category of validity is completely independent of any other category, each is discussed separately here for the purposes of clarity and convenience. (All true evidence of validity is as likely as not to be construct validity, and other, more detailed divisions including this one are for convenience of discussion.) Frequently encountered in bias research are the terms “single-group validity” and “differential validity.” Single-group validity refers to the phenomenon of a score interpretation being valid for one group but not another. Differential validity refers to a condition where an interpretation is valid for all groups concerned, but the degree of validity varies as a function of group membership. Although these terms have been most often applied to predictive or criterion-related validity (validity coefficients are then examined for significance and compared across groups), the concepts of single-group and differential validity are equally applicable to content and construct validity.

4.2.7.2 Systematic Error

It is a mistake to believe that systematic errors produce a constant bias or offset between the true value and the measured value. Bias can be inconsistent, especially when there are multiple sources of it.

2.2 Conjunctive Probabilities

Systematic errors in subjective probabilities of conjunctive events have also been identified by behavioral decision researchers (Tversky and Kahneman 1983). In another story, Tversky and Kahneman told participants about a woman named Linda who was described as 31 years old, single, outspoken, and very bright. She majored in philosophy and cared deeply about issues of discrimination and social justice. She also participated in anti-nuclear demonstrations. Then participants were asked to rank the likelihood of various statements, including ‘Linda is a bank teller’ and ‘Linda is a bank teller and a feminist.’ Participants report that the statement, ‘Linda is a bank teller and a feminist’ is more probable than ‘Linda is a bank teller.’ Tversky and Kahneman argued that these responses were violations of the conjunction rule, according to which the judged probability of the intersection of two events cannot exceed the judged probability of either single event. They claimed that people base their beliefs on the similarity of the target description to the category prototype, a strategy known as representativeness.

Gigerenzer and his collaborators have challenged this claim and have further explored the use of frequency representations to reduce, and even eliminate, conjunction effects. (Gigerenzer and Hoffrage 1995). They have had remarkable success at reducing base rate neglect and conjunction errors with frequency formats. This topic is yet another area of considerable controversy within the field.

Bias in psychological assessment

Bias refers to systematic error in the estimation of a value. For researchers, test bias is a deviation from examinees real level of performance. Bias goes by many names and has many characteristics but it always involves scores that are too low or too high to represent or predict an individual’s characteristics. Estimates of scores are required to reveal bias. Types of test bias include social desirability bias (self-enhancement and impression management; Paulhus & Trapnell, 2008), acquiescence, and cultural bias. A newly discovered type of bias is examining process bias (Baldini, Parker, Nelson, & Siegel, 2014).

Discussion

We analysed the systematic errors that healthy human subjects make when estimating the direction of gravity by setting a visual line (SVV) during constant ‘quasi-static’ velocity whole-body rotations about the naso-occipital axis.¹ The majority of subjects showed an instability zone around the upside-down position between 90° left-ear and 90° right-ear down where rapid switches between tilt underestimation (A-effect) and tilt overestimation (E-effect) occurred. When comparing SVV deviations measured during CCW and CW turntable rotations, we found that at identical whole-body positions the A-effect was larger when passing through upside-down than by ‘direct’ rotation from upright (peaks in the side positions). Because rotations were performed ‘quasi-statically,’ these results demonstrate that there is hysteresis for SVV estimation that is independent of roll velocity. Moreover, this static SVV hysteresis is already present at roll angles below 90°, where the A-effect dominates. We speculate that the hysteresis found by Kaptein and Van Giesbergen (2004, 2005) for the switching between A- and E-effects around 135° is a direct result of static hysteresis for the A-effect. In the instability zone, the direction-dependent magnitude of the A-effect may influence the roll position for switching between A- and E-effects.

Bias

Bias is the result of systematic error in the design and conduct of the study, such that the observed results in the sample will be different from the true results. Bias occurs due to flaws in the method of selection of study participants or in the process of gathering information regarding exposure and disease. This systematic error is different from random error due to sampling variability, which results from the use of a sample to estimate parameters for the reference population. We will discuss two broad categories of bias: selection bias and information bias.

Bias, or Systematic Error

Turning to bias, or systematic error, there is also a sampling component. First, the sample frame (i.e., the list or enumeration of elements in the population) may either omit or double count units. For example, the U.S. Census both misses people (especially African-Americans and immigrants) and counts others twice (especially people with more than one residence), and samples based on the census reflect these limitations. Second, certain housing units, such as new dwellings, secondary units (e.g., basement apartments in what appears to be a single-family dwelling), and remote dwellings, tend to be missed in the field. Likewise, within housing units, certain individuals, such as boarders, tend to be underrepresented and some respondent selection methods fail to work in an unbiased manner (e.g., the last/next birthday method overrepresents those who answer the sample-screening questions). Third, various statistical sampling errors occur. Routinely, the power of samples is overestimated because design effects are not taken into consideration. Also, systematic sampling can turn out to be correlated with various attributes of the target population. For example, in one study, both the experimental form and respondent selection were linked by systematic sampling in such a way that older household members were disproportionately assigned to one experimental version of the questionnaire, thus failing to randomize respondents to both experimental forms.

Nonsampling error comes from both nonobservational and observational errors. The first type of nonobservational error is coverage error, in which a distinct segment of the target population is not included in sample. For example, in the United States, preelection random-digit-dialing (RDD) polls want to generalize to the voting population, but systematically exclude all voters not living in households with telephones. Likewise, samples of businesses often underrepresent smaller firms. The second type of nonobservational error consists of nonresponse (units are included in the sample, but are not successfully interviewed). Nonresponse has three main causes: refusal to participate, failure to contact because people are away from home (e.g., working or on vacation), and all other reasons (such as illness and mental and/or physical handicaps).

Observational error includes collection, processing, and analysis errors. As with variable error, collection error is related to mode, instrument, interviewer, and respondent. Mode affects population coverage. Underrepresentation of the deaf and poor occurs in telephone surveys, and of the blind and illiterate, in mail surveys. Mode also affects the volume and quality of information gathered. Open-ended questions get shorter, less complete answers on telephone surveys, compared to in-person interviews. Bias also is associated with the instrument. Content, or the range of information covered, obviously determines what is collected. One example of content error is when questions presenting only one side of an issue are included, such as is commonly done in what is known as advocacy polling. A second example is specification error, in which one or more essential variable is omitted so that models cannot be adequately constructed and are therefore misspecified.

Various problematic aspects of question wordings can distort questions. These include questions that are too long and complex, are double-barreled, include double negatives, use loaded terms, and contain words that are not widely understood. For example, the following item on the Holocaust is both complex and uses a double negative: “As you know, the term ‘holocaust’ usually refers to the killing of millions of Jews in Nazi death camps during World War II. Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” After being presented with this statement in a national U.S. RDD poll in 1992, 22% of respondents said it was possible that the Holocaust never happened, 65% said that it was impossible that it never happened, and 12% were unsure. Subsequent research, however, demonstrated that many people had been confused by the wording and that Holocaust doubters were actually about 2% of the population, not 22%. Error from question wording also occurs when terms are not understood in a consistent manner.

The response scales offered also create problems. Some formats, such as magnitude measurement scaling, are difficult to follow, leaving many, especially the least educated, unable to express an opinion. Even widely used and simple scales can cause error. The 10-point scalometer has no clear midpoint and many people wrongly select point 5 on the 1–10 scale in a failed attempt to place themselves in the middle. Context, or the order of items in a survey, also influences responses in a number of quite different ways. Prior questions may activate certain topics and make them more accessible (and thus more influential) when later questions are asked. Or they may create a contrast effect under which the prior content is excluded from later consideration under a nonrepetition rule. A norm of evenhandedness may be created that makes people answer later questions in a manner consistent with earlier questions. For example, during the Cold War, Americans, after being asked if American reporters should be allowed to report the news in Russia, were much more likely to say that Russian reporters should be allowed to cover stories in the United States, compared to when the questions about Russian reporters were asked first. Even survey introductions can influence the data quality of the subsequent questions.

Although social science scholars hope that interviewers merely collect information, in actuality, interviewers also affect what information is reported. First, the mere presence of an interviewer usually magnifies social desirability effects, so that there is more underreporting of sensitive behaviors to interviewers than when self- completion is used. Second, basic characteristics of interviewers influence responses. For example, Whites express more support for racial equality and integration when interviewed by Blacks than when interviewed by Whites. Third, interviewers may have points of view that they convey to respondents, leading interviewers to interpret responses, especially to open-ended questions, in light of their beliefs.

Much collection error originates from respondents. Some problems are cognitive. Even given the best of intentions, people are fallible sources. Reports of past behaviors may be distorted due to forgetting the incidents or misdating them. Minor events will often be forgotten, and major events will frequently be recalled as occurring more recently than was actually the case. Of course, respondents do not always have the best of intentions. People tend to underreport behaviors that reflect badly on themselves (e.g., drug use and criminal records) and to overreport positive behaviors (e.g., voting and giving to charities).

Systematic error occurs during the processing of data. One source of error relates to the different ways in which data may be coded. A study of social change in Detroit initially found large changes in respondents’ answers to the same open-ended question asked and coded several decades apart. However, when the original open-ended responses from the earlier survey were recoded by the same coders who coded the latter survey, the differences virtually disappeared, indicating that the change had been in coding protocols and execution, not in the attitudes of Detroiters. Although data-entry errors are more often random, they can seriously bias results. For example, at one point in time, no residents of Hartford, Connecticut were being called for jury duty; it was discovered that the new database of residents had been formatted such that the “d” in “Hartford” fell in a field indicating that the listee was dead. Errors can also occur when data are transferred. Examples include incorrect recoding, misnamed variables, and misspecified data field locations. Sometimes loss can occur without any error being introduced. For example, 20 vocabulary items were asked on a Gallup survey in the 1950s and a summary scale was created. The summary scale data still survive, but the 20 individual variables have been lost. Later surveys included 10 of the vocabulary items, but they cannot be compared to the 20-item summary scale.

Wrong or incomplete documentation can lead to error. For example, documentation on the 1967 Political Participation Study (PPS) indicated that one of the group memberships asked about was “church-affiliated groups.” Therefore, when the group membership battery was later used in the General Social Surveys (GSSs), religious groups were one of the 16 groups presented to respondents. However, it was later discovered that church-affiliated groups had not been explicitly asked about on the earlier survey, but that the designation had been pulled out of an “other-specify” item. Because the GSS explicitly asked about religious groups, it got many more mentions than had appeared in the PPS; this was merely an artifact of different data collection procedures that resulted from unclear documentation.

Most discussions of total survey error stop at the data-processing stage. But data do not speak for themselves. Data “speak” when they are analyzed, and the analysis is reported by researchers. Considerable error is often introduced at this final stage. Models may be misspecified, not only by leaving crucial variables out of the survey, but also by omitting such variables from the analysis, even when they are collected. All sorts of statistical and computational errors occur during analysis. For example, in one analysis of a model explaining levels of gun violence, a 1 percentage point increase from a base incidence level of about 1% was misdescribed as a 1% increase, rather than as a 100% increase. Even when a quantitative analysis is done impeccably, distortion can occur in the write-up. Common problems include the use of jargon, unclear writing, the overemphasis and exaggeration of results, inaccurate descriptions, and incomplete documentation. Although each of the many sources of total survey error can be discussed individually, they constantly interact with one another in complex ways. For example, poorly trained interviewers are more likely to make mistakes with complex questionnaires, the race of the interviewer can interact with the race of respondents to create response effects, long, burdensome questionnaires are more likely to create fatigue among elderly respondents, and response scales using full rankings are harder to do over the phone than in person. In fact, no stage of a survey is really separate from the other stages, and most survey error results from, or is shaped by, interactions between the various components of a survey.

Assessing measurement invariance

Bias is a technical term that addresses systematic errors that lead to differential interpretation of scores. In order to evaluate bias, we must determine whether knowledge of an examinee’s group membership influences the examinee’s score on the measured variance (e.g., an item, subdomain, or test), given the examinee’s status on the latent variance of interest (Millsap, 2011). Consequently, for a test to be fair (from a psychometric perspective) it should exhibit measurement invariance across all distinctive subgroups being evaluated. The degree to which the construct measured by a test is consistent across subgroups is known as construct equivalence (CE). CE is of special concern in cross-cultural research whereby constructs, such as intelligence and morality can be culturally affected (Van de Vijver & Poortinga, 2005). It is also associated with test adaptations or computer-based testing whereby the conditions of test administration are altered. The degree to which such alterations affect the construct is unknown.

CE can be evaluated statistically by Confirmatory Factor Analysis (CFA) and weighted multidimensional scaling (MDS), as they are able to analyze the structure of data from multiple groups simultaneously. CFA evaluated the hypothesized test structure, whereas MDS is an exploratory analysis that fits dimensions to best account for the data in all groups. In CFA the degree to which the hypothetical structure adequately fits the data for multiple groups can be analyzed using descriptive statistics, such as root mean square error of approximation, standardized root mean square residual, and adjusted goodness-of-fit statistic.

There are numerous statistical approaches for assessing measurement in variance. These methods can be classified into three groups: (1) linear measurements models, (2) nonlinear measurements models, and (3) observed score methods (Millsap, 2011). These approaches can be broken down into methods that estimate invariance at the scale and item levels (Zumbo, 2003). Scale-level analyses focus on the degree of invariance observed within common factor analytic models across groups. In contrast, item-level analyses and differential item functioning (DIF) examine invariance separately for each item. DIF examines the situation in which examiners who have equal standing on the target construct but those who come from different groups (e.g., ethnicity, age) have different probabilities of responding to the item (Holland & Thayer, 1988). “DIF represents a statistical interaction between group membership and item performance after matching examinees across groups on some criterion (usually total test score)” (Sireci & Sukin, 2013).

3 Heuristics

The study of heuristics tends to focus on systematic errors in human decision making and these heuristics often help to understand anomalies of inferring expectations from evidence (see Heuristics in Social Cognition).

Three heuristics that deal with probabilistic thinking have received considerable attention: (a) availability, (b) representativeness, and (c) anchoring and adjustment. The availability heuristic refers to the tendency to assess the probability of an event based on the ease with which instances of that event come to mind. This heuristic has been investigated in a variety of domains and relates probability estimates to memory access. Generally people overestimate the probability of an event if concrete instances of that event are easily accessible in memory. Generally, ease of recall and frequency of occurrence are correlated. A number of factors that affect memory are, however, unrelated to probability. For example, vivid images are easier to recall than pallid ones. Thus, having been involved in a serious car accident is likely to be better remembered than annual statistics about the frequency of (types of) traffic accidents. The former is likely to have more impact on probability estimates than the latter. Dawes (1994) argued that the salience of negative and relatively extreme exemplars of drug addicts can bias policy-makers’ perceptions of the entire group and result in negative attitudes toward programs such as the provision of clean needles to prevent a further spread of the AIDS virus.

The representativeness heuristic refers to the tendency to assess the probability that a stimulus belongs to a particular class by judging the degree to which that event corresponds to an appropriate mental model. Kahneman and Tversky (1973) reported a well-known example of how ignoring prior probabilities can affect judgment. In their study, respondents were provided with brief personality sketches, supposedly of engineers and lawyers. They were asked to assess the probability that each sketch described a member of one profession or the other. Half the respondents were told the population from which the sketches were drawn consisted of 30 engineers and 70 lawyers, the remaining respondents were told that there were 70 engineers and 30 lawyers. Findings showed that the prior probabilities were essentially ignored, and that respondents estimated the probability of class membership by judging how similar each personality sketch was to their mental model of an engineer or a lawyer.

Anchoring and adjustment refers to a general judgment process in which an initially given or generated response serves as an anchor, and other information is insufficiently used to adjust that response. The anchoring and adjustment heuristic is based on the assumption that people often start their judgmental process by focusing on some initial value that serves as an anchor. The biases related to this heuristic stem from two distinct aspects. First, one could use irrelevant anchors, second one could insufficiently adjust up or down from an original starting value or anchor.

‘Experimenter artifacts’ and ‘subject artifacts’ refer to systematic errors that can be attributed to uncontrolled aspects of the interaction between the researcher and the research participants or to other potentially bias-prone aspects of a research situation, such as the analysis, interpretation, and reporting of the research results. Particular experimenter artifacts that have been studied include observer, interpreter, intentional, biosocial, psychosocial, situational, modeling, and experimenter-expectancy effects. Subject artifacts that have been primarily investigated are generally concerned with the motivation of the research participants and their sensitivity and compliance with task-orienting cues. Discussed in this article are the history of artifacts in the social and behavioral sciences, the specific nature and control of experimenter and subject artifacts, and the delicate balance between ethical accountability, and the avoidance of artifacts in research with human participants.

10.03.6 The Problem of Definition

The definition of test bias has produced considerable continuing debate among measurement and assessment experts (Angoff, 1976; Bass, 1976; Bernal, 1975; Bond, 1981; Cleary et al., 1975; Cole & Moss, 1989; Cronbach, 1976; Dana, 1993; Darlington, 1978; Einhorn & Bass, 1970; Flaugher, 1978; Gordon, 1984; Gross & Su, 1975; Helms, 1992; Humphreys, 1973; Hunter & Schmidt, 1976, 1978; Jackson, 1980; Linn, 1976; McNemar, 1975; Moreland, 1996; Novick & Petersen, 1976; Padilla, 1988; Petersen & Novick, 1976; Reschly, 1980; Reynolds, 1978; 1982b, 1995; Reynolds & Brown, 1984; Sawyer, Cole, & Cole, 1976; Schmidt & Hunter, 1974; Thorndike, 1971). Although the resulting debate has generated a number of selection models with which to examine bias, selection models focus on the decision-making system and not on the test itself. The various selection models are discussed at some length in Schmidt and Hunter (1974), Hunter, Schmidt, and Rauschenberger (1984), Jensen (1980), Petersen and Novick (1976), and Ramsey (1979). The choice of a decision-making system (especially a system for educational decision-making) must ultimately be a societal one; as such, it will depend to a large extent on the value system and goals of the society. Thus, before a model for test use in selection (whether ultimately selection is for a treatment program, a job, a college, etc.) can be chosen, it must be decided whether the ultimate goal of selection is equality of opportunity, equality of outcome, or representative equality (these concepts are discussed in more detail in Nichols, 1978).

“Equality of opportunity” is a competitive model wherein selection is based on ability. As more eloquently stated by Lewontin (1970), under equality of opportunity, “true merit … will be the criterion of men’s earthly reward” (p. 92). “Equality of outcome” is a selection model based on ability deficits. Schooling provides a good model to illustrate these concepts that are also applicable to mental health. Compensatory and remedial programs are typically constructed on the basis of the equality-of-outcome model. Children of low ability or children believed to be a high risk for academic failure are selected for remedial, compensatory, or other special educational programs. Adults vying for jobs may be placed in specialized job training programs. In a strictly predictive sense, tests are used in a similar manner under both of these models. However, under equality of opportunity, selection is based on the prediction of a high level of criterion performance; under equality of outcome, selection is determined by the prediction of “failure” or a preselected low level of criterion performance. Interestingly, it is the failure of compensatory and remedial education programs to bring the disadvantaged learner to “average” levels of performance that resulted in the charges of test bias now in vogue.

The model of “representative equality” also relies on selection, but selection that is proportionate to numerical representation of subgroups in the population under consideration. Representative equality is typically thought to be independent of the level of ability within each group; however, models can be constructed that select from each subgroup the desired proportion of individuals (i) according to relative ability level of the group, (ii) independent of group ability, or (iii) according to some decision rule between these two positions. Even under the conditions of representative equality, it is imperative to employ a selection device (test) that will rank order individuals within groups in a reliable and valid manner. The best way to ensure fair selection under any of these models is to employ tests whose scores are equally reliable and equally valid for all groups concerned. The tests employed should also yield the most reliable and most valid scores for all groups under consideration. The question of test bias per se then becomes a question of test validity. Test use (i.e., fairness) may be defined as biased or nonbiased only by the societal value system; at present, this value system within the USA is leaning strongly toward some variant of the representative-equality selection model. In other sociopolitical structures, other models may be more appropriate. As noted above, all models are facilitated by the use of a nonbiased test. That is, the use of a test with equivalent cross-group validities makes for the most parsimonious selection model, greatly simplifying the creation and application of the selection model that has been chosen.

There are three types of validity as traditionally conceived: content, construct, and predictive (or criterion-related). Test bias may exist under any or all of these categories of validity. Though no category of validity is completely independent of any other category, each is discussed separately here for the purposes of clarity and convenience. (All true evidence of validity is as likely as not to be construct validity, and other, more detailed divisions including this one are for convenience of discussion.) Frequently encountered in bias research are the terms “single-group validity” and “differential validity.” Single-group validity refers to the phenomenon of a score interpretation being valid for one group but not another. Differential validity refers to a condition where an interpretation is valid for all groups concerned, but the degree of validity varies as a function of group membership. Although these terms have been most often applied to predictive or criterion-related validity (validity coefficients are then examined for significance and compared across groups), the concepts of single-group and differential validity are equally applicable to content and construct validity.