Random sampling error

Discover 23 more articles on this topic

Anyone who reads polls on the internet, or in newspapers, should be aware that sampling errors could vastly influence the data and lead people to draw incorrect conclusions.

To further compound the random sampling errors, many survey companies, newspapers and pundits are well aware of this, and deliberately manipulate polls to give favorable results.

In any experiment where it is impossible to sample an entire population, usually due to practicality and expense, a representative sample must be used.

Of course, when you use a sample group, it can never fully match the entire population, and there will always be some likelihood of random sampling error.

Any researcher must strive to ensure that the sample is as representative as possible, and statistical tests have inbuilt checks and balances to take this into account.

To illustrate how to ensure that your statistics are as accurate as possible, we are going to use the example of an opinion poll. These are one of the most commonly misinterpreted representations of data, and failure to take into account the nuances of statistics can paint an incorrect picture.

Margin of Error — A False Picture

The problem is, when you see an opinion poll in a newspaper or internet site, you will usually see a margin of error, such a + or — 3%. The temptation is to think that the polls will be accurate within this figure.

For example, if a poll gives one political party (A) a 42% share of the vote, and the other (B) 39%, this opens up a number of possible results. (A) could have 45%, (B) 36%. Both could be 39% or (B) could actually be ahead, 42% versus 39%. Of course, the results could show any variation in between those extremes. Complicated enough?

To complicate the picture further, even this random sampling error can be wildly inaccurate. Any opinion poll may give the margin of error, but this can convey a false sense of security and make people assume that the results ‘must’ lie within this range.

In fact, these figures could actually be completely wrong, and the numbers are only ever an estimate.

The Problem With Random Sampling Error

The problem is that these results only show the random sampling error within that specific group. They show the chances of the results in that group occurring purely by chance, exactly like the 95% confidence margin employed by many scientific researchers.

However, this is a very narrow definition and is often misunderstood.

In an opinion poll, there is no guarantee that the sample of 1000 or 10 000 people is truly representative of the larger population as a whole.

There have been many extremely inaccurate polls conducted over the years, and they fell down due to poor design and not understanding all of the relevant factors.

For example, an opinion poll company conducting telephone polls may make the mistake of only telephoning during office hours, when most of the population is at work, skewing the data.

In addition, poorer families do not always have a fixed line telephone and use unregistered cell phones, again leaving a huge potential for inaccuracy. The margins of error would be perfectly acceptable, in these cases, but the overall findings would still be horribly wrong.

Modern polling companies are very skilled at designing polls to select samples from many elements of the population, and via various media, so big errors rarely happen. Despite this, opinion polls must always be taken as a guide only, not an exact representation of how an election is likely to unfold.

Random Sampling Error and Experimental Design

The mistakes made by pollsters relate directly to any type of experiment involving random sample groups.

Statistics can only work with the data provided and, if your design is poorly thought out, will not be able to cover up these errors. Garbage in definitely equals garbage out.

Bibliography

Husch, B. (1971). Planning a Forest Inventory. Rome, Italy: Food and Agriculture Organization of the United Nations

Urdan, T.C. (2005). Statistics in Plain English, Mahwah, NJ: Lawrence Erlbaum

Weisberg, H.F. (2005).The Total Survey Error Approach: A Guide to the New Science of Survey Research. Chicago: University of Chicago Press

What Is a Sampling Error?

A sampling error is a statistical error that occurs when an analyst does not select a sample that represents the entire population of data. As a result, the results found in the sample do not represent the results that would be obtained from the entire population.

Sampling is an analysis performed by selecting a number of observations from a larger population. The method of selection can produce both sampling errors and non-sampling errors.

Understanding Sampling Errors

A sampling error is a deviation in the sampled value versus the true population value. Sampling errors occur because the sample is not representative of the population or is biased in some way. Even randomized samples will have some degree of sampling error because a sample is only an approximation of the population from which it is drawn.

Calculating Sampling Error

The sampling error formula is used to calculate the overall sampling error in statistical analysis. The sampling error is calculated by dividing the standard deviation of the population by the square root of the size of the sample, and then multiplying the resultant with the Z-score value, which is based on the confidence interval.

Sampling Error

=

Z

×

σ

n

where:

Z

=

Z

 score value based on the

 confidence interval (approx

=

1.96

)

σ

=

Population standard deviation

n

=

Size of the sample

begin{aligned}&text{Sampling Error}=Ztimesfrac{sigma}{sqrt{n}}\&textbf{where:}\&Z=Ztext{ score value based on the}\&qquad text{confidence interval (approx}=1.96)\&sigma=text{Population standard deviation}\&n=text{Size of the sample}end{aligned}

​Sampling Error=Z×n​σ​where:Z=Z score value based on the confidence interval (approx=1.96)σ=Population standard deviationn=Size of the sample​

Types of Sampling Errors

There are different categories of sampling errors.

Population-Specific Error

A population-specific error occurs when a researcher doesn’t understand who to survey.

Selection Error

Selection error occurs when the survey is self-selected, or when only those participants who are interested in the survey respond to the questions. Researchers can attempt to overcome selection error by finding ways to encourage participation.

Sample Frame Error

A sample frame error occurs when a sample is selected from the wrong population data.

Non-response Error

A non-response error occurs when a useful response is not obtained from the surveys because researchers were unable to contact potential respondents (or potential respondents refused to respond).

Eliminating Sampling Errors

The prevalence of sampling errors can be reduced by increasing the sample size. As the sample size increases, the sample gets closer to the actual population, which decreases the potential for deviations from the actual population. Consider that the average of a sample of 10 varies more than the average of a sample of 100. Steps can also be taken to ensure that the sample adequately represents the entire population.

Researchers might attempt to reduce sampling errors by replicating their study. This could be accomplished by taking the same measurements repeatedly, using more than one subject or multiple groups, or by undertaking multiple studies.

Random sampling is an additional way to minimize the occurrence of sampling errors. Random sampling establishes a systematic approach to selecting a sample. For example, rather than choosing participants to be interviewed haphazardly, a researcher might choose those whose names appear first, 10th, 20th, 30th, 40th, and so on, on the list.

Examples of Sampling Errors

Assume that XYZ Company provides a subscription-based service that allows consumers to pay a monthly fee to stream videos and other types of programming via an Internet connection.

The firm wants to survey homeowners who watch at least 10 hours of programming via the Internet per week and that pay for an existing video streaming service. XYZ wants to determine what percentage of the population is interested in a lower-priced subscription service. If XYZ does not think carefully about the sampling process, several types of sampling errors may occur.

A population specification error would occur if XYZ Company does not understand the specific types of consumers who should be included in the sample. For example, if XYZ creates a population of people between the ages of 15 and 25 years old, many of those consumers do not make the purchasing decision about a video streaming service because they may not work full-time. On the other hand, if XYZ put together a sample of working adults who make purchase decisions, the consumers in this group may not watch 10 hours of video programming each week.

Selection error also causes distortions in the results of a sample. A common example is a survey that only relies on a small portion of people who immediately respond. If XYZ makes an effort to follow up with consumers who don’t initially respond, the results of the survey may change. Furthermore, if XYZ excludes consumers who don’t respond right away, the sample results may not reflect the preferences of the entire population.

Sampling Error vs. Non-sampling Error

There are different types of errors that can occur when gathering statistical data. Sampling errors are the seemingly random differences between the characteristics of a sample population and those of the general population. Sampling errors arise because sample sizes are inevitably limited. (It is impossible to sample an entire population in a survey or a census.)

Company XYZ will also want to avoid non-sampling errors. Non-sampling errors are errors that result during data collection and cause the data to differ from the true values. Non-sampling errors are caused by human error, such as a mistake made in the survey process.

If one group of consumers only watches five hours of video programming a week and is included in the survey, that decision is a non-sampling error. Asking questions that are biased is another type of error.

The Bottom Line

Sampling error occurs when a sample drawn from a population deviates somewhat from that true population. Large sampling errors can lead to incorrect estimates or inferences made about the population based on statistical analysis of that sample.

In general, sampling errors can be placed into four categories: population-specific error, selection error, sample frame error, or non-response error. A population-specific error occurs when the researcher does not understand who they should survey. A selection error occurs when respondents self-select their participation in the study. (This results in only those that are interested in responding, which skews the results.) A sample frame error occurs when the wrong sub-population is used to select a sample. Finally, a non-response error occurs when potential respondents are not successfully contacted or refuse to respond.

From Wikipedia, the free encyclopedia

In statistics, sampling errors are incurred when the statistical characteristics of a population are estimated from a subset, or sample, of that population. Since the sample does not include all members of the population, statistics of the sample (often known as estimators), such as means and quartiles, generally differ from the statistics of the entire population (known as parameters). The difference between the sample statistic and population parameter is considered the sampling error.^[1] For example, if one measures the height of a thousand individuals from a population of one million, the average height of the thousand is typically not the same as the average height of all one million people in the country.

Since sampling is almost always done to estimate population parameters that are unknown, by definition exact measurement of the sampling errors will not be possible; however they can often be estimated, either by general methods such as bootstrapping, or by specific methods incorporating some assumptions (or guesses) regarding the true population distribution and parameters thereof.

Description[edit]

Sampling Error[edit]

The sampling error is the error caused by observing a sample instead of the whole population.^[1] The sampling error is the difference between a sample statistic used to estimate a population parameter and the actual but unknown value of the parameter.^[2]

Effective Sampling[edit]

In statistics, a truly random sample means selecting individuals from a population with an equivalent probability; in other words, picking individuals from a group without bias. Failing to do this correctly will result in a sampling bias, which can dramatically increase the sample error in a systematic way. For example, attempting to measure the average height of the entire human population of the Earth, but measuring a sample only from one country, could result in a large over- or under-estimation. In reality, obtaining an unbiased sample can be difficult as many parameters (in this example, country, age, gender, and so on) may strongly bias the estimator and it must be ensured that none of these factors play a part in the selection process.

Even in a perfectly non-biased sample, the sample error will still exist due to the remaining statistical component; consider that measuring only two or three individuals and taking the average would produce a wildly varying result each time. The likely size of the sampling error can generally be reduced by taking a larger sample.^[3]

Sample Size Determination[edit]

The cost of increasing a sample size may be prohibitive in reality. Since the sample error can often be estimated beforehand as a function of the sample size, various methods of sample size determination are used to weigh the predicted accuracy of an estimator against the predicted cost of taking a larger sample.

Bootstrapping and Standard Error[edit]

As discussed, a sample statistic, such as an average or percentage, will generally be subject to sample-to-sample variation.^[1] By comparing many samples, or splitting a larger sample up into smaller ones (potentially with overlap), the spread of the resulting sample statistics can be used to estimate the standard error on the sample.

In Genetics[edit]

The term «sampling error» has also been used in a related but fundamentally different sense in the field of genetics; for example in the bottleneck effect or founder effect, when natural disasters or migrations dramatically reduce the size of a population, resulting in a smaller population that may or may not fairly represent the original one. This is a source of genetic drift, as certain alleles become more or less common), and has been referred to as «sampling error»,^[4] despite not being an «error» in the statistical sense.

See also[edit]

• Margin of error
• Propagation of uncertainty
• Ratio estimator
• Sampling (statistics)

References[edit]

When data is randomly sampled from a population, the randomness with which observations are selected from the population ensures that the value of statistics computed by analyzing the sample will differ to some extent from the values computed if analyzing the entire population. Differences between the sample values and the population values are known as sampling error.

When computing a statistic using a sample, it is often possible to compute the likely extent of sampling error. This makes it possible to make conclusions about the extent to which statistics computed from a sample will reflect the truth about the world.

The amount of sampling error is determined by four things. These are the sample size, the sampling method, the inherent variability in the population, and the population size.

Worked example

Stack Overflow, a popular online question-and-answer forum for software developers, recently conducted a survey that found that about 10% of people on their site were female. How much sampling error is attached to such an estimate? To understand the extent of sampling error, we will start by investigating the amount of sampling error we might expect if:

1. We assume that it is true that 10% of people who use Stack Overflow are women.
2. We have a sample of ten people.

If we randomly select ten people and 10% of people are female, this means that each of the people we choose will have a 10% chance of being female (this is a heroic assumption, due to factors discussed later in this article and also to non-sampling error, but it greatly simplifies the explanation of sampling error).

So, the chance of ten out of ten people in our sample being female is 0.1*0.1*0.1*0.1*0.1*0.1*0.1*0.1*0.1*0.1 =  0.00000001%. That is, there is essentially no chance that if in truth 10% of Stack Overflow users were women, that we would observe this as being 100% in a survey of 10 people.

By contrast, the probability of there being no women in our sample is 0.9*0.9*0.9*0.9*0.9*0.9*0.9*0.9*0.9*0.9 = 34.9%, where 0.9 is the probability that a man is selected at random, where this is 1 minus the probability of being a woman.

With a bit more math we can compute the chart below, which shows the probability that in our sample of ten people we would get 0% women, 10% women, etc.

What we really want to know is the true value in the population. Is it 10%, 5%, or 20%? Without talking to everybody we cannot know for sure, but we can make some inferences. We can see from the chart that if the true figure is 10% (which is what was assumed when creating the chart), there is a 34.9% chance of observing 0% women, 38.7% chance of observing 10% women, and 19.4% chance of observing 20% (2 or fewer) women. If we add these up, we can say if the true value is 10%, there is 93% chance that we would observe either 0%, 10%, or 20% – and virtually no chance of observing a sample of more than 40% female.

The next chart shows the calculations again, this time for 100 people. This chart tells us that if it was true that 10% of Stack Overflow visitors were female, then we could be confident of observing a value in a sample of 100 between 0% and 20%.

The actual Stack Overflow survey question about gender was answered by 35,990 people. Using the same math again, we can be extremely confident that if the truth is that 10% of all people on Stack Overflow are women, sampling error will mean that we observe a value between 9.4% and 10.5%. “Extremely confident” here means that there is a 99.9% probability that we would observe a value between 9.4% and 10.5%. Here, 9.4% and 10.5% are called the 99.9% confidence interval.

The role of sample size

As has been illustrated above, the bigger the sample size, the smaller the sampling error. The sampling error increases in proportion to the square root of the sample size. For example, when sample size is increased from 10 to 100, the sampling error halves, all else being equal.

The sampling method

In the calculations above, it was implicitly assumed that the samples were selected randomly (i.e., a simple random sample). With other types of samples, the math works differently. For example, with cluster sampling, the degree of sampling error is larger.

The inherent variability in the population

The example above investigated the sampling error for a proportion (i.e., a percentage). When investigating the sampling error for a mean (i.e., the average), we need to factor into the calculation the inherent variability in the population. The more variability, the greater the sampling error. For example, just as people vary more in their weights than their heights, the sampling error for weights is bigger than for heights.

The population size

The calculations above implicitly assumed an infinite population size. This assumption is rarely correct. If we modify the math to take into account the population size, the math gets more complicated. We see that the smaller the population, the smaller the sampling error. However, this effect is negligible unless the sample size is greater than 10% of the population size, so the effect of sample size can be safely ignored in most analyses.

Need to know more market research terminology? Brush up with our «What is» guides.