Median standard error

In statistics and probability theory, the median is the value separating the higher half from the lower half of a data sample, a population, or a probability distribution. For a data set, it may be thought of as «the middle» value. The basic feature of the median in describing data compared to the mean (often simply described as the «average») is that it is not skewed by a small proportion of extremely large or small values, and therefore provides a better representation of the center. Median income, for example, may be a better way to describe center of the income distribution because increases in the largest incomes alone have no effect on median. For this reason, the median is of central importance in robust statistics.

Finite data set of numbers[edit]

The median of a finite list of numbers is the «middle» number, when those numbers are listed in order from smallest to greatest.

If the data set has an odd number of observations, the middle one is selected. For example, the following list of seven numbers,

1, 3, 3, 6, 7, 8, 9

has the median of 6, which is the fourth value.

If the data set has an even number of observations, there is no distinct middle value and the median is usually defined to be the arithmetic mean of the two middle values.^[1][2] For example, this data set of 8 numbers

1, 2, 3, 4, 5, 6, 8, 9

has a median value of 4.5, that is . (In more technical terms, this interprets the median as the fully trimmed mid-range).

In general, with this convention, the median can be defined as follows: For a data set of elements, ordered from smallest to greatest,

if is odd,

if is even,

Comparison of common averages of values [ 1, 2, 2, 3, 4, 7, 9 ]

Type	Description	Example	Result
Midrange	Midway point between the minimum and the maximum of a data set	1, 2, 2, 3, 4, 7, 9	5
Arithmetic mean	Sum of values of a data set divided by number of values:	(1 + 2 + 2 + 3 + 4 + 7 + 9) / 7	4
Median	Middle value separating the greater and lesser halves of a data set	1, 2, 2, 3, 4, 7, 9	3
Mode	Most frequent value in a data set	1, 2, 2, 3, 4, 7, 9	2

Formal definition[edit]

Formally, a median of a population is any value such that at least half of the population is less than or equal to the proposed median and at least half is greater than or equal to the proposed median. As seen above, medians may not be unique. If each set contains less than half the population, then some of the population is exactly equal to the unique median.

The median is well-defined for any ordered (one-dimensional) data, and is independent of any distance metric. The median can thus be applied to classes which are ranked but not numerical (e.g. working out a median grade when students are graded from A to F), although the result might be halfway between classes if there is an even number of cases.

A geometric median, on the other hand, is defined in any number of dimensions. A related concept, in which the outcome is forced to correspond to a member of the sample, is the medoid.

There is no widely accepted standard notation for the median, but some authors represent the median of a variable x either as x͂ or as μ_1/2^[1] sometimes also M.^[3][4] In any of these cases, the use of these or other symbols for the median needs to be explicitly defined when they are introduced.

The median is a special case of other ways of summarizing the typical values associated with a statistical distribution: it is the 2nd quartile, 5th decile, and 50th percentile.

Uses[edit]

The median can be used as a measure of location when one attaches reduced importance to extreme values, typically because a distribution is skewed, extreme values are not known, or outliers are untrustworthy, i.e., may be measurement/transcription errors.

For example, consider the multiset

1, 2, 2, 2, 3, 14.

The median is 2 in this case, as is the mode, and it might be seen as a better indication of the center than the arithmetic mean of 4, which is larger than all but one of the values. However, the widely cited empirical relationship that the mean is shifted «further into the tail» of a distribution than the median is not generally true. At most, one can say that the two statistics cannot be «too far» apart; see § Inequality relating means and medians below.^[5]

As a median is based on the middle data in a set, it is not necessary to know the value of extreme results in order to calculate it. For example, in a psychology test investigating the time needed to solve a problem, if a small number of people failed to solve the problem at all in the given time a median can still be calculated.^[6]

Because the median is simple to understand and easy to calculate, while also a robust approximation to the mean, the median is a popular summary statistic in descriptive statistics. In this context, there are several choices for a measure of variability: the range, the interquartile range, the mean absolute deviation, and the median absolute deviation.

For practical purposes, different measures of location and dispersion are often compared on the basis of how well the corresponding population values can be estimated from a sample of data. The median, estimated using the sample median, has good properties in this regard. While it is not usually optimal if a given population distribution is assumed, its properties are always reasonably good. For example, a comparison of the efficiency of candidate estimators shows that the sample mean is more statistically efficient when—and only when— data is uncontaminated by data from heavy-tailed distributions or from mixtures of distributions.^{[citation needed]} Even then, the median has a 64% efficiency compared to the minimum-variance mean (for large normal samples), which is to say the variance of the median will be ~50% greater than the variance of the mean.^[7][8]

Probability distributions[edit]

For any real-valued probability distribution with cumulative distribution function F, a median is defined as any real number m that satisfies the inequalities

An equivalent phrasing uses a random variable X distributed according to F:

Note that this definition does not require X to have an absolutely continuous distribution (which has a probability density function f), nor does it require a discrete one. In the former case, the inequalities can be upgraded to equality: a median satisfies

Any probability distribution on R has at least one median, but in pathological cases there may be more than one median: if F is constant 1/2 on an interval (so that f=0 there), then any value of that interval is a median.

Medians of particular distributions[edit]

The medians of certain types of distributions can be easily calculated from their parameters; furthermore, they exist even for some distributions lacking a well-defined mean, such as the Cauchy distribution:

• The median of a symmetric unimodal distribution coincides with the mode.
• The median of a symmetric distribution which possesses a mean μ also takes the value μ.
  • The median of a normal distribution with mean μ and variance σ² is μ. In fact, for a normal distribution, mean = median = mode.
  • The median of a uniform distribution in the interval [a, b] is (a + b) / 2, which is also the mean.
• The median of a Cauchy distribution with location parameter x₀ and scale parameter y is x₀, the location parameter.
• The median of a power law distribution x^−a, with exponent a > 1 is 2^{1/(a − 1)}x_min, where x_min is the minimum value for which the power law holds^[10]
• The median of an exponential distribution with rate parameter λ is the natural logarithm of 2 divided by the rate parameter: λ⁻¹ln 2.
• The median of a Weibull distribution with shape parameter k and scale parameter λ is λ(ln 2)^1/k.

Properties[edit]

Optimality property[edit]

The mean absolute error of a real variable c with respect to the random variable X is

Provided that the probability distribution of X is such that the above expectation exists, then m is a median of X if and only if m is a minimizer of the mean absolute error with respect to X.^[11] In particular, m is a sample median if and only if m minimizes the arithmetic mean of the absolute deviations.^[12]

More generally, a median is defined as a minimum of

as discussed below in the section on multivariate medians (specifically, the spatial median).

This optimization-based definition of the median is useful in statistical data-analysis, for example, in k-medians clustering.

Inequality relating means and medians[edit]

If the distribution has finite variance, then the distance between the median and the mean is bounded by one standard deviation.

This bound was proved by Book and Sher in 1979 for discrete samples,^[13] and more generally by Page and Murty in 1982.^[14] In a comment on a subsequent proof by O’Cinneide,^[15] Mallows in 1991 presented a compact proof that uses Jensen’s inequality twice,^[16] as follows. Using |·| for the absolute value, we have

The first and third inequalities come from Jensen’s inequality applied to the absolute-value function and the square function, which are each convex. The second inequality comes from the fact that a median minimizes the absolute deviation function .

Mallows’s proof can be generalized to obtain a multivariate version of the inequality^[17] simply by replacing the absolute value with a norm:

where m is a spatial median, that is, a minimizer of the function The spatial median is unique when the data-set’s dimension is two or more.^[18][19]

An alternative proof uses the one-sided Chebyshev inequality; it appears in an inequality on location and scale parameters. This formula also follows directly from Cantelli’s inequality.^[20]

Unimodal distributions[edit]

For the case of unimodal distributions, one can achieve a sharper bound on the distance between the median and the mean:

.^[21]

A similar relation holds between the median and the mode:

Jensen’s inequality for medians[edit]

Jensen’s inequality states that for any random variable X with a finite expectation E[X] and for any convex function f

This inequality generalizes to the median as well. We say a function f: R → R is a C function if, for any t,

is a closed interval (allowing the degenerate cases of a single point or an empty set). Every convex function is a C function, but the reverse does not hold. If f is a C function, then

If the medians are not unique, the statement holds for the corresponding suprema.^[22]

Medians for samples[edit]

This section discusses the theory of estimating a population median from a sample. To calculate the median of a sample «by hand,» see § Finite data set of numbers above.

The sample median[edit]

Efficient computation of the sample median[edit]

Even though comparison-sorting n items requires Ω(n log n) operations, selection algorithms can compute the kth-smallest of n items with only Θ(n) operations. This includes the median, which is the n/2th order statistic (or for an even number of samples, the arithmetic mean of the two middle order statistics).^[23]

Selection algorithms still have the downside of requiring Ω(n) memory, that is, they need to have the full sample (or a linear-sized portion of it) in memory. Because this, as well as the linear time requirement, can be prohibitive, several estimation procedures for the median have been developed. A simple one is the median of three rule, which estimates the median as the median of a three-element subsample; this is commonly used as a subroutine in the quicksort sorting algorithm, which uses an estimate of its input’s median. A more robust estimator is Tukey’s ninther, which is the median of three rule applied with limited recursion:^[24] if A is the sample laid out as an array, and

med3(A) = median(A[1], A[n/2], A[n]),

then

ninther(A) = med3(med3(A[1 … 1/3n]), med3(A[1/3n … 2/3n]), med3(A[2/3n … n]))

The remedian is an estimator for the median that requires linear time but sub-linear memory, operating in a single pass over the sample.^[25]

Sampling distribution[edit]

The distributions of both the sample mean and the sample median were determined by Laplace.^[26] The distribution of the sample median from a population with a density function is asymptotically normal with mean and variance^[27]

where is the median of and is the sample size. A modern proof follows below. Laplace’s result is now understood as a special case of the asymptotic distribution of arbitrary quantiles.

For normal samples, the density is , thus for large samples the variance of the median equals ^[7] (See also section #Efficiency below.)

Derivation of the asymptotic distribution[edit]

We take the sample size to be an odd number and assume our variable continuous; the formula for the case of discrete variables is given below in § Empirical local density. The sample can be summarized as «below median», «at median», and «above median», which corresponds to a trinomial distribution with probabilities , and . For a continuous variable, the probability of multiple sample values being exactly equal to the median is 0, so one can calculate the density of at the point directly from the trinomial distribution:

.

Now we introduce the beta function. For integer arguments and , this can be expressed as . Also, recall that . Using these relationships and setting both and equal to allows the last expression to be written as

Hence the density function of the median is a symmetric beta distribution pushed forward by . Its mean, as we would expect, is 0.5 and its variance is . By the chain rule, the corresponding variance of the sample median is

.

The additional 2 is negligible in the limit.

Empirical local density[edit]

In practice, the functions and are often not known or assumed. However, they can be estimated from an observed frequency distribution. In this section, we give an example. Consider the following table, representing a sample of 3,800 (discrete-valued) observations:

v	0	0.5	1	1.5	2	2.5	3	3.5	4	4.5	5
f(v)	0.000	0.008	0.010	0.013	0.083	0.108	0.328	0.220	0.202	0.023	0.005
F(v)	0.000	0.008	0.018	0.031	0.114	0.222	0.550	0.770	0.972	0.995	1.000

Because the observations are discrete-valued, constructing the exact distribution of the median is not an immediate translation of the above expression for ; one may (and typically does) have multiple instances of the median in one’s sample. So we must sum over all these possibilities:

Here, i is the number of points strictly less than the median and k the number strictly greater.

Using these preliminaries, it is possible to investigate the effect of sample size on the standard errors of the mean and median. The observed mean is 3.16, the observed raw median is 3 and the observed interpolated median is 3.174. The following table gives some comparison statistics.

Sample size Statistic	3	9	15	21
Expected value of median	3.198	3.191	3.174	3.161
Standard error of median (above formula)	0.482	0.305	0.257	0.239
Standard error of median (asymptotic approximation)	0.879	0.508	0.393	0.332
Standard error of mean	0.421	0.243	0.188	0.159

The expected value of the median falls slightly as sample size increases while, as would be expected, the standard errors of both the median and the mean are proportionate to the inverse square root of the sample size. The asymptotic approximation errs on the side of caution by overestimating the standard error.

Estimation of variance from sample data[edit]

The value of —the asymptotic value of where is the population median—has been studied by several authors. The standard «delete one» jackknife method produces inconsistent results.^[28] An alternative—the «delete k» method—where grows with the sample size has been shown to be asymptotically consistent.^[29] This method may be computationally expensive for large data sets. A bootstrap estimate is known to be consistent,^[30] but converges very slowly (order of ).^[31] Other methods have been proposed but their behavior may differ between large and small samples.^[32]

Efficiency[edit]

The efficiency of the sample median, measured as the ratio of the variance of the mean to the variance of the median, depends on the sample size and on the underlying population distribution. For a sample of size from the normal distribution, the efficiency for large N is

The efficiency tends to as tends to infinity.

In other words, the relative variance of the median will be , or 57% greater than the variance of the mean – the relative standard error of the median will be , or 25% greater than the standard error of the mean, (see also section #Sampling distribution above.).^[33]

Other estimators[edit]

For univariate distributions that are symmetric about one median, the Hodges–Lehmann estimator is a robust and highly efficient estimator of the population median.^[34]

If data is represented by a statistical model specifying a particular family of probability distributions, then estimates of the median can be obtained by fitting that family of probability distributions to the data and calculating the theoretical median of the fitted distribution.^{[citation needed]} Pareto interpolation is an application of this when the population is assumed to have a Pareto distribution.

Multivariate median[edit]

Previously, this article discussed the univariate median, when the sample or population had one-dimension. When the dimension is two or higher, there are multiple concepts that extend the definition of the univariate median; each such multivariate median agrees with the univariate median when the dimension is exactly one.^{[34][35][36][37]}

Marginal median[edit]

The marginal median is defined for vectors defined with respect to a fixed set of coordinates. A marginal median is defined to be the vector whose components are univariate medians. The marginal median is easy to compute, and its properties were studied by Puri and Sen.^[34][38]

Geometric median[edit]

The geometric median of a discrete set of sample points in a Euclidean space is the^[a] point minimizing the sum of distances to the sample points.

In contrast to the marginal median, the geometric median is equivariant with respect to Euclidean similarity transformations such as translations and rotations.

Median in all directions[edit]

If the marginal medians for all coordinate systems coincide, then their common location may be termed the «median in all directions».^[40] This concept is relevant to voting theory on account of the median voter theorem. When it exists, the median in all directions coincides with the geometric median (at least for discrete distributions).

Centerpoint[edit]

An alternative generalization of the median in higher dimensions is the centerpoint.

[edit]

Interpolated median[edit]

When dealing with a discrete variable, it is sometimes useful to regard the observed values as being midpoints of underlying continuous intervals. An example of this is a Likert scale, on which opinions or preferences are expressed on a scale with a set number of possible responses. If the scale consists of the positive integers, an observation of 3 might be regarded as representing the interval from 2.50 to 3.50. It is possible to estimate the median of the underlying variable. If, say, 22% of the observations are of value 2 or below and 55.0% are of 3 or below (so 33% have the value 3), then the median is 3 since the median is the smallest value of for which is greater than a half. But the interpolated median is somewhere between 2.50 and 3.50. First we add half of the interval width to the median to get the upper bound of the median interval. Then we subtract that proportion of the interval width which equals the proportion of the 33% which lies above the 50% mark. In other words, we split up the interval width pro rata to the numbers of observations. In this case, the 33% is split into 28% below the median and 5% above it so we subtract 5/33 of the interval width from the upper bound of 3.50 to give an interpolated median of 3.35. More formally, if the values are known, the interpolated median can be calculated from

Alternatively, if in an observed sample there are scores above the median category, scores in it and scores below it then the interpolated median is given by

Pseudo-median[edit]

For univariate distributions that are symmetric about one median, the Hodges–Lehmann estimator is a robust and highly efficient estimator of the population median; for non-symmetric distributions, the Hodges–Lehmann estimator is a robust and highly efficient estimator of the population pseudo-median, which is the median of a symmetrized distribution and which is close to the population median.^[41] The Hodges–Lehmann estimator has been generalized to multivariate distributions.^[42]

Variants of regression[edit]

The Theil–Sen estimator is a method for robust linear regression based on finding medians of slopes.^[43]

Median filter[edit]

The median filter is an important tool of image processing, that can effectively remove any salt and pepper noise from grayscale images.

Cluster analysis[edit]

In cluster analysis, the k-medians clustering algorithm provides a way of defining clusters, in which the criterion of maximising the distance between cluster-means that is used in k-means clustering, is replaced by maximising the distance between cluster-medians.

Median–median line[edit]

This is a method of robust regression. The idea dates back to Wald in 1940 who suggested dividing a set of bivariate data into two halves depending on the value of the independent parameter : a left half with values less than the median and a right half with values greater than the median.^[44] He suggested taking the means of the dependent and independent variables of the left and the right halves and estimating the slope of the line joining these two points. The line could then be adjusted to fit the majority of the points in the data set.

Nair and Shrivastava in 1942 suggested a similar idea but instead advocated dividing the sample into three equal parts before calculating the means of the subsamples.^[45] Brown and Mood in 1951 proposed the idea of using the medians of two subsamples rather the means.^[46] Tukey combined these ideas and recommended dividing the sample into three equal size subsamples and estimating the line based on the medians of the subsamples.^[47]

Median-unbiased estimators[edit]

Any mean-unbiased estimator minimizes the risk (expected loss) with respect to the squared-error loss function, as observed by Gauss. A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as observed by Laplace. Other loss functions are used in statistical theory, particularly in robust statistics.

The theory of median-unbiased estimators was revived by George W. Brown in 1947:^[48]

An estimate of a one-dimensional parameter θ will be said to be median-unbiased if, for fixed θ, the median of the distribution of the estimate is at the value θ; i.e., the estimate underestimates just as often as it overestimates. This requirement seems for most purposes to accomplish as much as the mean-unbiased requirement and has the additional property that it is invariant under one-to-one transformation.

— page 584

Further properties of median-unbiased estimators have been reported.^{[49][50][51][52]} Median-unbiased estimators are invariant under one-to-one transformations.

There are methods of constructing median-unbiased estimators that are optimal (in a sense analogous to the minimum-variance property for mean-unbiased estimators). Such constructions exist for probability distributions having monotone likelihood-functions.^[53][54] One such procedure is an analogue of the Rao–Blackwell procedure for mean-unbiased estimators: The procedure holds for a smaller class of probability distributions than does the Rao—Blackwell procedure but for a larger class of loss functions.^[55]

History[edit]

Scientific researchers in the ancient near east appear not to have used summary statistics altogether, instead choosing values that offered maximal consistency with a broader theory that integrated a wide variety of phenomena.^[56] Within the Mediterranean (and, later, European) scholarly community, statistics like the mean are fundamentally a medieval and early modern development. (The history of the median outside Europe and its predecessors remains relatively unstudied.)

The idea of the median appeared in the 6th century in the Talmud, in order to fairly analyze divergent appraisals.^[57][58] However, the concept did not spread to the broader scientific community.

Instead, the closest ancestor of the modern median is the mid-range, invented by Al-Biruni.^{[59]: 31 [60]} Transmission of Al-Biruni’s work to later scholars is unclear. Al-Biruni applied his technique to assaying metals, but, after he published his work, most assayers still adopted the most unfavorable value from their results, lest they appear to cheat.^{[59]: 35–8} However, increased navigation at sea during the Age of Discovery meant that ship’s navigators increasingly had to attempt to determine latitude in unfavorable weather against hostile shores, leading to renewed interest in summary statistics. Whether rediscovered or independently invented, the mid-range is recommended to nautical navigators in Harriot’s «Instructions for Raleigh’s Voyage to Guiana, 1595».^{[59]: 45–8}

The idea of the median may have first appeared in Edward Wright’s 1599 book Certaine Errors in Navigation on a section about compass navigation. Wright was reluctant to discard measured values, and may have felt that the median — incorporating a greater proportion of the dataset than the mid-range — was more likely to be correct. However, Wright did not give examples of his technique’s use, making it hard to verify that he described the modern notion of median.^[56][60][b] The median (in the context of probability) certainly appeared in the correspondence of Christiaan Huygens, but as an example of a statistic that was inappropriate for actuarial practice.^[56]

The earliest recommendation of the median dates to 1757, when Roger Joseph Boscovich developed a regression method based on the L¹ norm and therefore implicitly on the median.^[56][61] In 1774, Laplace made this desire explicit: he suggested the median be used as the standard estimator of the value of a posterior PDF. The specific criterion was to minimize the expected magnitude of the error; where is the estimate and is the true value. To this end, Laplace determined the distributions of both the sample mean and the sample median in the early 1800s.^[26][62] However, a decade later, Gauss and Legendre developed the least squares method, which minimizes to obtain the mean. Within the context of regression, Gauss and Legendre’s innovation offers vastly easier computation. Consequently, Laplaces’ proposal was generally rejected until the rise of computing devices 150 years later (and is still a relatively uncommon algorithm).^[63]

Antoine Augustin Cournot in 1843 was the first^[64] to use the term median (valeur médiane) for the value that divides a probability distribution into two equal halves. Gustav Theodor Fechner used the median (Centralwerth) in sociological and psychological phenomena.^[65] It had earlier been used only in astronomy and related fields. Gustav Fechner popularized the median into the formal analysis of data, although it had been used previously by Laplace,^[65] and the median appeared in a textbook by F. Y. Edgeworth.^[66] Francis Galton used the English term median in 1881,^[67][68] having earlier used the terms middle-most value in 1869, and the medium in 1880.^[69][70]

Statisticians encouraged the use of medians intensely throughout the 19th century for its intuitive clarity and ease of manual computation. However, the notion of median does not lend itself to the theory of higher moments as well as the arithmetic mean does, and is much harder to compute by computer. As a result, the median was steadily supplanted as a notion of generic average by the arithmetic mean during the 20th century.^[56][60]

See also[edit]

• Absolute deviation
• Bias of an estimator
• Central tendency
• Concentration of measure for Lipschitz functions
• Median graph
• Median of medians – Algorithm to calculate the approximate median in linear time
• Median search
• Median slope
• Median voter theory
• Medoids – Generalization of the median in higher dimensions

Notes[edit]

References[edit]

External links[edit]

• «Median (in statistics)», Encyclopedia of Mathematics, EMS Press, 2001 [1994]
• Median as a weighted arithmetic mean of all Sample Observations
• On-line calculator
• Calculating the median
• A problem involving the mean, the median, and the mode.
• Weisstein, Eric W. «Statistical Median». MathWorld.
• Python script for Median computations and income inequality metrics
• Fast Computation of the Median by Successive Binning
• ‘Mean, median, mode and skewness’, A tutorial devised for first-year psychology students at Oxford University, based on a worked example.
• The Complex SAT Math Problem Even the College Board Got Wrong: Andrew Daniels in Popular Mechanics

This article incorporates material from Median of a distribution on PlanetMath, which is licensed under the Creative Commons Attribution/Share-Alike License.

Время прочтения
9 мин

Просмотры 6.7K

Предыдущий пост см. здесь.

Выборки и популяции

В статистической науке термины «выборка» и «популяция» имеют особое значение. Популяция, или генеральная совокупность, — это все множество объектов, которые исследователь хочет понять или в отношении которых сделать выводы. Например, во второй половине 19-го века основоположник генетики Грегор Йохан Мендель) записывал наблюдения о растениях гороха. Несмотря на то, что он изучал в лабораторных условиях вполне конкретные сорта растения, его задача состояла в том, чтобы понять базовые механизмы, лежащие в основе наследственности абсолютно всех возможных сортов гороха.

^{В статистической науке о группе объектов, из которых извлекается выборка, говорят, как о популяции, независимо от того, являются изучаемые объекты живыми существами или нет.}

Поскольку популяция может быть крупной — или бесконечной, как в случае растений гороха Менделя — мы должны изучать репрезентативные выборки, и на их основе делать выводы о всей популяции в целом. В целях проведения четкого различия между поддающимися измерению атрибутами выборок и недоступными атрибутами популяции, мы используем термин статистики, имея при этом в виду атрибуты выборки, и говорим о параметрах, имея в виду атрибуты популяции.

^{Статистики — это атрибуты, которые мы можем измерить на основе выборок. Параметры — это атрибуты популяции, которые мы пытаемся вывести статистически.}

В действительности, статистики и параметры различаются в силу применения разных символов в математических формулах:

Если вы вернетесь к уравнению стандартной ошибки, то заметите, что она вычисляется не из выборочного стандартного отклонения S_x, а из популяционного стандартного отклонения σ_x. Это создает парадоксальную ситуацию — мы не можем вычислить выборочную статистику, используя для этого популяционные параметры, которые мы пытаемся вывести. На практике, однако, предполагается, что выборочное и популяционное стандартные отклонения одинаковы при размере выборки порядка n ≥ 30.

Теперь вычислим стандартную ошибку средних значений за определенный день. Например, возьмем конкретный день, скажем, 1 мая:

def ex_2_8():  
    '''Вычислить стандартную ошибку 
       средних значений за определенный день'''
    may_1 = '2015-05-01'
    df = with_parsed_date( load_data('dwell-times.tsv') )  
    filtered = df.set_index( ['date'] )[may_1]
    se = standard_error( filtered['dwell-time'] )
    print('Стандартная ошибка:', se)

Стандартная ошибка: 3.627340273094217

Хотя мы взяли выборку всего из одного дня, вычисляемая нами стандартная ошибка очень близка к стандартному отклонению всех выборочных средних — 3.6 сек. против 3.7 сек. Это, как если бы, подобно клетке, содержащей ДНК, в каждой выборке была закодирована информация обо всей находящейся внутри нее популяции.

Интервалы уверенности

Поскольку стандартная ошибка выборки измеряет степень близости, с которой, по нашим ожиданиям, выборочное среднее соответствует среднему популяционному, то мы можем также рассмотреть обратное — стандартная ошибка измеряет степень близосто, с которой, по нашим ожиданиям, популяционное среднее соответствует измеренному среднему выборочному. Другими словами, на основе стандартной ошибки мы можем вывести, что популяционное среднее находится в пределах некого ожидаемого диапазона выборочного среднего с некоторой степенью уверенности.

Понятия «степень уверенности» и «ожидаемый диапазон», взятые вместе, дают определение термину интервал уверенности.

Примечание. В большинстве языков под термином «confidence» в контексте инференциальной статистики понимается именно уверенность в отличие от отечественной статистики, где принято говорить о доверительном интервале. На самом деле речь идет не о доверии (trust), а об уверенности исследователя в полученных результатах. Это яркий пример мягкой подмены понятия. Подобного рода ошибки вполне объяснимы — первые переводы появились еще в доколумбову доинтернетовскую эпоху, когда источников было мало, и приходилось домысливать в силу своего понимания. Сегодня же, когда существует масса профильных глоссариев, словарей и источников, такого рода искажения не оправданы. ИМХО.

При установлении интервалов уверенности обычной практикой является задание интервала размером 95% — мы на 95% уверены, что популяционный параметр находится внутри интервала. Разумеется, еще остается 5%-я возможность, что он там не находится.

Какой бы ни была стандартная ошибка, 95% популяционного среднего значения будет находиться между -1.96 и 1.96 стандартных отклонений от выборочного среднего. И, следовательно, число 1.96 является критическим значением для 95%-ого интервала уверенности. Это критическое значение носит название z-значения.

^{Название z-значение вызвано тем, что нормальное распределение называется z-распределением. Впрочем, иногда z-значение так и называют — гауссовым значением.}

Число 1.96 используется так широко, что его стоит запомнить. Впрочем, критическое значение мы можем вычислить сами, воспользовавшись функцией scipy stats.norm.ppf. Приведенная ниже функция confidence_interval ожидает значение для p между 0 и 1. Для нашего 95%-ого интервала уверенности оно будет равно 0.95. В целях вычисления положения каждого из двух хвостов нам нужно вычесть это число из единицы и разделить на 2 (2.5% для интервала уверенности шириной 95%):

def confidence_interval(p, xs):
    '''Интервал уверенности'''
    mu = xs.mean()
    se = standard_error(xs)
    z_crit = stats.norm.ppf(1 - (1-p) / 2)  
    return [mu - z_crit * se, mu + z_crit * se]

def ex_2_9():
    '''Вычислить интервал уверенности
       для данных за определенный день'''
    may_1 = '2015-05-01'
    df    = with_parsed_date( load_data('dwell-times.tsv') )  
    filtered = df.set_index( ['date'] )[may_1]
    ci = confidence_interval(0.95, filtered['dwell-time'])
    print('Интервал уверенности: ', ci)

Интервал уверенности: [83.53415272762004, 97.753065317492741]

Полученный результат говорит о том, что можно на 95% быть уверенным в том, что популяционное среднее находится между 83.53 и 97.75 сек. И действительно, популяционное среднее, которое мы вычислили ранее, вполне укладывается в этот диапазон.

Сравнение выборок

После вирусной маркетинговой кампании веб-команда в AcmeContent извлекает для нас однодневную выборку времени пребывания посетителей на веб-сайте для проведения анализа. Они хотели бы узнать, не привлекла ли их недавняя кампания более активных посетителей веб-сайта. Интервалы уверенности предоставляют нам интуитивно понятный подход к сравнению двух выборок.

Точно так же, как мы делали ранее, мы загружаем значения времени пребывания, полученные в результате маркетинговой кампании, и их резюмируем:

def ex_2_10():
    '''Сводные статистики данных, полученных 
       в результате вирусной кампании'''
    ts = load_data('campaign-sample.tsv')['dwell-time']     
    print('n:                      ', ts.count())
    print('Среднее:                ', ts.mean())
    print('Медиана:                ', ts.median())
    print('Стандартное отклонение: ', ts.std())
    print('Стандартная ошибка:     ', standard_error(ts))    

ex_2_10()

n:                       300
Среднее:                 130.22
Медиана:                 84.0
Стандартное отклонение:  136.13370714388034
Стандартная ошибка:      7.846572839994115

Среднее значение выглядит намного больше, чем то, которое мы видели ранее — 130 сек. по сравнению с 90 сек. Вполне возможно, здесь имеется некое значимое расхождение, хотя стандартная ошибка более чем в 2 раза больше той, которая была в предыдущей однодневной выборке, в силу меньшего размера выборки и большего стандартного отклонения. Основываясь на этих данных, можно вычислить 95%-й интервал уверенности для популяционного среднего, воспользовавшись для этого той же самой функцией confidence_interval, что и прежде:

def ex_2_11():
    '''Интервал уверенности для данных,
       полученных в результате вирусной кампании'''
    ts = load_data('campaign-sample.tsv')['dwell-time']     
    print('Интервал уверенности:', confidence_interval(0.95, ts))

Интервал уверенности: [114.84099983154137, 145.59900016845864]

95%-ый интервал уверенности для популяционного среднего лежит между 114.8 и 145.6 сек. Он вообще не пересекается с вычисленным нами ранее популяционным средним в размере 90 сек. Похоже, имеется какое-то крупное расхождение с опорной популяцией, которое едва бы произошло по причине одной лишь ошибки выборочного обследования. Наша задача теперь состоит в том, чтобы выяснить почему это происходит.

^{Ошибка выборочного обследования, также систематическая ошибка при взятии выборки, возникает, когда статистические характеристики популяции оцениваются исходя из подмножества этой популяции.}

Искаженность

Выборка должна быть репрезентативной, то есть представлять популяцию, из которой она взята. Другими словами, при взятии выборки необходимо избегать искажения, которое происходит в результате того, что отдельно взятые члены популяции систематически из нее исключаются (либо в нее включаются) по сравнению с другими.

Широко известным примером искажения при взятии выборки является опрос населения, проведенный в США еженедельным журналом «Литературный Дайджест» (Literary Digest) по поводу президентских выборов 1936 г. Это был один из самых больших и самых дорогостоящих когда-либо проводившихся опросов: тогда по почте было опрошено 2.4 млн. человек. Результаты были однозначными — губернатор-республиканец от шт. Канзас Альфред Лэндон должен был победить Франклина Д. Рузвельта с 57% голосов. Как известно, в конечном счете на выборах победил Рузвельт с 62% голосов.

Первопричина допущенной журналом огромной ошибки выборочного обследования состояла в искажении при отборе. В своем стремлении собрать как можно больше адресов избирателей журнал «Литературный Дайджест» буквально выскреб все телефонные справочники, подписные перечни журнала и списки членов клубов. В эру, когда телефоны все еще во многом оставались предметом роскоши, такая процедура гарантировано имела избыточный вес в пользу избирателей, принадлежавших к верхнему и среднему классам, и не был представительным для электората в целом. Вторичной причиной искажения стала искаженность в результате неответов — в опросе фактически согласились участвовать всего менее четверти тех, к кому обратились. В этом виде искаженности при отборе предпочтение отдается только тем респондентам, которые действительно желают принять участие в голосовании.

Распространенный способ избежать искаженности при отборе состоит в том, чтобы каким-либо образом рандомизировать выборку. Введение в процедуру случайности делает вряд ли возможным, что экспериментальные факторы окажут неправомерное влияние на качество выборки. Опрос населения еженедельником «Литературный Дайджест» был сосредоточен на получении максимально возможной выборки, однако неискаженная малая выборка была бы намного полезнее, чем плохо отобранная большая выборка.

Если мы откроем файл campaign_sample.tsv, то обнаружим, что наша выборка приходится исключительно на 6 июня 2015 года. Это был выходной день, и этот факт мы можем легко подтвердить при помощи функции pandas:

'''Проверка даты''' 
d = pd.to_datetime('2015 6 6') 
d.weekday() in [5,6]

True

Все наши сводные статистики до сих пор основывались на данных, которые мы отфильтровывали для получения только рабочих дней. Искаженность в нашей выборке вызвана именно этим фактом, и, если окажется, что поведение посетителей в выходные отличается от поведения в будние дни — вполне возможный сценарий — тогда мы скажем, что выборки представляют две разные популяции.

Визуализация разных популяций

Теперь снимем фильтр для рабочих дней и построим график среднесуточного времени пребывания для всех дней недели — рабочих и выходных:

def ex_2_12():  
    '''Построить график времени ожидания 
       по всем дням, без фильтра'''
    df = load_data('dwell-times.tsv')
    means = mean_dwell_times_by_date(df)['dwell-time']
    means.hist(bins=20)
    plt.xlabel('Ежедневное время ожидания неотфильтрованное, сек.')
    plt.ylabel('Частота')
    plt.show()

Этот пример сгенерирует следующую ниже гистограмму:

Распределение уже не является нормальным. Оно фактически является бимодальным, поскольку у него два пика. Второй меньший пик соответствует вновь добавленным выходным дням, и он ниже потому что количество выходных дней гораздо меньше количества рабочих дней, а также потому что стандартная ошибка распределения больше.

^{Распределения более чем с одним пиком обычно называются мультимодальными. Они могут указывать на совмещение двух или более нормальных распределений, и, следовательно, возможно, на совмещение двух или более популяций. Классическим примером бимодальности является распределение показателей роста людей, поскольку модальный рост мужчин выше, чем у женщин.}

Данные выходных дней имеют другие характеристики в отличие от данных будних дней. Мы должны удостовериться, что мы сравниваем подобное с подобным. Отфильтруем наш первоначальный набор данных только по выходным дням:

def ex_2_13():
    '''Сводные статистики данных,
       отфильтрованных только по выходным дням'''
    df = with_parsed_date( load_data('dwell-times.tsv') )
    df.index = df['date']
    df = df[df['date'].index.dayofweek > 4]   # суббота-воскресенье
    weekend_times = df['dwell-time']
  
    print('n:                      ', weekend_times.count())
    print('Среднее:                ', weekend_times.mean())
    print('Медиана:                ', weekend_times.median())
    print('Стандартное отклонение: ', weekend_times.std())
    print('Стандартная ошибка:     ', standard_error(weekend_times))        

n:                       5860
Среднее:                 117.78686006825939
Медиана:                 81.0
Стандартное отклонение:  120.65234077179436
Стандартная ошибка:      1.5759770362547678

Итоговое среднее значение в выходные дни (на основе 6-ти месячных данных) составляет 117.8 сек. и попадает в пределы 95%-ого интервала уверенности для маркетинговой выборки. Другими словами, хотя среднее значение времени пребывания в размере 130 сек. является высоким даже для выходных, расхождение не настолько большое, что его нельзя было бы приписать простой случайной изменчивости в выборке.

Мы только что применили подход к установлению подлинного расхождения в популяциях (между посетителями веб-сайта в выходные по сравнению с посетителями в будние дни), который при проведении проверки обычно не используется. Более традиционный подход начинается с выдвижения гипотетического предположения, после чего это предположение сверяется с данными. Для этих целей статистический метод анализа определяет строгий подход, который называется проверкой статистических гипотез.

Это и будет темой следующего поста, поста №3.

Примеры исходного кода для этого поста находятся в моем репо на Github. Все исходные данные взяты в репозитории автора книги.

In other sections of this guide on descriptive statistics, we taught you the basics of finding mean, median, standard deviation and percentiles. We provided you the formulas to each, as well as introduced some intermediate topics involving these measures of central tendency and variability. Here, we’ll present some advanced topics relating to these measures, namely the standard error and percentile rank.

Measures of Central Tendency and Variability

While measures like mean, median, standard deviation and percentiles can seem somewhat basic, the truth is that they are some of the most powerful tools of analysis within descriptive statistics and which form many of the fundamental concepts in more advanced, inferential statistics.

Percentiles, for example, is a concept that is used in quantile regression — which strives to identify patterns within different quantiles of a data set. These advanced applications rely on a strong, fundamental understanding of these more basic concepts. Below, you’ll find a brief recap of the formulas for sample mean, median and standard deviation as well as when they’re used and how to interpret them.

	Formula	Uses	Interpretation
Mean		Weighted mean Good representation of most typical value in data sets without extreme outliers	The average of the data set, capturing where most values are cantered
Median	No standard formula, is found by identifying the middle value of ordered data	Interquartile Range Good representation of the middle value when there’s outliers	The midpoint of the data, representing the point where half the data falls above and below
SD		Z-scores Variety of statistical tests on data	Is used to determine how typical a value is for a data set with a given mean and standard deviation

Standard Error

The concept of the standard deviation is a great starting point for understanding the standard error. It is defined as the estimate of the standard deviation of an estimate. Thus far, you’ve been calculating the standard deviation of an entire sample. Meaning, you’ve calculated a measure by which you can identify how likely any value would appear given the mean and sample size of your data set.

The standard deviation typically goes hand in hand with the distribution of a variable. Every distribution you will encounter reflects a probability density function, where each point on the horizontal axis corresponds with the probability of it occurring in a given data set on the vertical axis.

The question the standard deviation of a variable or data set attempts to answer is whether a given value is likely to be found for a given distribution. The question the standard error tries to answer is whether how likely a given statistic, such as the mean, is to be found for a given data set.

Instead of trying to find the distribution of a data set, the standard error tries to find the distribution of a given statistic. Recall the difference between a sample and a population. Measures of a population are called parameters, where the standard deviation of a parameter gives us information about the distribution of that parameter.

Measures of a sample, however, are estimates of parameters and are called statistics. The standard error, then, is an estimate of the standard deviation of a statistic.

If this is confusing, take the data sets below as an example. Each measure the test scores of students at different schools for the same test.

Observation	School A	School B	School C
1	45	29	66
2	64	54	57
3	38	59	58
4	70	67	52
5	62	68	32
Mean	56	55	53

Taking these three sample means, we can make an informed guess that the mean test score for the region is around 55 points. If we had information for ten more schools, then 100 more schools, and finally for all the schools in the region — we could start to get closer and closer to some middle value that may be higher or lower than 55 points.

Because we don’t always have the chance to take infinitely many samples, a probability distribution of means is a great way of estimating how likely a given mean is for a particular data set. In the table below, you’ll find the formulas for the standard deviation and the standard error of the mean for a population and sample, respectively.

Population	Sample

Problem 1

Given the following dataset, what can you say about the accuracy of the average of the data?

Observation	Value
1	45
2	67
3	28
4	32
5	29
6	46
7	61
8	58
9	49
10	36
11	34

Problem 2

Given the following information, determine which data set has a lower standard error of mean.

Observation	School A	School B	School C
Mean	56	55	53
SD	4	10	8
Sample Size	9	1 000	250

Problem 3

Given the following chart, determine one of  the measures of central tendency.

Problem 4

Given the following chart, choose which measure of central tendency would be most appropriate.

Solution Problem 1

In this problem, you were asked to:

• Say something about the accuracy of the mean

In this case, we first find the mean by following the steps below.

Observation	Value
1	45
2	67
3	28
4	32
5	29
6	46
7	61
8	58
9	49
10	36
11	34
Total	485

The mean is calculated as

   

To find the accuracy of the mean, we need to calculate the standard error of the mean by first finding the standard deviation.

   

Then, we plug the SD into the formula for standard error.

   

Because the standard error is relatively small when compared to the dataset, this suggests the mean is pretty accurate.

Solution Problem 2

You were asked to determine which data set has a lower standard error of mean. Using the information given, you can calculate the standard error for each sample.

Observation	School A	School B	School C
SE

Based on the calculations performed in the table above, the sample with the lowest standard error of the mean is School B.

Solution Problem 3

The only measure of central tendency we can get from this chart is the mode, which is Chemistry.

Solution Problem 4

Because there are extreme values between 107 and 119, either the mode or the median would be most appropriate depending on what we’d like to investigate.

Estimate Standard Error of Median Using the Bootstrap Strategy

getting the Empirical Distribution through sampling with replacement

In this article, I will discuss the method we can use to estimate the standard error of the median. I will first specify some definitions and discuss how we can estimate the standard error of the mean. Then I will discuss how to estimate the standard error of the medium with the bootstrap strategy.

What is the standard error?

According to Wikipedia, the standard error of a statistic is the standard deviation of its sampling distribution or an estimate of that standard deviation. There are several concepts in this sentence that needs to be clarified before move forward:

First, a statistic is a sampling estimation of a parameter. For example, the sample mean x_bar is a statistic, while the population mean µ is a parameter. Since µ is usually unknown, we need a statistic to estimate it. x_bar is considered as an unbiased estimation of µ.

Second, the standard deviation is a measurement of dispersion, and it is the square root of variance. It usually stands for the confidence of your estimation and is used in the confidence interval, hypothesis testing, etc.

Last, a sampling distribution is the probability distribution of a statistic from random samples. Let’s take an example. Suppose we are estimating the average income of college graduates in the US. The population is the income of all graduates from college. It is almost impossible to collect the data from all graduates and estimate their incomes. That is why we are taking represented samples from the population and calculate the sample mean to estimate the population mean µ.

What is the standard error of the mean (SEM)?

Estimating the standard error of the mean is straightforward thanks to the Central Limit Theory. The theory stated that no matter what is the distribution of the population, the sampling distribution of the sample mean approaches a normal distribution as the sample size gets larger. In addition, the normal distribution has the mean equals to the population mean, and the standard deviation equals to population standard deviation divided by the square root of sample size n. Thus, the standard error of the mean is the sample standard deviation divided by the square root of sample size n, if the population standard deviation is unknown. Given any random distribution, we will estimate SEM easily.

How to estimate the standard error of the median?

Unfortunately, the Central Limit Theory does not apply to the median, thus we need other methods to estimate the standard error of the median given only one sample. That is when the bootstrap strategy comes in handy.

The bootstrap is sampling with replacement. If there is a sample:

we can take n items from this sample with replacement. Here, we will generate a new sample, it can be:

with a sample size equals to n. Using the bootstrap strategy to estimate standard error follows the steps as stated below:

1, take n items from the given sample as a new sample:

from this sample, we can easily calculate the sample median:

2, repeat the previous procedure for B rounds, and we will get B new samples, with B sample medians:

3, now we have got an empirical distribution of medians, thus we can estimate the standard error of medians from them:

(1) calculate the mean of the previous sample of medians:

(2) calculate the variation:

(2) calculate the standard error, which is the standard deviation of the empirical sample:

Why is bootstrap working?

The reason bootstrap works is that bootstrap uses the Empirical Distribution Function (EDF) to estimate the CDF of the population. A statistic of a sample is a function of a sample randomly taken from the population. Thus its distribution will depend on the CDF of the population and the sample size. While the CDF of a population is usually unknown, we can estimate it using EDF, thus get the sample statistic distribution from it. We can find here the detailed mathematical explanation, along with some other restrictions.

Hope this helps! Thank you for reading!

---

How to estimate the standard error of the median: The Bootstrap Strategy was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.