4.2 The Maximum Likelihood Method

The method of maximum likelihood is only applicable if the form of the theoretical distribution from which the sample is taken is known. For most measurements in physics, this is either the Gaussian or Poisson distribution. But, to be more general, suppose we have a sample of n independent observations x₁, x₂, . . . ,x_n, from a theoretical distribution f(x | ) where is the parameter to be estimated. The method then consists of calculating the likelihood function,

(29)

which can be recognized as the probability for observing the sequence of values x₁, x₂, . . ., x_n. The principle now states that this probability is a maximum for the observed values. Thus, the parameter must be such that L is a maximum. If L is a regular function, can be found by solving the equation,

(30)

If there is more than one parameter, then the partial derivatives of L with respect to each parameter must be taken to obtain a system of equations. Depending on the form of L, it may also be easier to maximize the logarithm of L rather than L itself. Solving the equation

(31)

then yields results equivalent to (30). The solution, , is known as the maximum likelihood estimator for the parameter . In order to distinguish the estimated value from the true value, we have used a caret over the parameter to signify it as the estimator.

It should be realized now that is also a random variable, since it is a function of the x_i. If a second sample is taken, will have a different value and so on. The estimator is thus also described by a probability distribution. This leads us to the second half of the estimation problem: What is the error on ? This is given by the standard deviation of the estimator distribution We can calculate this from the likelihood function if we recall that L is just the probability for observing the sampled values x₁, x₂,. . ., x_n. Since these values are used to calculate , L is related to the distribution for . Using (9), the variance is then

(32)

This is a general formula, but, unfortunately, only in a few simple cases can an analytic result be obtained. An easier, but only approximate method which works in the limit of large numbers, is to calculate the inverse second derivative of the log-likelihood function evaluated at the maximum,

(33)

If there is more than one parameter, the matrix of the second derivatives must be formed, i.e.,

(34)

The diagonal elements of the inverse matrix then give the approximate variances,

(35)

A technical point which must be noted is that we have assumed that the mean value of is the theoretical . This is a desirable, but not essential property for an estimator, guaranteed by the maximum likelihood method only for infinite n. Estimators which have this property are non-biased. We will see one example in the following sections in which this is not the case. Equation (32), nevertheless, remains valid for all , since the error desired is the deviation from the true mean irrespective of the bias.

Another useful property of maximum likelihood estimators is invariance under transformations. If u = f(), then the best estimate of u can be shown to be = f().

Let us illustrate the method now by applying it to the Poisson and Gaussian distributions.