The above discussion concerned analysis of data which was in hand. Another important application of statistical techniques is to forecast the outcomes of future experiments, which is increasingly expected by funding agencies wishing to compare the capabilities of competing proposals. A Figure of Merit (FoM) is defined for each experiment, and used to rank experiments. More ambitiously, the same techniques can be used to optimize the design of a survey in order to maximize the likely science return (e.g. Ref. [43]).
The leading approach presently is the Fisher information matrix approach, introduced to cosmology in Ref. [44] and popularized especially when adopted by the initial report of the Dark Energy Task Force [45]. The Fisher matrix measures the second derivatives of the likelihood around its maximum, and in a gaussian approximation is used to estimate the expected uncertainty in parameters around some selected fiducial model. For instance, the DETF FoM considers the uncertainty on the two parameters of the dark energy equation of state defined by the form w = w0 + (1 − a)wa, taking the fiducial model to be ΛCDM (w0 = −1, wa = 0), defining the FoM to be the inverse area of the 95% confidence region. The likelihood is determined via a model of how well the instrument will perform, possibly through analysis of a simulated data stream.
An important caveat is the following: an experiment capable of reducing the volume of permitted parameter space by say a factor of 10 should in no way be considered as having a 90% chance of measuring those parameters as different from some special value. An example is whether upcoming data can exclude the cosmological constant model in favour of dark energy. What the Fisher FoM shows is the reduction in allowed parameter volume provided that the dark energy model is correct. However that assumption is actually what we wish to test; it is perfectly possible that it is the cosmological constant model which is correct and then no amount of improvement to the uncertainty will rule it out. Once again, to develop a full picture we need to consider multiple models, and hence model selection statistics.
5.2. Model selection approaches
Model selection forecasting, pioneered for cosmology in Refs. [36, 46], instead forecasts the ability of upcoming experiments to carry out model comparisons. Let's restrict to the simplest case of two models, one nested within another, though the generalization to other circumstances is straightforward.
The very simplest model selection question is to assume that the nested model (say ΛCDM) is true, and ask whether a given experiment would be able to rule out the more complex alternative. Data is simulated only for the ΛCDM model, as with the Fisher analysis, and the Bayes factor between ΛCDM and the parametrized dark energy model calculated. This shows how strongly the experiment will rule out the dark energy model if it is wrong.
More generally, one may wish to consider the dark energy model as the correct one, and ask whether the simpler model can be excluded. This is more complex, as the outcome depends on the actual parameters of the dark energy model (known as the fiducial parameters), and hence has to be considered as a function of them. This leads to the concept of the Bayes factor plot, showing the expected Bayes factor as a function of fiducial parameters [46]. A suitable FoM may be to minimize the area in fiducial parameter space in which the wrong model cannot be ruled out by the proposed experiment. Alternatively, one can study the distribution of the Bayes factor weighted by present knowledge of the parameters, to predict the probability distribution of expected outcomes of the experiment [36].
Examples of cosmological model selection forecasts can be found in Refs. [36, 46, 38].