Annu. Rev. Astron. Astrophys. 2011. 49:409-470 Copyright © 2011 by Annual Reviews. All rights reserved

7. MODELING CONSIDERATIONS

With such significant improvements in data quality expected and such profound questions to be addressed, modeling considerations will become increasingly important. The key issues will be modeling and mitigating all important sources of bias and systematic error in the analyses, and using the information efficiently.

For tests based on the mass function and clustering of clusters and their associated mass–observable scaling relations, the impact of survey biases must be accounted for (Section 2.5.1). Typically this requires simultaneous modeling of the cluster population and scaling relations in a single likelihood function. Such an approach also facilitates understanding the covariance between model parameters, and provides a structure within which to examine the impact of residual systematic uncertainties which often correlate with model parameters. To a large degree, the statistical frameworks required for such analyses have been developed (Section 4.1.1) and their application to existing data are discussed in this review.

The application of judicious, blind (see below) cuts can improve the balance of statistical versus systematic uncertainties. For example, with the f_gas test (Section 4.2), cosmological constraints are best derived from the most massive, dynamically relaxed clusters, which can be identified easily from short, snapshot Chandra or XMM-Newton X-ray observations. These clusters are also ideal targets for the XSZ test (Section 4.2). For the mass function and clustering tests, one must determine the optimal mass/flux/redshift range over which to compare the data and models, given the characteristics of the survey and follow-up data and one's understanding of mass–observable scaling relations.

Strong priors should be used with caution and their implications understood. In cluster cosmology, common assumptions include log-normal scatter in the mass–observable scaling relations, and negligible evolution of this scatter with redshift. When measuring masses from X-ray or optical dynamical data, the use of parameterized temperature, velocity and gas density models is also common. Where such strong assumptions are not necessary, they should be avoided. Where priors do become necessary, for example in parameterizing the impacts of known astrophysical effects, the validity of these assumptions should be checked empirically, where possible.

The covariance between different measured quantities is an important consideration that is often wrongly neglected. The simplest example of this is simply the statistical correlation of quantities measured from the same observation (e.g. due to Poisson noise). A more subtle case follows from the definition of cluster radius for scaling relations in terms of the mean density enclosed: because the measured mass and radius covary, other quantities measured within that radius necessarily covary with mass, even if measured from independent observations. Fortunately, within the framework of Bayesian analysis, it is straightforward to account for such measurement correlations. For example, given a set of scaling relation parameters to be tested, the data likelihood can be integrated over all possible true values of, e.g., the mass and temperature, with one of the terms in the integrand being the probability of the true mass and temperature values producing the observed values (the sampling distribution; see Section 4.1.1). The form of this probability density can be as general as is required – a multidimensional gaussian or log-gaussian in the simplest case – and in particular may have non-zero correlation. Kelly (2007) (see also Gelman et al. 2004) discusses this general approach in the astrophysical context, and provides useful tools for Bayesian linear regression.

In other situations, unnecessary covariance can be introduced by the model applied to the data. An example is the historically common practice of fitting parametric temperature and gas density profiles to X-ray data, and inferring overdensity radii and hydrostatic masses from these model profiles. Here, the procedure introduces an artificial covariance between mass and temperature (i.e. a prior on the scaling relation) which could be avoided simply by modeling the temperature and gravitating mass profiles independently.

Where binning will result in a loss of relevant information, it should be avoided. For example, the binning of X-ray surface brightness data to determine an integrated X-ray luminosity results in a significant loss of information; the center-excised luminosity provides a tighter mass proxy (Section 6.4).

Hypothesis testing will remain a critical element of future analyses. Wherever a model is fitted to data, one should check that it provides an adequate description (i.e. that the goodness of fit is acceptable); if it doesn't, then the model can be ruled out. This is particularly important in the context of cosmological surveys and scaling relations that are subject to selection bias, since no straightforward, visual check of the goodness of fit is typically possible. Where the simplest models fail to describe the data, one must evaluate carefully the additional degrees of freedom needed to alleviate the tension, in particular considering both astrophysical and cosmological possibilities; simulations will play a critical role in motivating and validating alternatives to the simplest scaling models. A related situation arises in the combination of constraints from independent experiments: before combining, one should ensure that the model in question provides an adequate description of the data sets individually, and that the parameter values are mutually consistent, i.e. that their multi-dimensional confidence contours overlap. Here it can be helpful to include the impact of all known systematics in the contours; where contours do not overlap, the model is incomplete and/or unidentified systematic errors are present. Historically, the combination of mutually inconsistent data sets has sometimes led to unphysically tight formal constraints.

Training sets can be useful to tune analyses. However, the biases introduced by this training must be understood and accounted for. The inclusion of sufficient redundancy into experiments (i.e. having more than one independent way to make a measurement) can also enhance significantly the robustness of conclusions. In cosmological studies, for example, we require more than a single way to measure both the growth and expansion histories, e.g. clusters and lensing, and SNIa and BAO, respectively.

A final modeling consideration is the impact of experimenter's bias: the subjective, subconscious bias towards a result expected by an experimenter. This is important where we are seeking to measure a quantity for which prior expectations exist, e.g. w or the growth index, γ. Evidence for experimenter's bias can be found readily in the literature, e.g. in historical measurements of the speed of light or the neutron lifetime (see Klein & Roodman 2005 for a review). The best approach to overcoming experimenter's bias is blind analysis.

A range of blind analysis techniques are commonly employed in the particle and nuclear physics communities. These are used to mask the ability of scientists to determine quantities of interest until all methods are finalized, the systematic errors that can be identified have been, and the final measurement is ready to be made. A technique with good applicability to cosmological studies is hiding the answer, wherein a fixed (unknown to the experimenter) offset is added to the parameter(s) of interest, and only revealed once the final measurements are made. Before un-blinding, it is advisable to think through how to proceed afterward, and what additional checks to employ. Double-blind analyses, where two independent teams repeat the same process (often used in biomedical research) target additional sources of error. Such techniques can be used powerfully, where resources allow.