Uncertainity estimatation
Uncertainties of the free parameters can be estimated either by sampling the posterior distribution of the model or by support plane analysis. ChiSurf uses emcee to sample the posterior model parameter distribution emcee. The support plane analysis is performed by varying the parameter if interest in a specified range on a linear grid, fixing the parameter to the value at the grid point, and optimizing all other parameters to yield a quality estimate of the model for the given fixed parameter that is scanned.
Sampling the full posterior distribution can be computationally expensive and is time consuming. Moreover, the convergence of the sampled distribution needs to be assessed. Hence, as a first step to estimated parameter uncertainties it is recommended to performed a support plane analysis.
After either a support plane analysis or a full sampling of the posterior distribution was performed statistical test, like an F-test, can be used to obtain an estimate for the parameter uncertainties at for a given confidence level.
Briefly, a support plane analysis of a free parameter of can is performed as follows.
- Create a fit
- Optimize the fit
- Change your view the fitting window to the parameter scan plot
- Change from the 'Analysis' to the 'Plot' panel
- Update the parameter list by clicking on update
- Select the parameter of interest
- Define your scanning region
- Click on scan
- Analyze your parameter scan
In detail, first a fit must be created and the fit must be optimized. The optimized parameters are used for the subsequent iterative fits of the support plane analysis. Hence, make sure that your initial fit describes your data sufficiently well. If this is not the case, the support plane analysis may produce inconsistent results.
Visually inspect the residuals, to make sure that your model describes your data.
Next, change from the 'Analysis' dock widget to the 'Plot'. The 'Plot' dock widget allows you to control options of the currently displayed plot. The plot can be changed on the lower bar of the fitting window.
While your work with your model the free parameters may change, the plot widget of the parameter scan only displays the free parameters of the model. Thus, before doing a parameter scan, the displayed parameters should be updated by clicking on the update button.
Next, decide on the your parameter of interest and select the parameter in the list next to the update button.
To define the scan range of the parameter modify the values displayed in the boxed labeled scan range. The first value defines the lower limit of the scan. The second value defines the upper limit of the scan. The third value defines the number of steps between the first and the lower and the upper limit. The lower and upper limits are given as fractions of the current value. A lower limit of 0.1 means that the lower limit is 0.9 times the current value of the model parameter. An upper limit of 0.1 means that the upper limit is 1.1 of the current parameter value. The current parameter value is the value that is currently displayed for the parameter of interest. The parameters for the support plane analysis are linearly spaced between the lower and the upper limit.
Finally, by clicking on the "scan" button, the parameter is varied in the defined range. At each point, the parameter is fixed and the other free parameters are optimized. You can inspect the fit quality as a function of the parameter value in the "Parameter scan" plot. If necessary, change the scan range to more accurate results.
A good support plane analysis should have a defined minimum. The maximum possible chi2 value for a given fit that is still acceptable at a confidence level can be estimated by an F-test. For an F-test, the effective degrees of freedom need to be known. The degrees of freedom are defined by the number of statistically independent observable and the number of model parameters. A tool for an F-test is available in the menu bar (menu bar->Tools->F-test).
Clicking twice on the ">" button in the F-test tool opens a list of the currently existing fit windows in ChiSurf. Selecting a fit of interest (here the fit of the parameter scan updates the values in the F-Test tool, i.e., the number of model parameters (para), the number of degrees of freedom (dof), and the minimum chi2 of the fit. These parameters can also be modified manually. The maximum chi2 (chi2 max) is updated when these parameters are changed.
Finally, using the maxim chi2 for a confidence level, upper and lower limits for an uncertainty estimation can be defined.
The intercept of the maximum possible chi2 and the parameter scan defines an uncertainty estimates.