Model Diagnostics

Several classes and/or routines are available to help with examining and diagnosing model fits.

class sails.diags.ModelDiagnostics

Class to store and display LinearModel diagnostic information. Several diagnostic criteria are computed including:

R_square (R_square)

the percentage of variance explained in each channel

Stability Index (SI)

Indicator of the stability of the MVAR parameters (SI<1 indicates a stable model)

Stability Ratio (SR)

A stronger test of stability computed from the ratio of the largest eigenvalue of A to all others.

Durbin-Watson (DW)

A test of autocorrelation in residuals of the model fit. Values should be close to 2 indicating no autocorrelation, values close to 0 indicate a positive autocorrelation and 4 and negative autocorrelation.

Log Likelihood (LL)

The log-likelihood of the model.

Akaike’s Information Criterion (AIC)

An indication of the model ‘quality’, lower values indicate a more accurate, less complex model.

Bayesian Information Criterion (BIC)

An indication of the model ‘quality’, lower values indicate a more accurate, less complex model.

Percent Consistency (PC)

Indicates how well a model captures the auto and cross correlation in a time-series. Only computed if compute_pc is passed to the relevant function.

classmethod combine_diag_list(diags)

Helper function for combining diagnostics from a list of ModelDiagnostics instances for easy comparison and visualisation.

Parameters

diags (list of ModelDiagnostics instances) – The ModelDiagnostics to concatenate

Returns

Return type

sails ModelDiagnostics instance

classmethod compute(model, data, compute_pc=False)

Classmethod for computing a set of model diagnostics from a fitted model applied to a time-series dataset.

Parameters

• model (sails LinearModel class) – A fitted linear model

• data (ndarray) – A 3d time-series of size [nchannels x nsamples x ntrials]

• compute_pc (bool) – Flag indicating whether to compute the percent consistency, this can be time-consuming for large datasets (Default=False).

Returns

Return type

sails ModelDiagnostics instance

summary(all_models=True)

Print the ModelDiagnostics in a pre-formatted table

class sails.diags.DelayDiagnostics

Class which computes the mutual information as a function of lag from zero lag to the first zero crossing of the autocorrelation function.

classmethod delay_search(data, maxdelay, step, sample_rate, constant_window=True)

Compute MI as a function of lag from zero lag to the first zero crossing of the autocorrelation function

Parameters

• data (numpy.ndarray) – array of [signals, samples, trials]

• maxdelay (int) – maximum delay to consider

• step (int) – step to increment the delay by

• sample_rate (float) – sample rate of data

• constant_window (bool) – Flag indicating that the same number of datapoints should be included at each delay. Default is True

Returns

Populated object containing diagnostics for each value in delay

Return type

DelayDiagnostics

sails.modelfit.get_residuals(data, parameters, delay_vect, backwards=False, mode='valid')

This is a helper function for computing the residuals of a dataset after the MVAR predictions have been removed.

Parameters

• data (ndarray) – Data to compute the residuals from, of size [nchannels x nsampes x nrealisations]

• parameters (ndarray) – MVAR parameter matrix, of size [nchannels x nchannels x model order]

• delay_vect (ndarray) – Vector of lag indices corresponding to the third dimension of the parameter matrix

• backwards (bool) – Flag indicating whether the forwards or backwards parameters havebeeen passed (Default value = False)

• mode ({'valid','full_nan','full'}) – Options for excluding or replacing residuals which do not have a full model prediction ie the third sample of an order 5 model. ‘valid’ removes samples without a full model prediction, ‘full_nan’ returns resids of the same size as the data with nans replacing excluded samples and ‘full’ returns resids keeping non-full model samples in place.

Returns

Residual data, of size [nchannels x nsamples x nrealisations]

Return type

ndarray

Model Fitting Model Decomposition