evaluation package

Base Classes

Model Evaluator

class autompc.evaluation.ModelEvaluator(system, trajs, metric, rng, horizon=1)

The ModelEvaluator class evaluates models by prediction accuracy.

__init__(system, trajs, metric, rng, horizon=1)

Parameters:

• system (System) – System for which prediction accuracy is evaluated

• trajs (List of Trajectory) – Trajectories to be used for evaluation

• metric (string or function (model, [Trajectory] -> float)) – Metric which evaluates the model against a set of trajectories. If string, one of “rmse”, “rmsmens”. See model_metrics for more details.

• rng (np.random.Generator) – Random number generator used in evaluation

• horizon (int) – Prediction horizon used in certain metrics. Default is 1.

abstract __call__(model_factory, configuration)

Accepts the model class and the configuration.

Parameters:

• model_factory (ModelFactory) – Model factory evaluate

• configuration (Configuration) – Hyperparameter configuration used to instantiate model

Returns:

score – Evaluated score.

Return type:

float

Evaluator Types

HoldoutModelEvaluator

class autompc.evaluation.HoldoutModelEvaluator(*args, holdout_prop=0.1, holdout_set=None, verbose=False, **kwargs)

Evaluate model prediction accuracy according to a holdout set.

__init__(*args, holdout_prop=0.1, holdout_set=None, verbose=False, **kwargs)

Parameters:

• system (System) – System for which prediction accuracy is evaluated

• trajs (List of Trajectory) – Trajectories to be used for evaluation

• metric (string or function (model, [Trajectory] -> float)) – Metric which evaluates the model against a set of trajectories. If string, one of “rmse”, “rmsmens”. See model_metrics for more details.

• rng (np.random.Generator) – Random number generator used in evaluation

• horizon (int) – Prediction horizon used in certain metrics. Default is 1.

• holdout_prop (float) – Proportion of dataset to hold out for evaluation

• holdout_set (List of Trajectory) – This argument can be passed to explicitly set holdout set, rather than randomly selecting it. Pass None otherwise.

• verbose (bool) – Whether to print information during evaluation

Model Metrics

autompc.evaluation.model_metrics.get_model_rmse(model, trajs, horizon=1)

Computes (unnormalized) RMSE at fixed horizon

Parameters:

• model (Model) – Model class to consider

• trajs (List of Trajectories) – Trajectories on which to evaluate

• horizon (int) – Prediction horizon at which to evaluate. Default is 1.

autompc.evaluation.model_metrics.get_model_rmsmens(model, trajs, horiz=1)

Compute root mean squared model error, normalized step-wise (RMSMENS).

Given a set of trajectories \(\mathcal{T}\), let \(x_{i,t} \in \mathbb{R}^n, u_{i,t} \in \mathbb{R}^m\) denote the state and control input in the \(i^\textrm{th}\) trajectory at time \(t\). Let \(L_i\) denote the length of the \(i\textrm{th}\) trajectory. Given a predictive model \(g\), let \(g(i,t,k)\) give the prediction for \(x_{i,t+k}\) given the states \(\mathbf{x}_{i,1:t}\) and controls \(\mathbf{u}_{i,1:t+k-1}\). Let

\[\sigma = \textrm{std} \{ x_{i,t+1} - x_{i,t} \mid i=1,\ldots,\left|\mathcal{T}\right| t=1,\ldots,L_i-1 \} \textrm{,}\]

where the mean and standard deviation are computed element-wise.

Denote the \(k\)-step Root Mean Squared Model Error, Normalized Step-wise, RMSMENS(\(k\)), of \(g\) with respect to \(\mathcal{T}\) as

\[\sqrt{ \frac{1}{n} \left\lVert \frac{1}{\left|\mathcal{T}\right|} \sum_{i=1}^{\left|\mathcal{T}\right|} \frac{1}{L_i-k } \sum_{t=1}^{L_i-k}\frac{1}{\sigma^2} e(i,t,k)^2 \right\rVert_1 } \textrm{,}\]

where

\[e(i,t,k) = g(i,t,k) - g(i,t,k-1) - (x_{i,t+k} - x_{i,t+k-1})\]

Parameters:

• model (Model) – Model class to consider

• trajs (List of Trajectories) – Trajectories on which to evaluate

• horizon (int) – Prediction horizon at which to evaluate. Default is 1.