• “Accuracy proxy”: Matthew's correlation coefficient (MCC) is a relatively general measure of “accuracy” of a predictive model that can give insights about how much we can rely on the model. Also, it can be helpful for detecting regimes, where our model is unreliable.

• Sharpe ratios (annualized, probabilistic, deflated): basic performance metrics that, however, have to be fixed for fat-tail and skewed distributions and multiple testing.
• “Smart” Sharpe ratios: we want our strategy to lack autocorrelation, be memoryless, and hence, not have any long burn periods. If we penalize our Sharpe with autocorrelation, it can help us to choose such strategies via optimization.
• Information ratio: this metric helps us to compare our strategy to the underlying or the benchmark beyond the “alpha”. It is the annualized ratio between the average excess return and the tracking error.

combinatorial cross-validation vs stochastic simulations

Backtesting through cross-validation

Long story short, we want to have more than one historical path to check our strategy performance. We could sample it somehow from the historical data, but in what way? We could take different parts from different times from the whole dataset as training and testing sets. For generating these parts we already know the mechanism — it’s called cross-validation. For our purposes, we need as rich as possible a set of simulations — all possible combinations of subsets for training and testing the algorithm, which brings us the Combinatorial Purged Cross-Validation algorithm: