Survivorship bias: why many backtests look better than reality

Imagine testing a strategy on the Nifty Smallcap 250 from 2016 to today, using the companies that are in the index today. The result will almost certainly look better than anything you could really have achieved. This is survivorship bias.

What goes wrong

Today's index contains companies that did well enough to get into it. Companies that collapsed, were delisted or fell to a smaller index are gone. A backtest on today's list therefore:

• buys future winners years before anyone could have known they would be in the index, and
• never holds the companies that failed along the way.

Both push the result up.

The fix: the index as it was on each date

A fair backtest uses, on every rebalance date, the index members as they were on that date. NSE publishes only today's lists, but it announces every change: semi-annual reviews and changes after mergers, demergers and suspensions.

MomentumScore rebuilds the past lists from those announcements. It starts from today's official list and undoes each announced change, newest first. The result:

• Nifty 500 lists from 2012,
• Midcap 150 and Smallcap 250 lists from 2016,
• Microcap 250 lists from 2021, when NSE launched it.

Years before an index existed are estimated from market capitalisation and turnover.

How big is the difference?

It depends on the strategy and the period. Small-cap and micro-cap strategies are hit hardest, because that is where companies move in and out most often. It is common for "today's members" backtests to show several percentage points more return a year than "historical members" backtests of the same rules.

If a backtest result looks too good, check which index list it used first.

In MomentumScore

In the strategy builder, step 1 (Universe), turn on Historical members. Every past rebalance then uses the index as it really was.