Travel websites were abuzz this month with the latest safety ranking from Airline Ratings, an industry data provider. Australian flag-carrier Qantas was crowned the safest airline in the world for the third year in a row, while other big names like American Airlines, Cathay Pacific and Emirates also made it into the top ten. At the bottom of the list were some rather more obscure carriers like Batik Air and Kalstar Aviation.

Airline Ratings came up with its ranking by looking at a range of factors perceived to reflect safety performance – prior accident history, current fleet type, recognized industry certifications and so forth. At least two other websites also claim to rank airlines by safety: the Air Transport Rating Agency; and the Jet Airliner Crash Data Evaluation Center.

Each of these reports, though, comes up with wildly different rankings. And that is not surprising, given that their authors are engaged in PR exercises bereft of any statistical validity.

The trouble with airline safety rankings is simple: the law of averages does not come into force with limited air crash data. No matter how canny a report’s methodology, it is impossible to collect a sufficiently large data pool on any one airline to assess its track-record. According to the International Air Transport Association (IATA), only one in every 4.4 million jet flights resulted in a hull loss in 2014. In order to know whether a given airline has historically performed above or below this incidence rate, researchers would need to look at hundreds of millions of flights by the company. Attempting to do that is futile even for the very largest, oldest carriers; it is a complete farce for small or newly established ones.

Paul Hayes, safety and insurance director at Ascend, an aviation consultancy, put it best when I interviewed him back in 2011:

Joe Public wants to be told that he’s flying on the safest airline in the world, but realistically no-one can give you that comfort. It’s a case of lies, damned lies and statistics. The problem is that when you look at individual carriers, no airline in the world is big enough to suffer loss of average. If you had hundreds of millions of flights and thousands of accidents, then you could do very robust statistics. But accidents are so incredibly rare that even huge airlines like Delta can’t suffer one incident in five years and still be average.

Basing safety rankings on recent crashes is the logical equivalent of stating that a lottery winner had a 1/52 chance of winning the lottery last year, based on the fact that their numbers came up in one fortuitous week. The claim is bogus, plain and simple. A true probability only emerges when you stretch the timeframe out for millennial.

Source: http://www.forbes.com/sites/martinrivers/2016/01/21/why-you-can-safely-ignore-airline-safety-rankings/#2ec292d87dc0