I wrote this post while listening to this song. Hit the play button and enjoy!
Have you heard about the Survivorship bias? The most important information might not be in your dataset.
During World War II, many planes were shot down in combat causing a huge loss of life and equipment. To reduce losses, the American military decided to reinforce the planes' armour. They knew armour would help, but couldn’t protect the whole plane or would be too heavy to fly well. Therefore, they carried out a study to define where the armour should be reinforced.
They examined the planes returning from combat, see where they were hit the worst, and then reinforced those areas. Until Abraham Wald came to the opposite conclusion: The bullet holes they were looking at actually indicated the areas a plane could be hit and keep flying. Areas that needed reinforcement were those where there were no holes. Planes hit in those areas were shot down and didn't return. The most important information wasn't in the dataset.
This phenomenon, called survivorship bias, can play a significant role in the data we analyse. It is just one example of how misunderstanding information may lead you to wrong conclusions. Knowing how to analyse data correctly is fundamental for any professional's career and any company's success.
That's one of the things we learn here at The Data School: how to look at the big picture and ask the right questions so we can solve the problem most efficiently.
Be careful when you analyse your data. You might be missing what's missing. The most important information might not be in your dataset.
- Consider following me on LinkedIn.
- You can check out my portfolio on my Tableau Public.
- If you want to know more about the survivorship bias, watch this Ted Talk.