Supervised learning (or “machine learning by examples”) is here to stay, but is it always the optimal solution? Whenever engineers need to tell the difference between cats and dogs they will turn to supervised learning models provided with lots of training examples. But what happens when you’re looking for a rare and extraordinary phenomenon such as a unicorn? In that case you might have abundant examples of horses but almost no unicorn examples. In our story, the unicorn is the problem of detecting “low and slow” DNS exfiltration attacks, such as the 2014 cyberattack on Home Depot that resulted in the theft of 65M credit card numbers. While the Home Depot attack establishes the importance of the problem, it’s rare to find enough similar examples in order to predict the next DNS exfiltration cyber campaign. In this talk we will discuss the advantages of Anomaly Detection in the absence of training samples and walk through our solution based on the Isolation Forest algorithm and the challenges we faced implementing it as a large-scale solution in Spark Scala. This is the story of how we had to take a different approach to our problem and how we got to catch and block a live ‘white-hat’ cyber attack on one of our clients’ platform.
