Photo by Igor Ovsyannykov (https://unsplash.com/ @igorovsyannykov)
Working through Opticore, I worked on a number of Data projects for Betfair. One of the more important ones was a Big Data Operations project.
The public API was taking a beating from account holders and the current process for determining abuse of the API was constantly several weeks behind.
Using a Map Reduce system, we were able to process the daily logs from the large number of API web servers down to an analysis set. This was made available through a dynamic Tableau dashboard and allowed the stakeholder to assess on a daily basis which accounts were genuine and which were guilty of abuse.
The abusing accounts were suspended and blocked within a day. After this developed process was put in place, the pressure on the web servers was drastically reduced resulting in a cost saving from scaling back the infrastructure required to met the previous traffic.