Large scale Hadoop BET

Building online integration platform requires collecting of large amount of business events from thousands of apps. Since each application generates tens to hundreds events per second, we decided to use HBase on EMR for processing all this data. The presentation covers:

Challenges of building a data collector with five nines uptime
Real time event triggers based on business events
Using Hadoop to process and present millions of events per day in "near real time"
Integration with ElasticSearch to correlate business events and application logs.

Mike founded 2 startups, lead the team that created the largest online b2b auction in automotive industry and recently has been responsible for building CloudHub - leader in integration PaaS.