How to Increase Performance of Your Hadoop Cluster by 30%: a Case Study on Optimizing a 100-node Hadoop Cluster

This case study-based workshop will walk attendees through every milestone of a month long effort to come up with an objective way to measure the performance of Hadoop clusters. The subject of the study was the influence of specific host operating systems and hypervisors on cluster performance.

The results identified significant deviation of cluster performance for equal workloads across various operating systems and hypervisors.

21% to 275% deviation was found between sort time of the same workloads across clusters running Ubuntu, RHEL and SmartOS.

In this panel, the Altoros and Joyent technical staff will share results of benchmarking Apache Hadoop on Ubuntu, RHEL and RedHat and Joyent hypervisors on 50-250 node clusters:

Benchmarking environment setup (will share Chef cookbook)
Configuration details
Anti-Patterns (things to avoid)
Tools used to measure cluster performance
How we achieved 2.75x performance improvement
Ideas on cost- and rule-based optimization

Renat Khasanshyn is CEO of Altoros Systems and Venture Partner at Runa Capital. Renat is a subject matter expert on open source database infrastructure software. Most recently Renat scaled an idea called "Altoros" into a global software services organization with 200+ engineers in Belarus, US, UK, Denmark, and Norway. In the past, Mr. Khasanshyn was selected as a finalist for the 2007 Emerging Executive of the Year award by the Massachusetts Technology Leadership Council. In 2006, Mr. Khasanshyn founded Apatar, the world's first on-demand, open source data mashup software project and won the 2007 IBM Business Mashup Challenge. Prior to founding Altoros in 2001, Renat was VP of Engineering for Tampa-based insurance company PriMed, Inc. Mr. Khasanshyn is a founder of Silicon Valley NewSQL User Group, and co-founder of Belarusian Java User Group. Renat studied Engineering at the Belarusian National Technical University.

Ben is responsible for Joyent's product marketing and messaging. Previously he was VP of Marketing and Sales at MongoLab, and Product Manager for IBM WebSphere, focused on DataPower SOA Appliances and Messaging Security. He worked in venture finance under Brad Feld as an Associate Partner at Mobius Venture Capital. Ben was co-founder of ThinkFish Productions, developing a non-photorealistic 3D rendering engine. Ben has an EECS SB degree from MIT where he worked on the massively parallel Project Alewife. In his spare time Ben loves to cook, photograph, hack in Emacs and spend time with his family.