Tune Hadoop Cluster to get Maximum Performance (Part 1)
I have been working on production Hadoop clusters for a while and have learned many performance tuning tips and tricks. In this blog I will explain how to tune Hadoop Cluster to get maximum performance. Just installing Hadoop for production clusters or to do some development POC does not give expected results, because default Hadoop configuration settings are done keeping in mind the minimum hardware configuration. Its responsibility of Hadoop Administrator to understand the hardware specs like amount of RAM, Total number of CPU Cores, Physical Cores, Virtual Cores, Understand if hyper threading is supported by Processor, NIC Cards, Number of Disks that are mounted on Datanodes etc.
For Better Understanding I have divided this blog into two main parts.
1. Tune your Hadoop Cluster to get Maximum Performance (Part 1) – In this part I will explain how to tune your operating system in order to get maximum performance for your Hadoop jobs.
2. Tune your Hadoop Cluster to get Maximum Performance (Part 2) – In this part I will explain how to modify your Hadoop configurations parameters so that it should use your hardware very efficiently.
How OS tuning will improve performance of Hadoop?
Tuning your Centos6.X/Redhat 6.X can increase performance of Hadoop by 20-25%. Yes! 20-25%
Let’s get started and see what parameters we need to change on OS level.
1. Turn off the Power savings option in BIOS:
This will increase the overall system performance and Hadoop Performance. You can go to your BIOS Settings and change it to PerfOptimized from power saving mode (this option may be different for your server based on vendor). If you have remote console command line available then you can use racadm commands to check the status and update it. You need to restart the system in order to get your changes in effect.
2. Open file handles and files:
By default number of open file count is 1024 for each user and if you keep it to default then you may face java.io.FileNotFoundException: (Too many open files) and your job will get failed. In order to avoid this scenario set this number of open file limit to unlimited or some higher number like 32832.
Commands:
ulimit –S 4096 ulimit –H 32832
Also, Please set the system wide file descriptors by using below command:
sysctl –w fs.file-max=6544018
As above kernel variable is temporary and we need to make it permanent by adding it to /etc/sysctl.conf. Just edit /etc/sysctl.conf and add below value at the end of it
fs.file-max=6544018
3. FileSystem Type & Reserved Space:
In order to get maximum performance for your Hadoop job, I personally suggest by using ext4 filesystem as it has some advantage over ext3 like multi-block and delayed allocation etc. How you mount your file-system will make difference because if you mount it using default option then there will excessive writes for file or directory access times which we do not need in case of Hadoop. Mount your local disks using option noatime will surely improve your performance by disabling those excessive and unnecessary writes to disks.
Below is the sample of how it should look like:
UUID=gfd3f77-6b11-4ba0-8df9-75feb03476efs /disk1 ext4 noatime 0 0 UUID=524cb4ff-a2a5-4e8b-bd24-0bbfd829a099 /disk2 ext4 noatime 0 0 UUID=s04877f0-40e0-4f5e-a15b-b9e4b0d94bb6 /disk3 ext4 noatime 0 0 UUID=3420618c-dd41-4b58-ab23-476b719aaes /disk4 ext4 noatime 0 0
Note – noatime option will also cover noadirtime so no need to mention that.
Many of you must be aware that after formatting your disk partition with ext4 partition, there is 5% space reserved for special operations like 100% disk full so root should be able to delete the files by using this reserved space. In case of Hadoop we don’t need to reserve that 5% space so please get it removed using tune2fs command.
Command:
tune2fs -m 0 /dev/sdXY
Note – 0 indicates that 0% space is reserved.
4. Network Parameters Tuning:
Network parameters tuning also helps to get performance boost! This is kinda risky stuff because if you are working on remote server and you did a mistake while updating Network parameters then there can be a connectivity loss and you may not be able to connect to that server again unless you correct the configuration mistake by taking IPMI/iDrac/iLo console etc. Modify the parameters only when you know what you are doing
Modifying the net.core.somaxconn to 1024 from default value of 128 will help Hadoop by as this changes will have increased listening queue between the master and slave services so ultimately number of connections between master and slaves will be higher than before.
Command to modify net.core.somaxconnection:
sysctl –w net.core.somaxconn=1024
To make above change permanent, simple add below variable value at the end of /etc/sysctl.conf
net.core.somaxconn=1024
MTU Settings:
Maximum transmission unit. This value indicates the size which can be sent in a packet/frame over TCP. By default MTU is set to 1500 and you can tune it have its value=9000, when value of MTU is greater than its default value then it’s called as Jumbo Frames.
Command to change value of MTU:
You need to add MTU=9000 in /etc/sysconfig/network-scripts/ifcfg-eth0 or whatever your eth device name. Restart the network service in order to have this change in effect.
Note – Before modifying this value please make sure that all the nodes in your cluster including switches are supported for jumbo frames, if not then *PLEASE DO NOT ATTEMPT THIS*
5. Transparent Huge Page Compaction:
This feature of linux is really helpful to get the better performance for application including Hadoop workloads however one of the subpart of Transparent Huge Pages called Compaction causes issues with Hadoop job(it causes high processor usage while defragmentation of the memory). When I was benchmarking our client’s cluster I have observed some fluctuations ~15% with the output and when I disabled it then that fluctuation was gone. So I recommend to disable it for Hadoop.
Command:
echo never > /sys/kernel/mm/redhat_transparent_hugepages/defrag
In order to make above change permanent, please add below script in your /etc/rc.local file.
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag ;fi
6. Memory Swapping:
For Hadoop Swapping reduces the job performance, you should have maximum data in-memory and tune your OS so that it will do memory swap only when there is situation like OOM (OutOfMemory). To do so we need to set vm.swappiness kernel parameter to 0
Command:
sysctl -w vm.swappiness=0
Please add below variable in /etc/sysctl.conf to make it persistent.
vm.swappiness=0
I hope this information will help someone who is looking for OS level tuning parameters for Hadoop. Please don’t forget to give your feedback via comments or ask questions if any.
Thank you I will publish second part in the next week!








Very very useful info. Thanks for sharing. Can you please more performance tuning techniques for admin n other.
Thank you Kishor for the valuable feedback. Sure part2 is on my list and it should be published soon
In part2 I will be explaining hadoop level performance tuning!
Thanks again! Happy hadooping
This is very useful blog to increase cluster performance.
Thanks for the positive feedback Ashnee
It’s good to see that you aggregated info which was available in bits and pieces. Great Work.!!!
Thank you so much Pawan!
My Aim is to share the knowledge that I learned while working on production clusters so that newbie can grab the concepts quickly!
Nice information Kuldeep, really very helpful
Thank you so much Pritam.
Great work…found very useful to me
Thanks Bhupendra!
Stay tuned for more updates.
Great work… I’m an hadoop aspirant. Your info is awesome.
Thanks Sai!
Just desire to say your article is as amazing. The clarity in your post is just nice and i can assume you’re an expert on this subject. Fine with your permission let me to grab your RSS feed to keep up to date with forthcoming post. Thanks a million and please keep up the enjoyable work.
I’m really impressed with your writing skills as well as with the layout on your blog. Is this a paid theme or did you customize it yourself? Anyway keep up the nice quality writing, it is rare to see a nice blog like this one today..
Sweet blog! I found it while browsing on Yahoo News. Do
you have any suggestions on how to get listed in Yahoo News?
I’ve been trying for a while but I never seem to get there!
Cheers
Pretty nice post. I just stumbled upon your blog and wished to say
that I have really enjoyed surfing around your blog posts.
In any case I will be subscribing to your rss feed and I hope you write again soon!
Great post! We will be linking to this great post on our site.
Keep up the great writing.
Thank you so much!
I have read so many articles or reviews concerning the blogger lovers but this post is actually a good
paragraph, keep it up.
fantastic issues altogether, you simply gained a brand new reader.
What may you suggest about your put up that you simply made some days ago?
Any certain?
I simply want to say I am all new to blogs and really savored your blog. Very likely I’m likely to bookmark your website . You absolutely have perfect posts. Many thanks for sharing your web-site.
Why visitors still make use of to read news papers when in this technological globe all is accessible on net?
Great article.
Regards for helping out, fantastic information.
you have a great blog here! would you like to make some invite posts on my blog?
Kuldeep this is very useful list of tuning parameters summarized in one place, thanks for sharing your wisdom.
one request if you can share some of the production issues you faced that would be very helful.
To the point and very useful tip for administrators.
Thank you so much for sharing this information with us.