Note – I have kept Namenodes on prodnode1 and prodnode3, you can change it according to your requirement. I have added few more services like Hive, Falcon, Oozie etc. You can remove them or add few more according to your requirement.
Step 4: Create an internal repository map
4.1: hdp repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in repo.json file.
Automated Kerberos Installation and Configuration – For this post, I have written a shell script which uses Ambari APIs to configure Kerberos on HDP Single or Multinode clusters. You just need to clone our github repository and modify property file according to your cluster environment, execute setup script and phew!! Within 5-10 minutes you should have your cluster completely secured by Kerberos! Cool isn’t it?
Detailed Steps(Demo on HDP Sandbox 2.4):
1. Clone our github repository on your local machine or one of the node in your Hadoop Cluster.
3. Copy setup_kerberos.sh and ambari.props to the host where you want to setup KDC Server
4. Edit and modify ambari.props file according to your cluster environment
Sample output for my Sandbox
[root@sandbox ambari]# cat ambari.props
##### Notes #####
#1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster
#2. Admin princial is admin/admin and password is hadoop
5. Start installation by simply executing setup_kerberos.sh
1. Please run setup_kerberos.sh from KDC_HOST only, you don’t need to setup or configure KDC, this script will do everything for you.
2. If you are running script on Sandbox then please turn OFF maintenance mode for HDFS and turn ON maintenance mode for Zepplin Notebook before executing the script.
1. Before Script Execution
2. Script execution is in progress
3. Script finished
4. Ambari UI shows Kerberos is enabled.
Please comment if you have any feedback/questions/suggestions. Happy Hadooping!!
When you install HDP for dev/test environment, you would repeat same commands to set up your host OS. To save time, created a BASH script which helps to set up the host OS (Ubuntu only) and docker image (CentOS).
What this script does:
Install packages on Ubuntu host OS
Set up docker, such as creating image and spawning containers
[Optional] Set up a local repository for HDP (not Ambari) with Apache2
Script will ask a few questions such as your choice of guest OS, Ambari version, HDP version etc. Normally default values should be OK, so you can just keep pressing Enter key. NOTE: The end of interview, it asks you to save your answer in a text file. You can reuse this file to skip interview when you install a new cluster.
Step 6: Confirm your answers
After saving your responses, it will ask you “Would you like to start setup this host? [Y]:“. If you answer yes, it starts setting up your Ubuntu host OS. After waiting for while, the scripts finishes, or if there is any error, it stops.
The time would be depending on your choice. If you selected to setup a local repo, downloading repo may take long time.
Step 7: Complete the setup
Once the script completed successfully, your choice of Ambari Server should be installed and running on your specified docker container on port 8080.
NOTE: At this moment, docker containers are installed in a private network, so that you would need to do one of followings (“1″ would be the easiest):
Following command creates proxy from your local PC port 18080
ssh -D 18080 username@ubuntu-hostname
Following command do port forwarding from your localhost:8080 to node1:8080
If you decided to set up a proxy, installing addon such as “SwitchySharp” would be handy.
Once you confirmed you can use Ambari web interface, please proceed to install HDP. If you choose to set up a HDP local repository, please replace “public-repo-1.hortonworks.com” to “dockerhost1.localdomain” (if you used default value)
Private key should be /root/.ssh/id_rsa in any node
Remaining steps should be same as installing normal HDP. NOTE: if you decided to install older Ambari version, there is a known issue AMBARI-8620
Host Start up step
If you shutdown the VM, next time you can just run “./start_hdp.sh -s” which starts up containers, Ambari Server, Ambari Agents and HDP services.
How to semi-automate deploying dev hdp cluster – Did you like this article ? please feel free to send an email to email@example.com if you have any further questions on this. Please don’t forget to like our facebook page. Happy Hadooping!!
Blogpost after long time okay, in this post we will see how to Automate HDP installation using Ambari Blueprints
What are Ambari Blueprints ?
Ambari Blueprints are definition of your HDP cluster in “JSON” format, it contents information about all the hosts in your cluster, their components, mapping of stack components with each hosts or hostgroups and other cool stuff. Using Blueprints we can call Ambari APIs to completely automate HDP installation process. Interesting stuff, isn’t it ?
Lets get started with single node cluster installation. Below are the steps to setup single-node HDP cluster with Ambari Blueprints.
Step 1: Install Ambari server using steps mentioned under below link