Automated Kerberos Installation and Configuration
Automated Kerberos Installation and Configuration – For this post, I have written a shell script which uses Ambari APIs to configure Kerberos on HDP Single or Multinode clusters. You just need to clone our github repository and modify property file according to your cluster environment, execute setup script and phew!! Within 5-10 minutes you should have your cluster completely secured by Kerberos! Cool isn’t it? 
Detailed Steps(Demo on HDP Sandbox 2.4):
1. Clone our github repository on your local machine or one of the node in your Hadoop Cluster.
git clone https://github.com/crazyadmins/useful-scripts.git
Sample Output:
[root@sandbox ~]# git clone https://github.com/crazyadmins/useful-scripts.git Initialized empty Git repository in /root/useful-scripts/.git/ remote: Counting objects: 29, done. remote: Compressing objects: 100% (25/25), done. remote: Total 29 (delta 4), reused 25 (delta 3), pack-reused 0 Unpacking objects: 100% (29/29), done.
2. Goto useful-scripts/ambari directory
[root@sandbox ~]# cd useful-scripts/ambari/ [root@sandbox ambari]# ls -lrt total 16 -rw-r--r-- 1 root root 5701 2016-04-23 20:33 setup_kerberos.sh -rw-r--r-- 1 root root 748 2016-04-23 20:33 README -rw-r--r-- 1 root root 366 2016-04-23 20:33 ambari.props [root@sandbox ambari]#
3. Copy setup_kerberos.sh and ambari.props to the host where you want to setup KDC Server
4. Edit and modify ambari.props file according to your cluster environment
Sample output for my Sandbox
[root@sandbox ambari]# cat ambari.props CLUSTER_NAME=Sandbox AMBARI_ADMIN_USER=admin AMBARI_ADMIN_PASSWORD=admin AMBARI_HOST=sandbox.hortonworks.com KDC_HOST=sandbox.hortonworks.com REALM=HWX.COM KERBEROS_CLIENTS=sandbox.hortonworks.com ##### Notes ##### #1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster #2. Admin princial is admin/admin and password is hadoop [root@sandbox ambari]#
5. Start installation by simply executing setup_kerberos.sh
Notes:
1. Please run setup_kerberos.sh from KDC_HOST only, you don’t need to setup or configure KDC, this script will do everything for you.
2. If you are running script on Sandbox then please turn OFF maintenance mode for HDFS and turn ON maintenance mode for Zepplin Notebook before executing the script.
sh setup_kerberos.sh
Screenshots:
1. Before Script Execution
2. Script execution is in progress
3. Script finished
4. Ambari UI shows Kerberos is enabled.
Please comment if you have any feedback/questions/suggestions. Happy Hadooping!!
by 
Loading ...
I run your scripts as you mentioned in README successfully. But the kerberos is not enabled and keytab files are also not created. I am using Ambari 2.1.2.1 and HDP 2.3. Below are the logs:
kerberos.props Kerb_setup.log payload setup_kerberos.sh
[root@hdp1 scripts]# cat Kerb_setup.log
2016-04-24,21:16:29 Installing kerberos RPMs
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: http://ftp.iitm.ac.in
* extras: http://ftp.iitm.ac.in
* updates: http://ftp.iitm.ac.in
Setting up Install Process
Package krb5-server-1.10.3-42z1.el6_7.x86_64 already installed and latest version
Package krb5-libs-1.10.3-42z1.el6_7.x86_64 already installed and latest version
Package krb5-workstation-1.10.3-42z1.el6_7.x86_64 already installed and latest version
Nothing to do
2016-04-24,21:16:35 Configuring Kerberos
Loading random data
Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘ARGOTECH.COM’,
master key name ‘K/M@ARGOTECH.COM’
2016-04-24,21:18:10 Starting KDC services
Starting Kerberos 5 KDC: [ OK ]
Starting Kerberos 5 Admin Server: [ OK ]
2016-04-24,21:18:10 Creating admin principal
Authenticating as principal root/admin@ARGOTECH.COM with password.
Principal “admin/admin@ARGOTECH.COM” created.
2016-04-24,21:18:11 Restarting kadmin
Stopping Kerberos 5 Admin Server: [ OK ]
Starting Kerberos 5 Admin Server: [ OK ]
2016-04-24,21:18:14 Adding KERBEROS Service to cluster
HTTP/1.1 201 Created
User: admin
Set-Cookie: AMBARISESSIONID=15p6n2gg03h781t4pdmtgnwx7s;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
2016-04-24,21:18:14 Adding KERBEROS_CLIENT component to the KERBEROS service
HTTP/1.1 201 Created
User: admin
Set-Cookie: AMBARISESSIONID=1luaj6ebadcom1ejqd9k5ul7vd;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
User: admin
Set-Cookie: AMBARISESSIONID=14wv8tphqrzrxryhijqcfr0d6;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 1640
Server: Jetty(8.1.17.v20150415)
{
“resources” : [
{
“href” : “http://hdp1.argotech.com:8080/api/v1/clusters/argotech/configurations/service_config_versions?service_name=KERBEROS&service_config_version=2″,
“configurations” : [
{
“clusterName” : “argotech”,
“stackId” : {
“stackName” : “HDP”,
“stackVersion” : “2.3”,
“stackId” : “HDP-2.3″
},
“type” : “kerberos-env”,
“versionTag” : “version1″,
“version” : 1,
“serviceConfigVersions” : null,
“configs” : {
“kdc_type” : “mit-kdc”,
“password_min_uppercase_letters” : “1”,
“password_min_whitespace” : “0”,
“password_min_punctuation” : “1”,
“password_min_digits” : “1”,
“encryption_types” : “aes des3-cbc-sha1 rc4 des-cbc-md5″,
“admin_server_host” : “hdp1.argotech.com.com”,
“password_min_lowercase_letters” : “1”,
“password_length” : “20”,
“case_insensitive_username_rules” : “false”,
“manage_identities” : “true”,
“service_check_principal_name” : “-“,
“kdc_host” : “hdp1.argotech.com.com”,
“install_packages” : “true”,
“realm” : “ARGOTECH.COM”,
“executable_search_paths” : “/usr/bin, /usr/kerberos/bin, /usr/sbin, /usr/lib/mit/bin, /usr/lib/mit/sbin”
},
“configAttributes” : { }
}
],
“group_id” : -1,
“group_name” : null,
“service_config_version” : 2,
“service_config_version_note” : null,
“service_name” : “KERBEROS”
}
]
}
2016-04-24,21:18:17 Creating the KERBEROS_CLIENT host components for each host
HTTP/1.1 201 Created
User: admin
Set-Cookie: AMBARISESSIONID=1p4wixclhn97fe64vv00o8yjr;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
2016-04-24,21:18:18 Installing the KERBEROS service and components
HTTP/1.1 202 Accepted
User: admin
Set-Cookie: AMBARISESSIONID=1ag9cj7bwkkkr9ll6byw7r5f;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 146
Server: Jetty(8.1.17.v20150415)
{
“href” : “http://hdp1.argotech.com:8080/api/v1/clusters/argotech/requests/61″,
“Requests” : {
“id” : 61,
“status” : “Accepted”
}
}
2016-04-24,21:18:18 Sleeping for 1 minute
2016-04-24,21:19:18 Stopping all the services
HTTP/1.1 200 OK
User: admin
Set-Cookie: AMBARISESSIONID=g3w5b7n1d8zpr3uiy9y0rvl7;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(8.1.17.v20150415)
2016-04-24,21:19:18 Sleeping for 2 minutes
2016-04-24,21:21:18 Enabling Kerberos
HTTP/1.1 500 Internal Server Error
User: admin
Set-Cookie: AMBARISESSIONID=llojdxrepoyrjxakmmnudsaf;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 189
Server: Jetty(8.1.17.v20150415)
{
“status” : 500,
“message” : “org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Unexpected error condition executing the kadmin command”
}
2016-04-24,21:23:26 Starting all services
HTTP/1.1 202 Accepted
User: admin
Set-Cookie: AMBARISESSIONID=fzz2064m9k4j1gdusmy3rjpc2;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 146
Server: Jetty(8.1.17.v20150415)
{
“href” : “http://hdp1.argotech.com:8080/api/v1/clusters/argotech/requests/63″,
“Requests” : {
“id” : 63,
“status” : “Accepted”
}
}
2016-04-24,21:23:26 Please check Ambari UI
Thank You!
Hi Avimanyu,
Looks like below curl call did not work, can you please try it manually and let me know how it goes?
Error in your log:
{
“status” : 500,
“message” : “org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Unexpected error condition executing the kadmin command”
}
This error is because below curl did not complete successfully:
curl -H “X-Requested-By:ambari” -u $AMBARI_ADMIN_USER:$AMBARI_ADMIN_PASSWORD -i -X PUT -d @$LOC/payload http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME
Payload for above command is:
{
“session_attributes” : {
“kerberos_admin” : {
“principal” : “admin/admin”,
“password” : “hadoop”
}
},
“Clusters”: {
“security_type” : “KERBEROS”
}
}
Hi,
Is there any possibility to configure HDFS Federation with High Availability (Active/Standby). If it is, could you please post the steps.
Thanks in advance.
Regards,
Sathish G.