Oozie Tutorials – SSH Action

Oozie Tutorials – SSH Action

Oozie ssh action executes shell script on remote machine in secure shell, workflow will wait until ssh script is complete and then move to next action. 

 

Prerequisites:

1. Shell script must be present on remote host at given path.

2. Shell script will be executed in home directory of specified use on remote host.

3. Password-less ssh must be set from oozie@<oozie-host> to user@<remote-host>

To Configure passwordless ssh:

3.1 Login to Oozie server host via oozie user

3.2 Run below command ( press enter for every question to keep all the options default )

ssh-keygen

3.3 On Oozie host copy contents of ~/.ssh/id_rsa.pub and paste it in <remote-host>’s ~/.ssh/authorized_keys file

3.4 Please make sure to have below permissions in order to get password-less ssh working successfully

700 to ~/.ssh directory on oozie as well as remote-host
600  to ~/.ssh/authorized_keys file on remote-host
600 to ~/.ssh/id_rsa on oozie host
644 on ~/.ssh/id_rsa.pub on oozie host

3.5 Test ssh from oozie@<oozie-host> to <username>@<remote-host>, ssh command should work without password.

 

Steps:

1. Configure job.properties for your cluster environment

Example:

#************************************************* 
# job.properties
#*************************************************
nameNode=hdfs://<namenode-machine-fqdn>:8020
jobTracker=<resource-manager-fqdn>:8050
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
oozieProjectRoot=${nameNode}/user/${user.name}
appPath=${oozieProjectRoot}
oozie.wf.application.path=${appPath}
inputDir=${oozieProjectRoot}
remoteNode=<username>@<remote-host-where-you-have-your-shell-script(s)>
shellScriptPath=/tmp/test.sh
emailToAddress=<email-id>

 

2. Create and configure workflow.xml which contains ssh action

Example:

<!--******************************************-->
<!--workflow.xml -->
<!--******************************************-->
<workflow-app name="DemoSshAction" xmlns="uri:oozie:workflow:0.1">
 <start to="sshAction"/>
 <action name="sshAction">
 <ssh xmlns="uri:oozie:ssh-action:0.1">
 <host>${remoteNode}</host>
 <command>${shellScriptPath}</command>
 <capture-output/>
 </ssh>
 <ok to="sendEmail"/>
 <error to="killAction"/>
 </action>
 <action name="sendEmail">
 <email xmlns="uri:oozie:email-action:0.1">
 <to>${emailToAddress}</to>
 <subject>Output of workflow ${wf:id()}</subject>
 <body>Status of the file move: ${wf:actionData('sshAction')['STATUS']}</body>
 </email>
 <ok to="end"/>
 <error to="end"/>
 </action>
 <kill name="killAction">
 <message>"Killed job due to error"</message>
 </kill>
 <end name="end"/>
</workflow-app>

 

3. Write sample /tmp/test.sh shell script

Example:

#!/bin/bash
hadoop fs -put ~/test /user/<username>/uploadedbyoozie

 

4. Upload workflow.xml to the HDFS at ${appPath} defined in job.properties.

hadoop fs -put $local_path/workflow.xml /user/<user-name>/

 

5. Submit Oozie workflow using below command

oozie job -oozie http://<oozie-host>:11000/oozie -config $local-path/job.properties -run

 

Screenshots:

Note – Hit below URL to open Oozie WebUI

http://<oozie-server>:11000/oozie

 

1. Oozie job started.

 

oozie tutorials - ssh action

 

2. Job completed successfully

 

oozie tutorials - ssh action

 

oozie tutorials - ssh action

 

 

Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! :)

 

 

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

2 comments

  • munikrishna

    Please help on , i am getting below issue.

    JOB[0000088-160531014243114-oozie-oozi-W] ACTION[0000088-160531014243114-oozie-oozi-W@load_hadoop] Attempting to copy ssh base scripts to remote host [jithin@edgenode.cityanalytics.ibm.com]
    2016-07-19 04:39:20,288 WARN SshActionExecutor:523 – SERVER[masternode.cityanalytics.ibm.com] USER[jithin] GROUP[-] TOKEN[] APP[Dallas_Test] JOB[0000088-160531014243114-oozie-oozi-W] ACTION[0000088-160531014243114-oozie-oozi-W@load_hadoop] Error while executing ssh EXECUTION
    2016-07-19 04:39:20,288 WARN ActionStartXCommand:523 – SERVER[masternode.cityanalytics.ibm.com] USER[jithin] GROUP[-] TOKEN[] APP[Dallas_Test] JOB[0000088-160531014243114-oozie-oozi-W] ACTION[0000088-160531014243114-oozie-oozi-W@load_hadoop] Error starting action [load_hadoop]. ErrorType [TRANSIENT], ErrorCode [FNF], Message [FNF: Required Local file /var/tmp/oozie/oozie-oozi7179698092277115649.dir/ssh/ssh-base.sh not present.]
    org.apache.oozie.action.ActionExecutorException: FNF: Required Local file /var/tmp/oozie/oozie-oozi7179698092277115649.dir/ssh/ssh-base.sh not present.
    at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:572)
    at org.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:206)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.IOException: Required Local file /var/tmp/oozie/oozie-oozi7179698092277115649.dir/ssh/ssh-base.sh not present.
    at org.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:367)
    at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:208)
    at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206)
    at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:550)
    … 10 more

Leave a Reply to munikrishna Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>