Archive for : October, 2015

Writing Custom Gradle Plugin using Java

What is Gradle ?

Gradle is build automation tool based on Apache ant and Maven. Gradle avoids traditional .xml file based configuration by introducing groovy based domain specific language. In gradle project have .gradle files instead of .pom files. Gradle was designed for multi-project builds and supports incremental build.

Gradle plugin groups together reusable pieces of build logic which can be then used across many different projects and builds. We can use any language whose compiled code gets converted to byte code for  developing custom gradle plugin. As gradle is mainly designed using groovy language its very easy to develop gradle plugin using groovy but  lets see how to develop custom gradle plugin using Java language:

d

Here are the steps :

 

1. Create new Java project using eclipse or any other IDE.

 

2. Create plugin folder in your project which will have source code of your custom gradle plugin.

 

3. create your package structure in this plugin folder.

    Eg: com.sample.gradle

This package structure will have your custom plugins source code.

 

4. Create Plugin class which will implement Plugin<Project> interface.Plugin is represents extension to gradle.This interface is having apply method which applies configuration to gradle Project object.

Eg :

package com.sample.gradle;
import org.gradle.api.Project;
import org.gradle.api.Plugin;
import com.sample.gradle.SamplePluginExtension;
public class SampleGradlePlugin implements Plugin<Project> {
                @Override
    public void apply(Project target) {
                                target.getExtensions().create("samplePlugin",
                                                                SamplePluginExtension.class);    }
}

 

5. All the user defined values to custom plugin are provided through extension object so creaete extension class and register it with plugin as shown above to receive inputs from user.If user doesnot provided input then default values will be assumed.

 

6. Create extension class which is similar to java pojo class it will contain user defined properties and their getter setter methods.If user provides values for these properties during run time then these values will be accepted otherwise default values will be considered.

 

Eg :

public class SamplePluginExtension {
                private String sampleFilePath="/home/mahendra/abc”;
                public void setSampleFilePath(String sampleFilePath){
                  this.sampleFilePath=sampleFilePath;
                }
                public String getSampleFilePath(){
                                return sampleFilePath;
                }
}

 

7. Create task class which will have your plugin logic this task class contains your main plugin logic.This task class extends org.gradle.api.DefaultTask class and defines method with @TaskAction annotation.This method will have actual logic.

Eg:

package com.sample.gradle;
import org.gradle.api.DefaultTask;
import org.gradle.api.tasks.TaskAction;
import org.gradle.api.tasks.TaskExecutionException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class SampleTask extends DefaultTask {
                private final Logger log = LoggerFactory.getLogger(this.getClass());
                @TaskAction
                public void samplePluginTasks() throws TaskExecutionException {
                                log.info("Starting  sample task");
                                try {
                                                SamplePluginExtension extension = getProject().getExtensions()
                                                                                .findByType(
amplePluginExtension.class);
                                                String filePath = extension.getSampleFilePat
();
                                                log.debug("Sample file path is: {}",filePath
;
                                                /* Here comes
                                                 *
                                                 * your main logic
                                                 *
                                                 */
                                                log.info("Successfully completed sample Task
);
                                }catch(Exception e){
                                                log.error("",e);
                                                throw new TaskExecutionException(this,new Ex
eption("Exception occured while processing sampleTask",e));
                                }
                }
}

 

For logging into your custom plugin use slf4j or any other logging framework of your choice.If you want to fail the build on exception in your task then throw TaskExecutionException which will cause BuildFailures other exceptions will not cause build failure.TaskExecutionException accepts task object and Throwable object as input.

 

Here DefaultTask is standard gradle task implementation class and we need to extend it while implementing custom tasks.@TaskAction annotation makes method action method and whenever task executes this method will be executed.

 

8. Registering plugin class : create resources folder in plugin/src/main folder.Inside resources create META-INF/gradle-plugin folder, in this gradle-plugin folder create properties file.Name of this property file is used for registering plugin in build.gradle build file.

 

eg: sample-plugin.properties:

In sample-plugin.properties file add following line:

implementation-class=com.sample.gradle.SampleGradlePlugin

value of implementation-class if path of plugin class.

 

9. Create settings.gradle file in your plugin folder which will have below line:

rootProject.name = 'customGradlePlugin'

value of rootProject.name is name of your root project.

 

10. Create build.gradle build file for you plugin in plugin folder as follows:

apply plugin: 'java'
dependencies {
    compile gradleApi()
}
apply plugin: 'maven-publish'
repositories {
    mavenCentral()
    mavenLocal()
    jcenter()
}
dependencies {
    compile 'org.slf4j:slf4j-simple:1.6.1'
    testCompile 'junit:junit:4.11'
}
group = 'com.sample.gradle'
version = '1.0.0'
publishing {
                publications {
                                maven(MavenPublication) {
                                                groupId "$group"
                                                artifactId 'CustomGradlePlugin'
                                                version "$version"
                                                from components.java
                                }
                }
}
uploadArchives {
    repositories {
            mavenLocal()
    }
}

as we are developing gradle plugin using java add apply plugin:’java’ line.whatever external dependencies your plugin is dependent upon add these dependencies in dependencies section.the repositories in which your plugin should look for should be mentioned in repositories section.mentioned group id,version of plugin in group and version tag.

For making plugin available to other projects gradle plugin should be published to repository or its archives should be uploaded for this purpose either use publishing or uploadArchives functionality.

 

for publishing plugin use following command if you are publishing plugin to local maven repository:

“gradle clean build publishToMavenLocal”

 

If you are uploading plugin to local maven repository then use below command:

“gradle clean build uploadToArchives”

 

11. For using plugin in another projects make following changes in build.gradle file of your project.

apply plugin: 'java'
buildscript {
repositories {
         mavenLocal()
         mavenCentral()
    }
    dependencies {
                                classpath "com.sample.gradle:CustomGradlePlugin:1.0.0"
    }
}
apply plugin: 'sample-plugin'
 task sampleTask(type: com.sample.gradle.SampleTask) {
 samplePlugin.sampleFilePath = "$sampleFilePath"
}

 

here plugin dependency must be defined in buildscript section and  to tell gradle which repositories to scan for getting plugin dependencies  add repositories section in buildscript section this repository section must come ahead of dependencies section.Afer this add apply plugin line.

Sample task provided will be used for executing our plugin logic in task value of type is path of our custom task.whatever custom arguments we want to provide to plugin that we need to define in task section .if we want to run with default parameters then comment samplePlugin.sampleFilePath line in task section.

To run plugin with custom parameter use below command:

gradle clean build sampleTask -PsampleFilePath='/home/mahendra/abc.sample'

 

To run plugin without custom parameter use below command:

gradle clean build sampleTask

 

Sample plugin project structure is as follows:

 

m

 

Here we are done with developing gradle custom plugin with java.

 

please let me know if you have any issues or suggestions.The sample project is present on github you can download it from below link:

https://github.com/omtonape/CustomPlugin-GradleJava.git

 

Please do not forget to comment your feedback on this Article! :-)

 

 

 

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Kafka integration with Ganglia

In this blog post I will show you kafka integration with ganglia, this is very interesting & important topic for those who want to do bench-marking, measure performance by monitoring specific Kafka metrics via ganglia.

Before going ahead let me briefly explain about what is Kafka and Ganglia.

Kafka – Kafka is open source distributed message broker project developed by Apache, Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds.

Ganglia – Ganglia is distributed system for monitoring high performance computing systems such as grids, clusters etc.

ganglia_logo

Now lets get started, In this example we have a Hadoop cluster with 3 Kafka brokers, First we will see how to install and configure ganglia on these machines.

 

Step 1: Setup and Configure Ganglia gmetad and gmond

 

First thing is you need to install EPEL repo on all the nodes

yum install epel-release

On master node (ganglia-server) download below packages

yum install rrdtool ganglia ganglia-gmetad ganglia-gmond ganglia-web httpdphpaprapr-util

On slave nodes (ganglia-client) download below packages

yum install ganglia-gmond

On master node do the following

chown apache:apache -R /var/www/html/ganglia

Edit below config file and allow ganglia webpage from any IP

vi /etc/httpd/conf.d/ganglia.conf

It should look like below:

#
# Ganglia monitoring system php web frontend
#
Alias /ganglia /usr/share/ganglia
<Location /ganglia>
Order deny,allow
Allow from all                    #this is very important or else you won’t be able to see ganglia web UI
Allow from 127.0.0.1
Allow from ::1
# Allow from .example.com
</Location>

On master node edit gmetadconfig file and it should look like below (Please change highlighted IP address to your ganglia-server private IP address)

#cat /etc/ganglia/gmetad.conf |grep -v ^# 
data_source "hadoopkafka" 172.30.0.81:8649 
gridname "Hadoop-Kafka"
setuid_username ganglia
case_sensitive_hostnames 0

On master node edit gmond.conf, keep other parameters to default except below ones

Copy gmond.conf to all other nodes in the cluster

cluster {
name = "hadoopkafka"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}
/* Feel free to specify as many udp_send_channels as you like. Gmond
used to only support having a single channel */
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
                       # This option tells gmond to use a source address
                       # that resolves to the machine's hostname. Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
#mcast_join = 239.2.11.71
host = 172.30.0.81
port = 8649
#ttl = 1
}
/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
#retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
# buffer = 10485760
}

Start apache service on master node

service httpd start

Start gmetad service on master node

service gmetad start

Start gmond service on every node in the server

service gmond start

 

 

This is it!  Now you can see basic ganglia metrics by visiting web UI at http://IP-address-of-ganglia-server/ganglia

 

Step 2: Ganglia Integration with Kafka

 

Enable JMX Monitoring for Kafka Brokers

In order to get custom Kafka metrics we need to enable JMX monitoring for Kafka Broker Daemon.

To enable JMX Monitoring for Kafka broker, please follow below instructions:

Edit kafka-run-class.sh and modify KAFKA_JMX_OPTS variable like below (please replace red text with your Kafka Broker hostname)

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka.broker.hostname -Djava.net.preferIPv4Stack=true"

Add below line in kafka-server-start.sh (in case of Hortonworks hadoop, path is /usr/hdp/current/kafka-broker/bin/kafka-server-start.sh)

export JMX_PORT=${JMX_PORT:-9999}

 

That’s it! Please do the above steps on all Kafka brokers and restart the kafka brokers ( manually or via management UI whatever applicable)

 

Verify that JMX port has been enabled!

 You can use jconsole to do so.

 

Download, install and configure jmxtrans

Download jmxtrans rpm from below link and install it using rpm command

http://code.google.com/p/jmxtrans/downloads/detail?name=jmxtrans-250-0.noarch.rpm&can=2&q=

 

Once you have installed jmxtrans, please make sure that java &jps configured in $PATH variable

 

Write a JSON for fetching MBeans on each Kafka Broker.

 

I have written JSON for monitoring custom Kafka metrics, please download it from here.

Please note that, you need to replace “IP_address_of_kafka_broker” with your kafka broker’s IP address in downloaded JSON, same is the case for ganglia server’s IP address.

 

Once you are done with writing JSON, please verify the syntax using any online JSON validator( http://jsonlint.com/ ).

 

Start the jmxtrans using below command

cd /usr/share/jmxtrans/
sh jmxtrans.sh start $name-of-the-json-file

Verify that jmxtrans has started successfully using simple “ps” command

Repeat above procedure on all Kafka brokers

 

Verify custom metrics

Login to ganglia server and go to rrd directory ( by default it is /var/lib/ganglia/rrds/ ) and check if there are new rrd files for kafka metrics.

 

You should see output like below (output is truncated)

kafka-ganglia-output

 

 

 

 

 

Go to ganglia web UI –>  select hadoopkafka from below highlighted dropdown

kafka-ganglia-output1

 

 

 

 

 

Select “custom.metrics” from below highlighted dropdown

kafka-ganglia-output2

 

 

 

 

 

That’s all! :-)

 

Note – You can choose which Mbeans to monitor by adding/removing them from json file.

 

Please share your feedback on this blog via comments! :-)

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather