How to create Jar using Maven:

In my previous post (Apache Spark IDE Setup with Scala with Maven & Spark Installation in Windows:) we have seen how to setup eclipse with scala & spark so that we can create and run scala objects. In this post we will see how to create jar out of eclipse with/wihtout Maven and submitting it to spark.

Creating Jar using Maven (mvn):

I am assuming that you have some scala code created in your maven project and ready to create jar. If you haven't created any program please look in to this Apache Spark IDE Setup with Scala with Maven. 

Now we will add maven dependencies and class path to pom.xml and will Compile, Test, Package maven project, for more details please look in to below links:
1. Maven pom.xml update for mvn dependencies: pom.xml basics & pom.xml update
2. Maven Build Life Cycle: Maven Build Life Cycle.

Let's get started by updating pom.xml:

Open your pom.xml and you will see pom.xml will have xml tags as below:

Now we will add build tags which will contain maven plugins, compiler level, Maven Assembly Plugin and MainClass in mainfest make a executable jar. I am pasting my maven project pom.xml for your reference.


__________________________________Starting of POM.XML____________________________
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>a-z.in.hadoop</groupId>
  <artifactId>Maven_Example1</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Maven_Example1</name>
  <url>http://maven.apache.org</url>

  <properties>
  <jdk.version>1.8</jdk.version>
  <junit.version>3.8.1</junit.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>

  <dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.4.1</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>


  <build>
<finalName>Maven_Example1</finalName>
<plugins>


<!-- download source code in Eclipse, best practice -->
  <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<version>4.0.0</version>
<configuration>
<downloadSources>true</downloadSources>
<downloadJavadocs>false</downloadJavadocs>
</configuration>


</plugin>

<!-- Set a compiler level -->

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>


<!-- Maven Assembly Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>a_z.in.hadoop.Maven_Example1.Line_Count</mainClass>
</manifest>
</archive>

</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase> <!-- bind to the packaging phase -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>

  </plugins>
  </build>
</project>

__________________________________End of POM.XML____________________________

If you know maven build life cycle hierarchy then you can directly crate mvn package. To all the others who are new to maven we can follow all the steps:

Right click on your maven project --> Run as --> you can select Maven clean as an direct option OR you can click on "Maven build..." to clean the project.

With this option it will clean the target folder under your project only if the Build is success.

Now we will create package using "Maven build..."


Right click on your maven project --> Run as --> Maven build... to crate package.

After successful Build we should be able to see target directory with jar created as shown below:

Note: Refresh your project after every Maven build, so that you can see the changes on eclipse.

Now we submit this jar to spark cluster using spark-submit:

Hoping that this post helps you to create maven project and creating jar out of it. Pleas comments on this if you have any questions.

Key Words: MAVEN , Maven , mvn , SPARK  , Spark, spark , Scala , scala , SCALA . Eclipse , eclipse , ECLIPSE , Hadoop , Big Data .