The Hadoop Development Tools (HDT) is a set of plugins for the Eclipse IDE for developing against the Hadoop platform.

Currently we are in the process of porting the existing MapReduce tools from the Apache Hadoop project to allow working with multiple versions of Hadoop from within one IDE.

Come get involved as we move towards our first release!

This project is currently a member of the Apache Incubator, so check back for updates, or come join us dev@hdt.incubator.apache.org.

Tools and Technologies used in this article :

  1. Apache Hadoop 2.2.0

  2. Spring Tool Suite 3.5.0

  3. Windows 7 OS

  4. JDK 1.6

  5. Apache Ant

 

1. Download

Download winghc/hadoop2x-eclipse-plugin zip.

 

2. Extract

Extract to a local directory (say, 'C:\hadoop2x-eclipse-plugin').

 

3. Build

  1. Open '<hadoop2x-eclipse-plugin-directory>\src\contrib\eclipse-plugin' in the Command prompt.

    C:\>cd C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin
    
  2. Run ANT build

    C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin>ant jar -Dversion=2.2.0 -Declipse.home=C:/IDE/sts-3.5.0 -Dhadoop.home=c:/hadoop
    Buildfile: C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin\build.xml
    
    check-contrib:
    
    init:
         [echo] contrib: eclipse-plugin
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\classes
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\test
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\system
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\system\classes
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\examples
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\test\logs
    
    init-contrib:
    
    ivy-download:
          [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
          [get] To: C:\hadoop2x-eclipse-plugin\ivy\ivy-2.1.0.jar
    
    ivy-probe-antlib:
    
    ivy-init-antlib:
    
    ivy-init:
    [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
    [ivy:configure] :: loading settings :: file = C:\hadoop2x-eclipse-plugin\ivy\ivysettings.xml
    
    ivy-resolve-common:
    
    ivy-retrieve-common:
    [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
    [ivy:cachepath] :: loading settings :: file = C:\hadoop2x-eclipse-plugin\ivy\ivysettings.xml
    
    compile:
         [echo] contrib: eclipse-plugin
        [javac] C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin\build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
        [javac] Compiling 45 source files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\classes
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
        [javac] c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0.jar(org/apache/hadoop/hdfs/DistributedFileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] Note: Some input files use or override a deprecated API.
        [javac] Note: Recompile with -Xlint:deprecation for details.
        [javac] Note: Some input files use unchecked or unsafe operations.
        [javac] Note: Recompile with -Xlint:unchecked for details.
        [javac] 7 warnings
    
    jar:
        [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying 9 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-app-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-app-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-common-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-core-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-core-2.2.0.jar
    
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-hs-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-plugins-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-hs-plugins-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-jobclient-2.2.0-tests.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-jobclient-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-shuffle-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-shuffle-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-examples-2.2.0.jar
         [copy] Copying 3 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\hadoop-common-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-common-2.2.0-tests.jar
         [copy] Copying c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-common-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\common\hadoop-nfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-nfs-2.2.0.jar
         [copy] Copying 3 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-2.2.0-tests.jar
         [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-nfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-nfs-2.2.0.jar
         [copy] Copying 11 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-api-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-api-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-distributedshell-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-applications-distributedshell-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-client-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-client-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-common-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-common-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-nodemanager-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-nodemanager-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-resourcemanager-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-resourcemanager-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-tests-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-tests-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-web-proxy-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-web-proxy-2.2.0.jar
         [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-site-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-site-2.2.0.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\protobuf-java-2.5.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\protobuf-java-2.5.0.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\log4j-1.2.17.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\log4j-1.2.17.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-cli-1.2.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-cli-1.2.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-configuration-1.6.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-configuration-1.6.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-lang-2.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-lang-2.5.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\jackson-core-asl-1.8.8.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\jackson-core-asl-1.8.8.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\jackson-mapper-asl-1.8.8.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\jackson-mapper-asl-1.8.8.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\slf4j-log4j12-1.7.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\slf4j-log4j12-1.7.5.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\slf4j-api-1.7.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\slf4j-api-1.7.5.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\guava-11.0.2.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\guava-11.0.2.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\hadoop-auth-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-auth-2.2.0.jar
         [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib
         [copy] Copying c:\hadoop\share\hadoop\common\lib\netty-3.6.2.Final.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\netty-3.6.2.Final.jar
          [jar] Building jar: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\hadoop-eclipse-plugin-2.2.0.jar
    
    BUILD SUCCESSFUL
    Total time: 42 seconds
    
    eclipse.home:
    Installation directory of Eclipse IDE.
    hadoop.home:
    Hadoop installation directory.

    Note: Internet connection is required as 'ivy-2.1.0.jar' will be downloaded.

 

4. Install

On successful build, 'hadoop-eclipse-plugin-2.2.0.jar' will be generated inside '<hadoop2x-eclipse-plugin-directory>\src\contrib\eclipse-plugin'. Copy this jar and paste it to the 'plugins' directory of your IDE.

 

5. Configure

Restart the Eclipse IDE if already started. Otherwise start it.

  1. Goto Window --> Open Perspective --> Other and select 'Map/Reduce' perspective.

  2. Click 'New Hadoop location...' (Blue Elephant icon) and define Hadoop location to run MapReduce applications. Click 'Finish' button.

    Define Hadoop location
    Map/Reduce(V2) Master:
    Address of the Map/Reduce master node (The Job Tracker).
    DFS Master:
    Address of the Distributed FileSystem Master node (The Name Node).

    To know the 'Port' numbers, start Hadoop and open http://localhost:8088/cluster in a browser. Click Tools --> Configuration and search for the following properties.

    DFS Master
    			<property>
    				<name>fs.defaultFS</name>
    				<value>hdfs://localhost:9000</value>
    				<source>core-site.xml</source>
    			</property>
    		
    Map/Reduce(V2) Master
    			<property>
    				<name>mapreduce.jobtracker.http.address</name>
    				<value>0.0.0.0:50030</value>
    				<source>mapred-default.xml</source>
    			</property>
    		
    Hadoop Configuration
  3. Now we can browse the Hadoop file system and perform different files / folder operations using the GUI only.

    DFS locations

    Also, we can easily create Map/Reduce Project, Mapper, Reducer and MapReduce Driver using the wizard (File --> New --> Other... --> Map/Reduce) and jump into Hadoop programming.

    Map/Reduce Project, Mapper, Reducer and MapReduce Driver

 

References