Rhipe Installation

Rhipe was first developed by Saptarshi Guha.
Rhipe needs R and Hadoop. So first install R and hadooop. Installation of R and hadoop are well explained in my previous posts. The latest version of Rhipe as of now is Rhipe-0.73.1. and  latest available version of R is R-3.0.0. If you are using CDH4 (Cloudera distribution of hadoop) , use Rhipe-0.73 or later versions, because older versions may not work with CDH4.
Rhipe is an R and Hadoop integrated programming environment. Rhipe integrates R and Hadoop. Rhipe is very good for statistical and analytical calculations of very large data. Because here R is integrated with hadoop, so it will process in distributed mode, ie  mapreduce.
Futher explainations of Rhipe are available in http://www.datadr.org/

Prerequisites

Hadoop, R, protocol buffers and rJava should be installed before installing Rhipe.
We are installing Rhipe in a hadoop cluster. So the job submitted may execute in any of the tasktracker nodes. So we have to install R and Rhipe in all the tasktracker nodes, otherwise you will face an exception “Cannot find R” or something similar to that.

Installing Protocol Buffer

Download the protocol buffer 2.4.1 from the below link

http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz

tar -xzvf protobuf-2.4.1.tar.gz

cd protobuf-2.4.1

chmod -R 755 protobuf-2.4.1

./configure

make

make install

Set the environment variable PKG_CONFIG_PATH

nano /etc/bashrc

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

save and exit

Then executed the following commands to check the installation

pkg-config --modversion protobuf

This will show the version number 2.4.1
Then execute

pkg-config --libs protobuf

This will display the following things

-pthread -L/usr/local/lib -lprotobuf -lz –lpthread

If these two are working fine, This means that the protobuf is properly installed.

Set the environment variables for hadoop

For example

nano /etc/bashrc

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_BIN=/usr/lib/hadoop/bin

export HADOOP_CONF_DIR=/etc/hadoop/conf

save and exit

Then


cd /etc/ld.so.conf.d/

nano Protobuf-x86.conf

/usr/local/lib   # add this value as the content of Protobuf-x86.conf

Save and exit

/sbin/ldconfig

Installing rJava

Download the rJava tarball from the below link.

http://cran.r-project.org/web/packages/rJava/index.html

The latest version of rJava available as of now is rJava_0.9-4.tar.gz

install rJava using the following command

R CMD INSTALL rJava_0.9-4.tar.gz

Installing Rhipe

Rhipe can be downloaded from the following link
https://github.com/saptarshiguha/RHIPE/blob/master/code/Rhipe_0.73.1.tar.gz

R CMD INSTALL Rhipe_0.73.1.tar.gz

This will install Rhipe

After this type R in the terminal

You will enter into R terminal

Then type

library(Rhipe)

#This will display

------------------------------------------------

| Please call rhinit() else RHIPE will not run |

————————————————

rhinit()

#This will display

Rhipe: Detected CDH4 jar files, using RhipeCDH4.jar
Initializing Rhipe v0.73
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/client-0.20/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/client/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Initializing mapfile caches

Now you can execute you Rhipe scripts.

Advertisements

About amalgjose
I am an Electrical Engineer by qualification, now I am working as a Software Engineer. I am very much interested in Electrical, Electronics, Mechanical and now in Software fields. I like exploring things in these fields. I like travelling, long drives and very much addicted to music.

108 Responses to Rhipe Installation

  1. Manju says:

    rJava installation gives an error message on centos 64 with hadoop cluster.
    The website does not talk about rJava installation.Please guide.
    Thanks

    • amalgjose says:

      Download the rJava tar ball and copy it to the linux machine. After installing R, execute the command
      R CMD INSTALL rJava.x.x.tar.gz
      This will install rJava. I have mentioned the same in the post. No need to extract the rJava tar ball and compile it manually. It may through JNI compiler issue due to the version difference in java.

  2. Lalit Jha says:

    rJava can also be installed with all dependencies from R console using command install.packages(“rJava”)

  3. Lalit Jha says:

    Thanks….Very good post … helpmed me to set u Rhipe. I was stuck in coniguring protobuf (ldconfig part), which is not mentioned in Rhipe installation doc.

    • amalgjose says:

      I actually wrote this post because I faced several issues while trying to install Rhipe. All those were explained in this blog. Thank you for reading my blog.. 🙂

  4. diviya says:

    how ro run rhipe code after installation

    • amalgjose says:

      Juz type R in the command line of R installed machine.. After that type library(Rhipe) , this will load Rhipe library. Then type rhinit(). After that you can execute your Rhipe program.. You can write it is a script and save it in a file and can execute using Rscript command.

  5. Peter K says:

    After
    >R
    >library(Rhipe)
    >rhinit()

    returns the following error:

    Initiailizing Rhipe v0.73
    Error in .jnew(“org/godhuli/rhipePersonalServer”)
    java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable

    Do you haev any solutions to this? Google did not help.

    • amalgjose says:

      This is due the issue in setting the env variables. Set HADOOP_HOME HADOOP_CMD HADOOP_BIN properly, this will work..

    • amalgjose says:

      Set the env variables HADOOP_BIN hadoop_home hadoop_cmd correctly. This will work. This problem happened bcz it is unable to find the hadoop libraries. Can you please post the values you set for these env variables. Are u using R studio.?

    • Doreswamy says:

      Dear I have been getting the same kind of error when i installed Rhipe and intializing. Please send me the answer if any thing you got it.

      • amalgjose says:

        Hi Doreswamy,

        This error is coming because the Rhipe is not able to find out the location where hadoop-common*.jar is present.
        Check your hadoop env variables properly. Else put that jar in your Rhipe lib directory.

    • Doreswamy says:

      > library(Rhipe)
      ————————————————
      | Please call rhinit() else RHIPE will not run |
      ————————————————
      > rhinit()
      Rhipe: Using Rhipe.jar file
      Initializing Rhipe v0.73
      Error in .jnew(“org/godhuli/rhipe/PersonalServer”) :
      java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
      >

      Dear I have been getting the above kind of error when i installed Rhipe and start intializing. Please send me the answer if any thing you got it.

      How ever the following variables have been set in .bashrc file in home directory
      Ubuntu 14.04.1, open JDK with 32 bit JVM

      #HADOOP VARIABLES END

      export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
      export HADOOP_INSTALL=/usr/local/hadoop
      export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin/java
      export PATH=$PATH:$JAVA_HOME/bin
      export PATH=$PATH:$HADOOP_INSTALL/bin
      export PATH=$PATH:$HADOOP_INSTALL/sbin
      export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
      export HADOOP_COMMON_HOME=$HADOOP_INSTALL
      export HADOOP_HDFS_HOME=$HADOOP_INSTALL
      export YARN_HOME=$HADOOP_INSTALL
      export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
      export HADOOP_OPTS=-Djava.library.path=$HADOOP_INSTALL/lib

      #HADOOP VARIABLES END

      export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server
      export HADOOP_PREFIX=/usr/local/hadoop
      export HADOOP_CMD=/usr/local/hadoop/bin/hadoop
      export HADOOP_STREAMING=/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar

      # Hbase Installation
      export HBASE_HOME=/usr/local/Hbase
      export PATH=$PATH:$HBASE_HOME/bin
      # export HADOOP_LIB = $HADOOP_INSTALL/lib
      export HADOOP_HOME=/usr/local/hadoop/share/hadoop:$HADOOP_HOME/common:$HADOOP_HOME/common/hdfs
      export R_HOME=/usr/local/lib/R/site-library
      # export HADOOP_HOME=/home/hduser/hdfs_data/namenode/current
      # export HADOOP_HOME=/home/hduser/hdfs_data/namenode
      export HADOOP_BIN=/usr/local/hadoop/bin
      export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
      export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig/

  6. diviya says:

    i installed Rhipe in linux terminal but i cant access library(Rhipe) in R as it throwing error: library(Rhipe)
    Error in library(Rhipe) : there is no package called âRhipeâ

    • amalgjose says:

      Check the env variable R_HOME. Go to R_HOME/library and check whether Rhipe is present or not. In your case it may not be there.. Go to /usr/local/lib64/R/library and check whether Rhipe is present or not. If it is there, set R_HOME as /usr/local/lib64/R and refresh and try again. This will work. Good luck.. 🙂

  7. Kiran Kaipa says:

    Amal,
    Thanks for summarizing the setup – I too faced issues setting up RHIPE. One question, did you install R and RHipe on all the nodes?
    For map-reduce jobs, do we need all nodes having the two components?
    Thanks!
    Kiran

  8. Shrida says:

    Hello,

    I am getting the following error when i run rhinit()
    My configuration is as below:

    Hadoop 2.2
    Openjdk 7
    rJava installed
    R 2.15.1

    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, : java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

    Thanks and Regards,
    Shrida

    • amalgjose says:

      You are using YARN. I am not sure whether the Rhipe version using by you are compatible with hadoop-2.2. The doc mentioned by me will work with CDH4 only.

      • bedantaguru says:

        Hi Amal
        Thanks for nice tutorial.
        I’m trying to install Rhipe stable version as on date [0.73.1] with following setup
        Ubuntu 14.04 LTS
        Java 1.8.0_40-b25
        Hadoop 2.6.0 with YARN [Usual distribution not CDH 5]
        Google Protocol Buffer 2.4.1

        I think I’m also getting above error. So please help me understanding the issue. In case you have any update like when Rhipe is going to support YARN or any workaround please let us know,

      • amalgjose says:

        Hi,
        Rhipe supports YARN. The version that you are using is not compatible with YARN.
        Various versions of Rhipe are available in the below repository.
        http://ml.stat.purdue.edu/rhipebin/
        rJava error may come if there is any version mismatch. Not all rJava versions are compatible with Rhipe.

        Try executing the below script for rJava. Copy this to a file and execute that file as a shell script. This will be easier

        sudo R CMD javareconf
        sudo su – -c “R -e \”install.packages(‘rJava’, repos=’http://www.rforge.net/’)\””
        sudo su – -c “R -e \”install.packages(‘codetools’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘lattice’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘MASS’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘boot’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘shiny’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”options(repos = ‘http://cran.rstudio.com/’); library(devtools); install_github(‘hafen/housingData’)\””
        sudo su – -c “R -e \”install.packages(‘maps’,repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘mixtools’, repos=’http://cran.rstudio.com/’)\””
        sudo su – -c “R -e \”install.packages(‘lubridate’, repos=’http://cran.rstudio.com/’)\””

      • bedantaguru says:

        Hi Amal,
        Thanks for immediate help. I have not tried yet due to some technical difficulties in accessing the PC I was working.
        But just asking what all these command do. I have installed latest rJava and few packages which you mentioned were installed also.
        So just curious what will the command do. Also if possible can you guide me which version of Rhipe will work with 2.6 ? [I’m trying with Rhipe_0.75.1.3_hadoop-2.tar.gz. As my hadoop is not CDH 5]

      • amalgjose says:

        Hi,
        The command that I had sent you will reconfigure the rJava.
        For vanilla apache hadoop 2.6, Rhipe_0.75.1.3_hadoop-2.tar.gz will be perfect. For Rhipe 0.75, the versions of rJava and protobuf are different. The versions I have mentioned in my blog is only for Rhipe 0.73.

        Regards,
        Amal G Jose

  9. Zach says:

    Hi Amal,

    I am running rhinit(), and R says that Rhipe is initializing but it never finishes. As a result, I can’t run any other commands in R, or even quit R without exiting the putty window. It also provides no error statement. (There were other errors earlier.)

    We updated all the environment variables but we are having trouble identifying what the current issue is. Google does not provide any useful or helpful suggestions.

    Do you have any idea what could be causing RHIPE to never finish initializing? Or what we can do to fix it?

    Thanks,
    Zach

  10. Jun says:

    Excuse me dear Amal,

    I actually well copied your step that I eventually stayed at just before rhinit() command in the R shell. but I encountered an error line below. and my environment is 1NN /1 SNN / 5 data nodes with CentOS 6.4 / CDH standard v.4.5. and all the R related version is followed your condition.
    May I ask your help to cure it?

    > rhinit()
    Rhipe: Using RhipeCDH4.jar
    Initializing Rhipe v0.73
    java.io.IOException: failure to login
    at java.lang.Throwable.(libgcj.so.10)
    at java.lang.Throwable.(libgcj.so.10)
    at java.lang.Exception.(libgcj.so.10)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:694)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:573)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2464)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2456)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2323)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
    at org.godhuli.rhipe.PersonalServer.run(PersonalServer.java:321)
    at java.lang.reflect.Method.invoke(libgcj.so.10)
    at RJavaTools.invokeMethod(RJavaTools.java:386)
    Caused by: javax.security.auth.login.LoginException: could not load module com.sun.security.auth.module.UnixLoginModule
    at javax.security.auth.login.LoginContext.lookupModule(libgcj.so.10)
    at javax.security.auth.login.LoginContext.(libgcj.so.10)
    at org.apache.hadoop.security.UserGroupInformation.newLoginContext(UserGroupInformation.java:527)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:672)
    …9 more
    Caused by: java.lang.ClassNotFoundException
    at RJavaClassLoader.findClass(RJavaClassLoader.java:383)
    at java.lang.ClassLoader.loadClass(libgcj.so.10)
    at java.lang.ClassLoader.loadClass(libgcj.so.10)
    at java.lang.Class.forName(libgcj.so.10)
    at javax.security.auth.login.LoginContext.lookupModule(libgcj.so.10)
    …12 more
    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
    java.lang.NullPointerException

  11. prashant says:

    > rhinit()
    Initializing Rhipe v0.73
    Error in .jnew(“org/godhuli/rhipe/PersonalServer”) :
    java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
    Hi amol, I am getting this error in rstudio on calling rhinit()
    then as per you suggested plz tell me what is mean by “HADOOP_CMD”

    • amalgjose says:

      This error is coming due to improper hadoop env variables..
      Set HADOOP_HOME properly. It is the location where the hadoop-core jar is present. Set hadoop_bin also. Then try again. This error will get resolved once you point your hadoop-core jar properly.

  12. Jenny McFadden says:

    Hi, thanks for the great tutorial! I am also having the same error as Prashant. Here are my relevant environment variables:

    PKG_CONFIG_PATH=/usr/lib/protobuf-2.4.1
    export PKG_CONFIG_PATH
    HADOOP_HOME=/usr/lib/hadoop-0.20-mapreduce
    export HADOOP_HOME
    HADOOP=$HADOOP_HOME
    export HADOOP
    HADOOP_CONF_DIR=$HADOOP/conf
    export HADOOP_CONF_DIR
    HADOOP_BIN=$HADOOP/bin
    export HADOOP_BIN
    HADOOP_CMD=$HADOOP
    export HADOOP_CMD
    PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$MAVEN_HOME/bin:$MAHOUT_HOME:$HADOOP_HOME
    export PATH

    Note I’m running CDH4 MRv1 on a 32-bit CentOS 6 installation, with jdk-1.7.0-51. I get the error when I try to run rhinit() in R command line or R studio.

    Do you have any suggestions for me? Thank you!

    • amalgjose says:

      Hi Jenny,
      better if you set the env variables like

      export HADOOP_HOME=/usr/lib/hadoop-0.20-mapreduce
      export HADOOP_CONF_DIR=/etc/hadoop/conf
      export HADOOP_BIN=/usr/lib/hadoop-0.20-mapreduce/bin
      export HADOOP_CMD=/usr/lib/hadoop-0.20-mapreduce

      In your case, the problem is happening because you are not setting the HADOOP_HOME to the location where hadoop-core jar exists. Set it properly.

      After setting this, refresh the file in which you set the evn variables (/etc/bashrc or ~/.bashrc)
      Please try this.

      • Jenny McFadden says:

        Amal, thanks for your reply! I’ve modified things a little, and am still getting errors. I do have HADOOP_HOME pointing to the dir where hadoop-core.jar exists, but am still getting NoClassDefFoundError errors:

        > library(Rhipe)
        ————————————————
        | Please call rhinit() else RHIPE will not run |
        ————————————————
        > rhinit()
        Rhipe: Using RhipeCDH4.jar
        Initializing Rhipe v0.73
        log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
        java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/FileOutputFormat
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at RJavaClassLoader.findClass(RJavaClassLoader.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at org.godhuli.rhipe.PersonalServer.run(PersonalServer.java:322)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at RJavaTools.invokeMethod(RJavaTools.java:386)
        Caused by: java.lang.ClassNotFoundException
        at RJavaClassLoader.findClass(RJavaClassLoader.java:383)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        … 18 more
        Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
        java.lang.NullPointerException

        output (in R) of .jclassPath():

        > .jclassPath()
        [1] “/home/cloudera/R/i686-redhat-linux-gnu-library/3.0/rJava/java”
        [2] “/home/cloudera/R/i686-redhat-linux-gnu-library/3.0/Rhipe/java/guava-13.0.1.jar”
        [3] “/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.6.0.jar”
        [4] “/usr/lib/hadoop/hadoop-annotations.jar”
        [5] “/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.6.0.jar”
        [6] “/usr/lib/hadoop/hadoop-auth.jar”
        [7] “/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0-tests.jar”
        [8] “/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.6.0.jar”
        [9] “/usr/lib/hadoop/hadoop-common.jar”
        [10] “/usr/lib/hadoop/lib/activation-1.1.jar”
        [11] “/usr/lib/hadoop/lib/asm-3.2.jar”
        [12] “/usr/lib/hadoop/lib/avro-1.7.4.jar”
        [13] “/usr/lib/hadoop/lib/cloudera-jets3t-2.0.0-cdh4.6.0.jar”
        [14] “/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar”
        [15] “/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar”
        [16] “/usr/lib/hadoop/lib/commons-cli-1.2.jar”
        [17] “/usr/lib/hadoop/lib/commons-codec-1.4.jar”
        [18] “/usr/lib/hadoop/lib/commons-collections-3.2.1.jar”
        [19] “/usr/lib/hadoop/lib/commons-compress-1.4.1.jar”
        [20] “/usr/lib/hadoop/lib/commons-configuration-1.6.jar”
        [21] “/usr/lib/hadoop/lib/commons-digester-1.8.jar”
        [22] “/usr/lib/hadoop/lib/commons-el-1.0.jar”
        [23] “/usr/lib/hadoop/lib/commons-httpclient-3.1.jar”
        [24] “/usr/lib/hadoop/lib/commons-io-2.1.jar”
        [25] “/usr/lib/hadoop/lib/commons-lang-2.5.jar”
        [26] “/usr/lib/hadoop/lib/commons-logging-1.1.1.jar”
        [27] “/usr/lib/hadoop/lib/commons-math-2.1.jar”
        [28] “/usr/lib/hadoop/lib/commons-net-3.1.jar”
        [29] “/usr/lib/hadoop/lib/guava-11.0.2.jar”
        [30] “/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar”
        [31] “/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar”
        [32] “/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar”
        [33] “/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar”
        [34] “/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar”
        [35] “/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar”
        [36] “/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar”
        [37] “/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar”
        [38] “/usr/lib/hadoop/lib/jersey-core-1.8.jar”
        [39] “/usr/lib/hadoop/lib/jersey-json-1.8.jar”
        [40] “/usr/lib/hadoop/lib/jersey-server-1.8.jar”
        [41] “/usr/lib/hadoop/lib/jets3t-0.6.1.jar”
        [42] “/usr/lib/hadoop/lib/jettison-1.1.jar”
        [43] “/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar”
        [44] “/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar”
        [45] “/usr/lib/hadoop/lib/jline-0.9.94.jar”
        [46] “/usr/lib/hadoop/lib/jsch-0.1.42.jar”
        [47] “/usr/lib/hadoop/lib/jsp-api-2.1.jar”
        [48] “/usr/lib/hadoop/lib/jsr305-1.3.9.jar”
        [49] “/usr/lib/hadoop/lib/junit-4.8.2.jar”
        [50] “/usr/lib/hadoop/lib/kfs-0.3.jar”
        [51] “/usr/lib/hadoop/lib/log4j-1.2.17.jar”
        [52] “/usr/lib/hadoop/lib/mockito-all-1.8.5.jar”
        [53] “/usr/lib/hadoop/lib/paranamer-2.3.jar”
        [54] “/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar”
        [55] “/usr/lib/hadoop/lib/protobuf-java-2.4.1.jar”
        [56] “/usr/lib/hadoop/lib/servlet-api-2.5.jar”
        [57] “/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar”
        [58] “/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar”
        [59] “/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar”
        [60] “/usr/lib/hadoop/lib/stax-api-1.0.1.jar”
        [61] “/usr/lib/hadoop/lib/xmlenc-0.52.jar”
        [62] “/usr/lib/hadoop/lib/xz-1.0.jar”
        [63] “/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh4.6.0.jar”
        [64] “/usr/lib/hadoop/parquet-avro-1.2.5-cdh4.6.0.jar”
        [65] “/usr/lib/hadoop/parquet-cascading-1.2.5-cdh4.6.0.jar”
        [66] “/usr/lib/hadoop/parquet-column-1.2.5-cdh4.6.0.jar”
        [67] “/usr/lib/hadoop/parquet-common-1.2.5-cdh4.6.0.jar”
        [68] “/usr/lib/hadoop/parquet-encoding-1.2.5-cdh4.6.0.jar”
        [69] “/usr/lib/hadoop/parquet-format-1.0.0-cdh4.6.0.jar”
        [70] “/usr/lib/hadoop/parquet-generator-1.2.5-cdh4.6.0.jar”
        [71] “/usr/lib/hadoop/parquet-hadoop-1.2.5-cdh4.6.0.jar”
        [72] “/usr/lib/hadoop/parquet-hive-1.2.5-cdh4.6.0.jar”
        [73] “/usr/lib/hadoop/parquet-pig-1.2.5-cdh4.6.0.jar”
        [74] “/usr/lib/hadoop/parquet-pig-bundle-1.2.5-cdh4.6.0.jar”
        [75] “/usr/lib/hadoop/parquet-scrooge-1.2.5-cdh4.6.0.jar”
        [76] “/usr/lib/hadoop/parquet-test-hadoop2-1.2.5-cdh4.6.0.jar”
        [77] “/usr/lib/hadoop/parquet-thrift-1.2.5-cdh4.6.0.jar”
        [78] “/home/cloudera/R/i686-redhat-linux-gnu-library/3.0/Rhipe/java/RhipeCDH4.jar”

        Any ideas? Thank you!

  13. Jenny McFadden says:

    Update: if I run .jinit() the error stack changes to:

    > .jinit()
    [1] 0
    > rhinit()
    Rhipe: Using RhipeCDH4.jar
    Initializing Rhipe v0.73
    java.lang.NoClassDefFoundError: org/godhuli/rhipe/RHMapFileOutputFormat
    at org.godhuli.rhipe.PersonalServer.run(PersonalServer.java:322)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at RJavaTools.invokeMethod(RJavaTools.java:386)
    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
    java.lang.NullPointerException

    org/godhuli/rhipe/RHMapFileOutputFormat is in RhipeCDH4.jar, which is appears to be able to see. Is there a reason it can’t find that class file? According to my classpath output, it sees it:

    [78] “/home/cloudera/R/i686-redhat-linux-gnu-library/3.0/Rhipe/java/RhipeCDH4.jar”

  14. Kuntal Ghosh says:

    Hi Amal,

    Have you tried installing RHIPE on CDH5 running YARN? I can see they have uploaded one Rhipe_0.75.0_cdh5mr2.tar.gz on their website http://ml.stat.purdue.edu/rhipebin/. However nowhere can I find any relevant instructions for running it on YARN. Any insight is highly appreciated.

    Regards,
    Kuntal

  15. Subramaniyabharathi says:

    Hi amal,
    This is Subramaniyabharathi. Student from IITMadras. I am getting the below error when i execute rhinit(). Can you please suggest some way to overcome this. Thanks in advance

    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    14/12/31 11:12:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    14/12/31 11:12:07 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    14/12/31 11:12:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
    java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

    • amalgjose says:

      Hi,
      Can you tell me some more details such as hadoop version, R version and Rhipe version.

      Regards,
      Amal

      • Subramaniyabharathi says:

        Hi Amal, I am using Apache Hadoop 2.5.1, R 3.1.2 and Rhipe 0.73.1

      • Subramaniyabharathi says:

        Hi Amal,
        I am using Apache Hadoop 2.5.1, R 3.1.2 and Rhipe 0.73.1. Also I am running Hadoop as a multinode cluster with three datanodes for a project. do i need to install rhipe on every node

      • Subramaniyabharathi says:

        Thanks for your reply. will try that.

    • amalgjose says:

      Seems like the error is because of version compatibility issues.
      You are using the Rhipe build for MR1 hadoop. Rhipe now works with YARN also. I haven’t tested it.
      Build the Rhipe for YARN from the source code available in this github repo.

      https://github.com/tesseradata/RHIPE/

      The details for building for YARN is described there. You can check the pom.xml and modify it according to your requirement.

      Regards,
      Amal G Jose

    • amalgjose says:

      Some ready made installable for some specific version of hadoop are already available in the following URL.
      The mrv2 version is specific for CDH5.
      http://ml.stat.purdue.edu/rhipebin/

      • doreswamyh hosahalli says:

        Dear Amul,

        May Light always surround you; Hope kindle and rebound you. May your Hurts turn to Healing; Your Heart embrace Feeling. May Wounds become Wisdom; Every Kindness a Prism. May Laughter infect you; Your Passion resurrect you. May Goodness inspire  your Deepest Desire. Through all that you Reach For,  May your arms Never Tire. This is the new year the new you. You can pass through another year, coasting on cruise control. Or you can step out of your comfort zone, trying things you have never done before, and make 2015 as the year that you elevate from where you are & soar high. Make it happen! WISH YOU A VERY HAPPY AND PEACEFUL NEW YEAR-2015  Thanks and Regards, Dr. Doreswamy,Ph.D  Professor, Department P.G Studies and Research in Computer Science, Mangalore University,Mangalagangotri- 574 199, Karnataka,INDIA.Mobile:91-9845032853Office:0824-2284607 

      • amalgjose says:

        Thank you very much.
        Happy New Year ..:)

  16. Manohar says:

    I am getting the bellow error, Can anyone help please….

    > rhinit()
    Initializing Rhipe v0.73
    Error in .jnew(“org/godhuli/rhipe/PersonalServer”) :
    java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
    >

    • amalgjose says:

      Hi,
      This is because the Rhipe is not able to find the hadoop jars. This happened because of improper setting of environment variable.
      Check the HADOOP_HOME, HADOOP_BIN and HADOOP_CMD values. Sorry for the delay in response.

      Regards,
      Amal

  17. majedchaffai says:

    HI thank you for this job , I’m getting an arror :
    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
    java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.setUserClassesTakesPrecedence(Z)V

    when i run myscript.R (map an reduce)
    library(Rhipe)

    rhinit()

    Map(function(k,v){
    x <- runif(v)
    rhcollect(k, c(Min=min(x),Max=max(x)))
    })
    rhoptions(HADOOP.TMP.FOLDER = "/app/hadoop/tmp")

    job<-rhwatch(
    map=Map,
    input=10,
    reduce=0,
    output="/app/hadoop/test",
    jobname="test")
    rhex(job)

    my platform is hadoop 1.0.3

    Can you show me how to fix it.

  18. majedchaffai says:

    thank you very much, I’ll try cloudera.

  19. Thomas V Joseph says:

    Hi Amal

    Was trying to install the protocol buffer files in the Cloudera image in Ubuntu for integrating with RHIpe.

    However , when I try to execute the “make” command it says ” make *** no targets specified and no make file found . Stop”

    I tried various things but the same error

    I use CDH3 and the R version installed in the Ubuntu environment is R-2.15.1

    I use a Windows 7 system.

    I also have R and R studio installed in the Windows environment.

    Please suggest what to be done

    Rgd
    Thomas

    • amalgjose says:

      Hi,
      Check for an autogen.sh file inside the protobuf. If present, execute that first. For the execution of autogen, you need the libraries automake and libtools. So install that first. Rhipe is an R library that works on top of hadoop using hadoop streaming. Rhipe needs hadoop and R. So you have to install R and Rhipe on all the nodes of hadoop cluster. Try this option and update me.

      Regards,
      Amal

      • Thomas V Joseph says:

        Hi Amal

        autogen.sh is present inside the protobuf. However, please let me know how I install automake and libtools in the cloudera image of vmware. Tried to search vairous resources, but couldnt get any instructions

        Rgds
        Thomas

      • amalgjose says:

        Hi,
        If it is an ubuntu VM, you can install automake using the following command
        sudo apt-get install automake

        Install autoconf
        sudo apt-get install autoconf

        Install libtool
        sudo apt-get install libtool

        This will help you. You need internet connection on your machine for doing this process. If it is a centos vm, use yum install command instead of apt-get install command.

        Regards,
        Amal G Jose

  20. Thomas V Joseph says:

    Thanks Amal. I installed it.

    But how do I execute the autogen.sh after installing these tools ??

    Sorry for asking all these trivial questions. I am new to ubantu and vmware

    Rgds
    Thomas

    • amalgjose says:

      chmod u+x autpgen.sh
      sh autogen.sh

      This will work.. 🙂

      • Thomas V Joseph says:

        Hi Amal

        I executed the above commands . The first one worked. After executing the second command I got the following message
        “autogen.sh : 1 autoreconf:not found”

        Does this indicate any problem

        Anyway after this step, I did the following steps as mentioned in your blog

        cd protobuf-2.4.1

        chmod -R 755 protobuf-2.4.1

        When I executed the chmod command it says “Cannot access ‘Protobuf-2.4.1 No such file or directory”

        However I was on the Protobuf directory

        Besides, I need to update you that, R has been installed in the root directory and not in the protobuf directory.

        Please help
        Rgds
        Thomas

      • amalgjose says:

        Do the command chmod -R 755 protobuf-2.4.1 outside the protobuf-2.4.1 directory.

  21. Thomas V Joseph says:

    Thanks Amal. I did that and ran the ./configure command. The new error at the end of a long list of messages says

    “Configure error : in ‘/home/cloudera/protobuf-2.4.1’
    “Configure error : C++ preprocessor “lib/cpp” fails sanity check

    • amalgjose says:

      Seems like the c ad cpp libs are not present. Install it using
      sudo apt-get install gcc

    • Doreswamy says:

      Dear Rhipe User

      Please do the following you can install the protocol:

      At the command prompt $ chmod -R 755 protobuf-2.4.1

      It recursively set rwxr-xr-x permissions to all the files and folders present in the directory .protobuf-2.4.1

      After that try to install Rhipe package. It will work smoothly. After that it may ask you set the java properly.

      Alternatively Go for installation R-Hadoop(rmr2, rhdf,rhbase) in Linux environment. It provides the same environment for r programming. It good for big data analysis.

      Thanks.

  22. kota says:

    Hi,… please let us know whether Rhipe is compatible with YARN, unable to proceed with YARN. Let me know if any document available. Thanks.

  23. Balasairamkumar says:

    Hi Amal

    when i ran rhinit() i am getting this error

    Initializing Rhipe v0.75.1
    Error in .jnew(“org/godhuli/rhipe/PersonalServer”) :
    java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable

    and I am using Hadoop-2.3.0, Rhipe_0.75.1 and protobuf 2.5.0

    I read all your comments. You asked to change the HADOOP_HOME variable to the location where hadoop-core.jar is present. But later i found out for hadoop 2.0, no hadoop-core file is there. So now what i supposed to do. These are my environment variables

    export HADOOP_PREFIX=/home/hdfs/hadoop-2.3.0
    export HADOOP_HOME=$HADOOP_PREFIX
    export HADOOP_BIN=$HADOOP_PREFIX/bin
    export HADOOP_COMMON_HOME=$HADOOP_PREFIX
    export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
    export HADOOP_HDFS_HOME=$HADOOP_PREFIX
    export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
    export HADOOP_YARN_HOME=$HADOOP_PREFIX
    export HADOOP_LIBS=$HADOOP_PREFIX/share/hadoop/common:$HADOOP_PREFIX/share/hadoop/hdfs:$HADOOP_PREFIX/share/hadoop/mapreduce
    export HADOOP_LIBS=$HADOOP_LIBS:$HADOOP_PREFIX/share/hadoop/httpfs:$HADOOP_LIBS:$HADOOP_PREFIX/share/hadoop/tools

    export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
    export HADOOP_OPTS=”-Djava.library.path=$HADOOP_PREFIX/lib”
    export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
    export HADOOP_OPTS=”-Djava.library.path=$HADOOP_PREFIX/lib”
    export LD_LIBRARY_PATH=/usr/local/lib:/usr/lib/jvm/jdk1.8.0_31/lib/amd64/jli
    export PKG_CONFIG_PATH=/usr/local/lib
    export HADOOP_BIN=$HADOOP_PREFIX/bin

    export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_31/
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin
    export CLASSPATH=/usr/lib/jvm/jdk1.8.0_31/lib

    for the past 2 days i’m trying this.. can u help me in sorting this out..

    • amalgjose says:

      Hi,
      org/apache/hadoop/io/Writable class is present in hadoop-commons jar. So add that jar to the location where you specified the HADOOP_HOME

      • Balasairamkumar says:

        Thank you Amal for your quick help. I did the correction.
        But just now i noticed when i load Rhipe in Rstudio i am getting warning messages like

        Rhipe requires HADOOP_HOME or HADOOP or HADOOP_BIN environment variable to be present
        $HADOOP/bin/hadoop or $HADOOP_BIN/hadoop should exist
        Rhipe: HADOOP_BIN is missing, using $HADOOP/bin
        HADOOP_HOME missing
        HADOOP_CONF_DIR missing, you are probably going to have a problem running RHIPE.

        But yes.. i have given all these details in .bashrc file in the home folder of the user account using which i am using Rstudio. But still i am getting this warning means Rstudio couldn’t read them.

        So where i should keep them so that Rstudio can get them properly.

      • Balasairamkumar says:

        Thanks for your reply i added the environment variables in Renviron.site file..

        Now variables have been taken by R. But now i am getting this error.

        Error in .jnew(“org/godhuli/rhipe/PersonalServer”) :
        java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory

        again i searched and found this will be in hadoop-core.jar..

        now which file i should choose..

      • amalgjose says:

        This is because of commons-logging.jar. Add it to the proper place.

      • Balasairamkumar says:

        Thanks again.. I did it.. but yes.. once i do that i’m getting this error

        java.lang.NoClassDefFoundError: Could not initialize class org.godhuli.rhipe.PersonalServer

        This i din’t get anywhere in the hadoop directory.

      • amalgjose says:

        This class is inside Rhipe jar. So check the location where the Rhipe jar is present.

      • balasairamkumar says:

        Thank you.. after that similarly many NoClassDefFoundError came.. Solved it.. But finally got stuck at

        NoClassFoundError: org/apache/hadoop/yarn/client/ClientRMProxy

        I searched and found that.. When updating from Hadoop-2.2.0 to Hadoop2.3.0 they removed this ClientRMProxy.. Is it true.. Or what should I do for this error..

      • amalgjose says:

        Check for hadoop-yarn-client-2.x.x.jar.

      • Balasairamkumar says:

        Hi amal, thanks for your timely replies..

        I used Hadoop.yarn.client-2.2.0.jar.. to get away with org/apache/hadoop/yarn/client/ClientRMProxy error.

        But now it is showing

        NoClassDefFoundError: org/apache/hadoop/yarn/client/RMProxy

        I couldnot find which jar has this class..

        Am i chosen the hadoop version wrongly to use Rhipe.. Which hadoop is best suited for Rhipe. At leaset i can start that version of hadoop setting from scratch and get away these kind of error chains..

  24. Chandan says:

    Hi Amal,
    I have installed below softwares
    Ubuntu 14.04 LTS i686 32 bit
    hadoop-1.2.1 (/usr/local/hadoop)
    java-8-oracle (/usr/lib/jvm/java-8-oracle)
    rJava_0.9-4 (/usr/local/lib/R/site-library)
    Rhipe_0.73.1 (/usr/local/lib/R/site-library)
    hbase-0.94.27 (/usr/local/hbase)
    R version 3.0.2 (/usr/local/lib/R)
    protobuf-2.4.1 (/usr/local/hadoop)

    I have refered your blog and done with all the configuration and enviroment variable settings. but when I run rhinit() it will give below error. Please can you help me. thanks.

    > rhinit()
    Rhipe: Using Rhipe.jar file
    Initializing Rhipe v0.73
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    15/04/13 22:04:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    15/04/13 22:04:33 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    15/04/13 22:04:33 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
    java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

    • amalgjose says:

      Which version of Java are you using. ?
      I tested this with jdk 1.6.

      • Chandan says:

        java version “1.8.0_40”

      • amalgjose says:

        Can you downgrade your java to 1.6

      • Chandan says:

        Hi Amal,
        I have tried with java-6-oracle but I got below result

        > rhinit()
        Rhipe: Using Rhipe.jar file
        Initializing Rhipe v0.73
        Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, :
        java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

        can you provide link to download for jdk1.6 which you have installed. Mine is ubuntu14.04 32 bit OS.
        Thanks

  25. Chandan says:

    I tried with jdk 1.6 aslo. I got platform error so I went for higher version. can you access remotely?

  26. anuraginitmJ says:

    Hi Amal,

    First of all its a very good post and detailed.
    I am trying to follow the post as you have mentioned and installed R (yum install), protocol buffer 2.4.1, latest rJava on CDH5.3. I have also updated the PKG_CONFIG_PATH to point to the /usr/local/lib/pkgconfig directory.
    I have also created the Protobuf-x86_64.conf (the VM i am using is 64 bit).
    But still when i am trying to install Rhipe i am getting below error –

    Package protobuf was not found in the pkg-config search path.
    Perhaps you should add the directory containing `protobuf.pc’
    to the PKG_CONFIG_PATH environment variable
    No package ‘protobuf’ found

    Any suggestions dost?

    • anuraginitmJ says:

      One more thing, I have installed for rJava since from terminal it was giving some error related to JAVA_HOME.

    • amalgjose says:

      Hi can you check whether the PKG_CONFIG_PATH contains anything.
      Did u installed protobuf package properly. Can you check the installation again.?

      • anuraginitmJ says:

        Below is the output, please see if this makes sense (its a single node cluster installation of Hadoop on a centos 6.5 based VM having 32 bit architecture) –

        [edureka@localhost ~]$ ls /usr/local/lib/pkgconfig/
        protobuf-lite.pc protobuf.pc
        [edureka@localhost ~]$ echo $PKG_CONFIG_PATH
        /usr/local/lib/pkgconfig

        [edureka@localhost ~]$ pkg-config –modversion protobuf
        2.4.1
        [edureka@localhost ~]$ pkg-config –libs protobuf
        -pthread -L/usr/local/lib -lprotobuf -lz -lpthread
        [edureka@localhost ~]$

      • anuraginitmJ says:

        Output for the sudo R CMD INSTALL Rhipe…

        * installing to library ‘/usr/lib/R/library’
        * installing *source* package ‘Rhipe’ …
        ** libs
        g++ -m32 -I/usr/include/R -DNDEBUG -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -I. -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -DUSEAUTOSHORT -DHAVE_UINTPTR_T `/usr/lib/R/bin/R CMD config –cppflags` `pkg-config –cflags protobuf` -c rexp.pb.cc -o rexp.pb.o
        Package protobuf was not found in the pkg-config search path.
        Perhaps you should add the directory containing `protobuf.pc’
        to the PKG_CONFIG_PATH environment variable
        No package ‘protobuf’ found
        g++ -m32 -I/usr/include/R -DNDEBUG -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -I. -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -DUSEAUTOSHORT -DHAVE_UINTPTR_T `/usr/lib/R/bin/R CMD config –cppflags` `pkg-config –cflags protobuf` -c message.cc -o message.o
        Package protobuf was not found in the pkg-config search path.
        Perhaps you should add the directory containing `protobuf.pc’
        to the PKG_CONFIG_PATH environment variable
        No package ‘protobuf’ found
        message.cc: In function ‘void writeSexp64(FILE*, REXP*, SEXPREC*)’:
        message.cc:253: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result
        message.cc: In function ‘void writeSexp32(FILE*, REXP*, SEXPREC*)’:
        message.cc:295: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result
        g++ -m32 -I/usr/include/R -DNDEBUG -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -I. -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -DUSEAUTOSHORT -DHAVE_UINTPTR_T `/usr/lib/R/bin/R CMD config –cppflags` `pkg-config –cflags protobuf` -c fileio.cc -o fileio.o
        Package protobuf was not found in the pkg-config search path.
        Perhaps you should add the directory containing `protobuf.pc’
        to the PKG_CONFIG_PATH environment variable
        No package ‘protobuf’ found

      • amalgjose says:

        Are you sure that the protobuf installation was success.?. Can you install the protobuf again.

      • anuraginitmJ says:

        First, thanks a ton for all the support.

        Logs for the google protobuf installation –
        ———————————————————————-
        Libraries have been installed in:
        /usr/local/lib

        If you ever happen to want to link against installed libraries
        in a given directory, LIBDIR, you must either use libtool, and
        specify the full pathname of the library, or use the `-LLIBDIR’
        flag during linking and do at least one of the following:
        – add LIBDIR to the `LD_LIBRARY_PATH’ environment variable
        during execution
        – add LIBDIR to the `LD_RUN_PATH’ environment variable
        during linking
        – use the `-Wl,-rpath -Wl,LIBDIR’ linker flag
        – have your system administrator add LIBDIR to `/etc/ld.so.conf’

        See any operating system documentation about shared libraries for
        more information, such as the ld(1) and ld.so(8) manual pages.
        ———————————————————————-

      • amalgjose says:

        What is the output when u execute the following command ?
        pkg-config –libs protobuf

      • anuraginitmJ says:

        [edureka@localhost ~]$ ls /usr/local/lib/pkgconfig/
        protobuf-lite.pc protobuf.pc
        [edureka@localhost ~]$ echo $PKG_CONFIG_PATH
        /usr/local/lib/pkgconfig

        [edureka@localhost ~]$ pkg-config –modversion protobuf
        2.4.1
        [edureka@localhost ~]$ pkg-config –libs protobuf
        -pthread -L/usr/local/lib -lprotobuf -lz -lpthread
        [edureka@localhost ~]$

      • amalgjose says:

        Hi,
        Can you verify these steps also

        cd /etc/ld.so.conf.d/

        nano Protobuf-x86.conf

        /usr/local/lib # add this value as the content of Protobuf-x86.conf

        Save and exit
        Execute the following command

        /sbin/ldconfig

        Then try the Rhipe installation again.

    • anuraginitmJ says:

      Hey Amal,

      [edureka@localhost ~]$ cd /etc/ld.so.conf.d/
      [edureka@localhost ld.so.conf.d]$ ls
      kernel-2.6.32-431.29.2.el6.i686.conf kernel-2.6.32-431.el6.i686.conf mysql-i386.conf Protobuf-x86.conf qt-i386.conf R-i386.conf xulrunner-32.conf

      [edureka@localhost ld.so.conf.d]$ nano Protobuf-x86.conf
      [edureka@localhost ld.so.conf.d]$ /sbin/ldconfig
      ^C
      [edureka@localhost ld.so.conf.d]$ sudo /sbin/ldconfig
      [sudo] password for edureka:
      [edureka@localhost ld.so.conf.d]$ cd
      [edureka@localhost ~]$ cd Desktop/setups/

      [edureka@localhost setups]$ sudo R CMD INSTALL Rhipe_0.73.1.tar.gz

      * installing to library ‘/usr/lib/R/library’
      * installing *source* package ‘Rhipe’ …
      ** libs
      g++ -m32 -I/usr/include/R -DNDEBUG -I/usr/local/include -fpic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -I. -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -DUSEAUTOSHORT -DHAVE_UINTPTR_T `/usr/lib/R/bin/R CMD config –cppflags` `pkg-config –cflags protobuf` -c rexp.pb.cc -o rexp.pb.o
      Package protobuf was not found in the pkg-config search path.
      Perhaps you should add the directory containing `protobuf.pc’
      to the PKG_CONFIG_PATH environment variable
      No package ‘protobuf’ found
      ^Cmake: *** wait: No child processes. Stop.
      make: *** Waiting for unfinished jobs….
      make: *** wait: No child processes. Stop.
      ERROR: compilation failed for package ‘Rhipe’
      * removing ‘/usr/lib/R/library/Rhipe’
      [edureka@localhost setups]$

  27. anuraginitmJ says:

    [edureka@localhost ~]$ ls /usr/local/lib/pkgconfig/
    protobuf-lite.pc protobuf.pc
    [edureka@localhost ~]$ echo $PKG_CONFIG_PATH
    /usr/local/lib/pkgconfig

    [edureka@localhost ~]$ pkg-config –modversion protobuf
    2.4.1
    [edureka@localhost ~]$ pkg-config –libs protobuf
    -pthread -L/usr/local/lib -lprotobuf -lz -lpthread
    [edureka@localhost ~]$

  28. Lijo John says:

    Hi there. Can you please suggest what could be the reason for the following error ?
    [FYI, I have installed rjava on Rstudio and Hadoop 2.7.0 on Centos 7]

    library(rhdfs)
    Loading required package: rJava
    Error : .onLoad failed in loadNamespace() for ‘rJava’, details:
    call: dyn.load(file, DLLpath = DLLpath, …)
    error: unable to load shared object ‘/home/cmm25/R/x86_64-redhat-linux-gnu-library/3.2/rJava/libs/rJava.so’:
    libjvm.so: cannot open shared object file: No such file or directory
    Error: package ‘rJava’ could not be loaded

    • amalgjose says:

      Which version of rJava you used ?. This post is very old. Now the latest version of hadoop will not support these versions of Rhipe and other dependencies. You have to use the updated versions.

  29. Nitin Pant says:

    Sir, Please Help me about the procedure to install hadoop, hive and also there connection with R.\

    • amalgjose says:

      Lot of tutorials are available for hadoop and hive installation. You can follow apache documentation or cloudera documentation. Once you install hadoop and hive, start hive-server2. Then install R using the command yum install R. Then install an R package for hive called Rhive. This is very easy to install.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: