Skip to main content

Integrating Apache Nutch and Hbase using Gora

There are six steps for using hbase as gora-nutch backend.

First, you have to download Nutch 2 from a mirror site of Apache. Download and extraxt it to where you want to install.

# tar -zxvf apache-nutch-2.X-src.tar.gz

Second, you have to downlad and install Hbase from a mirror site of Apache. Now Gora 0.2 supports Hbase 0.90.X branch.

Third, Gora backend must be specified in nutch-site.xml. Before compiling nutch, all usual configuration parameters should be set in the nutch-site.xml.

<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
<description>Default class for storing data</description>
</property>

Fourth, gora-hbase dependency must be available in the ivy/ivy.xml.

<!-- Uncomment this to use HBase as Gora backend. -->
<dependency org="org.apache.gora" name="gora-hbase" rev="0.2" conf="*->default" />

Fifth, in the gora.properties file default datastore must be specified as HBaseStore.

gora.datastore.default=org.apache.gora.hbase.store.HBaseStore

Sixth, compile nutch. (Ant has to be available)

# ant runtime

Now, you should then be able to use nutch.


Etiketler:

Comments

  1. AnonymousAugust 28, 2013 at 10:29 AM

    hi, I have followed your steps. But when I run the nutch crawl command I am getting below exception..

    Exception in thread "main" java.lang.RuntimeException: job failed: name=generate: null, jobid=job_local1224515128_0002
    at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
    at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199)
    at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:152)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)

    I use this command. PLease let me know your suggestions to fix this issue

    Thanks,
    RP

    ReplyDelete
  2. Kadir SertSeptember 3, 2013 at 9:45 AM

    Hi, can you give full crawl command you run?
    Second, have you run nutch as distributed or standalone mode?
    When you run nutch in $NUTCH_HOME/runtime/local folder that's standalone,
    When you run nutch from $NUTCH_HOME/runtime/deploy that's distributed.
    You may have forgotten specifying urls directory that should be injected for the first run.

    ReplyDelete
  3. AnonymousSeptember 28, 2013 at 2:18 AM

    Hi! I have the same error
    Run nutch in standalone folder
    Where is my script
    bin/nutch crawl urls -depth 3 -topN 5



    ReplyDelete
  4. AnonymousJanuary 13, 2016 at 4:57 AM

    Hello! I have the same error, but it happens with fetch. Here's my stacktrace:

    Exception in thread "main" java.lang.RuntimeException: job failed: name=fetch, jobid=job_local_0007
    at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
    at org.apache.nutch.fetcher.FetcherJob.run(FetcherJob.java:194)
    at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:161)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)
    I ran the command bin/nutch crawl urls -depth 20 -topN 10000

    ReplyDelete

Post a Comment

Popular posts from this blog

Creating Multiple VLANs over Bonding Interfaces with Proper Routing on a Centos Linux Host

In this post, I am going to explain configuring multiple VLANs on a bond interface. First and foremost, I would like to describe the environment and give details of the infrastructure. The server has 4 Ethernet links to a layer 3 switch with names: enp3s0f0, enp3s0f1, enp4s0f0, enp4s0f1 There are two bond interfaces both configured as active-backup bond0, bond1 enp4s0f0 and enp4s0f1 interfaces are bonded as bond0. Bond0 is for making ssh connections and management only so corresponding switch ports are not configured in trunk mode. enp3s0f0 and enp3s0f1 interfaces are bonded as bond1. Bond1 is for data and corresponding switch ports are configured in trunk mode. Bond0 is the default gateway for the server and has IP address 10.1.10.11 Bond1 has three subinterfaces with VLAN 4, 36, 41. IP addresses are 10.1.3.11, 10.1.35.11, 10.1.40.11 respectively. Proper communication with other servers on the network we should use routing tables. There are three
Post a Comment
Read more

Listing Zimbra Accounts with Some Details

Recently I have to export the user list for a particular domain. Luckily Zimbra has Admin GUI with a search feature. When you search accounts, you can download search results as a comma-separated csv file. So I did a search and download the result file, but the result did not have all the columns I need and also there is no option for customizing columns for search results. So I had to write a bash script to get the desired list. Here is the bash script ( It can be customized by adding or removing field names. Run it under zimbra user like ./zimbra_account_list.sh <domain_name_here> ):
Post a Comment
Read more