Open CV with Java and Python

Last week i was very excited about computer vision and the on going activity on this field of science. So i started looking at OpenCV,i was really amazed to see what it offer. Although i have just started playing with OpenCV,so i will show in this tutorial,how to get started with OpenCV in both Java and Python.

Now its just the beginner's tutorial,as i have no problem in admitting that currently i'm a novice or beginner in the computer vision field.Next time the example will be more realistic :)

Step 1: Setting up OpenCV

I have installed it on my Laptop having Ubuntu 14.04.Please follow the steps given here for installation.

After you have installed the OpenCV,set up  your IDE. I have used Eclipse-IDE.

• Create a Java Project

• Add External Lib and include the following jar: opencv-2410.jar
• Also add native library to  opencv jar in our build-path,which is located in the path:  /usr/local/share/OpenCV/java

You can download the source-code from here.Hope you enjoy it!

Step 2: Coding Time with Java

HelloWorld->Lets start our first HelloWorld example with OpenCV to make sure,its working fine.

import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;

public class HelloWorld {
public static void main(String[] args) {
System.loadLibrary( Core.NATIVE_LIBRARY_NAME );
System.out.println("Hello OpenCV..!!");
Mat mat = Mat.eye(3, 3, CvType.CV_8UC1);
System.out.println("mat = " + mat.dump());
}
}

Output:
Hello OpenCV..!!
mat = [1, 0, 0;
  0, 1, 0;

  0, 0, 1]

OpenCV_Drawing-> Now we have done our HelloWorld working,its time to get little fancier.Lets draw a circle filled with green over a picture.

public class OpenCV_Drawing {
public static void main(String[] args) {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat mat = Highgui.imread("/home/kuntal/Pictures/rock.jpeg");
PictureFrame.bufferedImageShow(mat, "Original");
Core.circle(mat, new Point(mat.width() * 0.5, mat.height() * 0.5), 40,
new Scalar(0, 255, 0), Core.FILLED);
Core.putText(mat, "Hello World!", new Point(30, 30), 100, 1,
new Scalar(0, 0, 0));
PictureFrame.bufferedImageShow(mat, "Drawing");
}

}

Output:

OpenCV_EdgeDetect-> We will use the same picture and detect the edge of this picture.

public class OpenCV_EdgeDetect {
public static void main(String[] args) {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat mat = Highgui.imread("/home/kuntal/Pictures/rock.jpeg");
PictureFrame.bufferedImageShow(mat, "Original");
int kernelSize = 3;
Mat kernel = new Mat(kernelSize, kernelSize, CvType.CV_32F) {
{
put(0, 0, 0);
put(0, 1, -1);
put(0, 2, 0);
put(1, 0, -1);
put(1, 1, 4);
put(1, 2, -1);
put(2, 0, 0);
put(2, 1, -1);
put(2, 2, 0);
}
};
Imgproc.filter2D(mat, mat, -1, kernel);
PictureFrame.bufferedImageShow(mat, "Laplacian");

}

}

Output:

OpenCV_FaceDetect-> Now Face or Object detection is a very widely used technique in modern day application.You will see how easy it is to do this OpenCv,just few lines of code. Obviously you can tune your algorithm,but default work good.

public class OpenCV_FaceDetect {
public static void main(String[] args) {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
CascadeClassifier faceDetector = new CascadeClassifier(
"/home/kuntal/knowledge/software/opencv-2.4.10/data/lbpcascades/lbpcascade_frontalface.xml");
Mat mat = Highgui.imread("/home/kuntal/Pictures/rock.jpeg");
MatOfRect faceDetections = new MatOfRect();
faceDetector.detectMultiScale(mat, faceDetections);

for (Rect rect : faceDetections.toArray()) {
Core.rectangle(mat, new Point(rect.x, rect.y), new Point(rect.x
+ rect.width, rect.y + rect.height), new Scalar(0, 255, 0));
}
PictureFrame.bufferedImageShow(mat, "faceDetection");
JOptionPane.showMessageDialog(null,
"Detected " + faceDetections.toArray().length + " faces");
}

}

Output:

Note: I have used Swing JFrame in PictureFrame class for showing picture.You can use any other.


Step 3: Coding time with Python

Testing OpenCv with python

• Open terminal, then launch python interpeter:

            python
then, import opencv:

import cv2
cv2.__version__

Output:
'2.4.10'



Reading Writing image(to gray Scale)

import cv2
grayImage = cv2.imread('/home/kuntal/Pictures/rock.jpeg', cv2.CV_LOAD_IMAGE_GRAYSCALE)
cv2.imwrite('/home/kuntal/Pictures/rock_modified.jpeg', grayImage)

Output: (Original and Gray Scale)

Tracking Faces with Haar Cascades Classifier

For this example you need numpy and matplotlib installed in your system.

import numpy as np
import cv2
from matplotlib import pyplot as plt

face_cascade = cv2.CascadeClassifier('/home/kuntal/knowledge/software/opencv-2.4.10/data/haarcascades/haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier('/home/kuntal/knowledge/software/opencv-2.4.10/data/haarcascades/haarcascade_eye.xml')

img = cv2.imread('/home/kuntal/Pictures/rock.jpeg')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray, 1.3, 5)


for (x,y,w,h) in faces:
    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w]
    eyes = eye_cascade.detectMultiScale(roi_gray)
    for (ex,ey,ew,eh) in eyes:
        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

cv2.imshow('img',img)
cv2.waitKey(0)

cv2.destroyAllWindows()

Output:

Hope you Enjoy It !!

Building a Recommender system with Apache Mahout

Recently i was playing with Apache Mahout for building recommend-er based system. I wanted to first test state of the art collaborative filtering algorithms before to build a customized solution (potentially on top of those algorithms).Here's a basic idea behind the recommendation system using Apache Mahout:

Collaborative Filtering

It is a technique for producing recommendations solely based on the user’s preferences for products (instead of including product features and/or user properties). Well, collaborative filtering can be user- or item-based.

• User-based recommendation- promotes products to the user that are bought by users who are similar to his/her.

User-based Recommendation: recommend products to a user based on what similar users have bought

• Item-based recommendation- proposes products that are similar to the ones the user already buys.

Item-based Recommendation: recommend products to a user that are similar to the ones he/she already bought



User-Item Preferences and Similarity

So what does similar mean in this context? In collaborative filtering similarity between users (for user-based recommendations) or items (for item-based recommendations) is computed based on the user-item preference only. We use the number of how often a user bought a product as a proxyfor the user’s preference.

Based on these user-item preferences we can use the Euclidean distance or the Pearson correlation to determine the similarity between users respectively items (products).

1. Based on the Euclidean distance, two users are similar if the distance between their preference vectors projected into a Cartesian coordinate system is small. 
2. In fact, the Pearson correlation (based on demeaned user-item preferences) coincides with the cosine of the angle between the preference vectors. That is, two users are similar if the angle between their preference vectors is small, or formulated in terms of correlation, two users are similar if they rate the same products high and other products low. 
3. The Tanimoto similarity between 2 users is computed as the number of products the 2 users have in common divided by the total number of products they bought (respectively clicked or viewed) overall.

Now lets implement the above ideas -Coding Time:

 Lets start playing by building a simple recommendation engine based on the movie lens data.

To see a recommender engine in action, you can for  download one of the movie Lens ratings data sets (I will show with one million ratings). Unzip the archive somewhere. The file that will interest you is u.data. Its format(separated by tab) is as follows:

userId | movieId | rating | timestamp

I have modified the file for mahout taste FileDataModel with the simple following format:

userId,movieId,rating

Sample data:

196,242,3
186,302,3
22,377,1
244,51,2
166,346,1
298,474,4
115,265,2
253,465,5

305,451,3
.....

Let's build a classic user based recommender algorithm using the Pearson correlation similarity with a nearest 10 users neighborhood with the code below:

public class UserRecommenderPlaying {

 public static void main(String[] args) throws TasteException, IOException {
// specifying the user id to which the recommendations have to be generated for
int userId=6;

//specifying the number of recommendations to be generated
int noOfRecommendations=5;

//Get the dataset using FileData Model
DataModel model = new FileDataModel(new File("/home/kuntal/knowledge/IDE/workspace/MahoutTest/data/rating.csv"));

//Use a pearson similarity algorithm
UserSimilarity similarity = new PearsonCorrelationSimilarity (model);

/*NearestNUserNeighborhood is preferred in situations where we need to have control on the exact no of neighbors*/
UserNeighborhood neighborhood = new NearestNUserNeighborhood (10, similarity, model);

/*Initalizing the recommender */
Recommender recommender = new GenericUserBasedRecommender ( model, neighborhood, similarity);

//calling the recommend method to generate recommendations
List<RecommendedItem> recommendations = recommender.recommend(userId, noOfRecommendations);

for (RecommendedItem recommendedItem : recommendations) {
System.out.println("Recommended Movie Id: "+recommendedItem.getItemID()+"  .Strength of Preference: "+recommendedItem.getValue());
}

}
}

Output:
Recommended Movie Id: 878  .Strength of Preference: 4.464102
Recommended Movie Id: 300  .Strength of Preference: 4.2047677
Recommended Movie Id: 322  .Strength of Preference: 4.0203676
Recommended Movie Id: 313  .Strength of Preference: 4.008741
Recommended Movie Id: 689  .Strength of Preference: 4.0

Let's build a classic item based recommender algorithm using the Pearson correlation similarity with the code below:

public class ItemRecommenderPlaying {

public static void main(String args[])throws TasteException, IOException   {
// specifying the user id to which the recommendations have to be generated for
int userId=308;

//specifying the number of recommendations to be generated
int noOfRecommendations=3;

// Data model created to accept the input file
FileDataModel dataModel = new FileDataModel(new File("/home/kuntal/knowledge/IDE/workspace/MahoutTest/data/rating.csv"));

/*Specifies the Similarity algorithm*/
ItemSimilarity itemSimilarity = new PearsonCorrelationSimilarity(dataModel);

/*Initalizing the recommender */
ItemBasedRecommender recommender =new GenericItemBasedRecommender(dataModel, itemSimilarity);

//calling the recommend method to generate recommendations
List<RecommendedItem> recommendations =recommender.recommend(userId, noOfRecommendations);


for (RecommendedItem recommendedItem : recommendations)
System.out.println("Recommended Movie Id: "+recommendedItem.getItemID()+"  .Strength of Preference: "+recommendedItem.getValue());


}
}


Output:
Recommended Movie Id: 245  .Strength of Preference: 5.0
Recommended Movie Id: 34  .Strength of Preference: 5.0
Recommended Movie Id: 35  .Strength of Preference: 5.0

Evaluation of the Algorithms:

In my opinion the most valuable part of the whole process is evaluating your algorithm/model. To feel immediately if your intuition of choosing a particular algorithm is a good one, or to see the good or bad impact of your own customized algorithm, you need a way to evaluate and compare them on the data.
You can easily do that with mahout RecommenderEvaluator interface. Two different implementations of that interface are given: AverageAbsoluteDifferenceRecommenderEvaluator and RMSRecommenderEvaluator. The first one is the average absolute difference between predicted and actual ratings for users and the second one is the classic RMSE (a.k.a. RMSD).

One way to check whether the recommender returns good results is by doing a hold-out test. We partition our dataset into two sets: a training-set consisting of 90% of the data and a test-set consisting of 10%. Then we train our recommender using the training set and look how well it predicts the unknown interactions in the testset.

public  class EvaluationUserExample{

public static void main(String[] args) throws IOException, TasteException, OptionException {

RecommenderBuilder builder = new RecommenderBuilder() {
public Recommender buildRecommender(DataModel model) throws TasteException{
UserSimilarity similarity = new PearsonCorrelationSimilarity (model);
//Splitting of data(.1) done using 90% in training-set & 10% test-set
UserNeighborhood neighborhood = new ThresholdUserNeighborhood (.1, similarity, model);
Recommender recommender = new GenericUserBasedRecommender ( model, neighborhood, similarity);
return new CachingRecommender(recommender);
}
};

RecommenderEvaluator evaluator = new AverageAbsoluteDifferenceRecommenderEvaluator();
DataModel model = new FileDataModel(new File("/home/kuntal/knowledge/IDE/workspace/MahoutTest/data/rating.csv"));
/*0.9 here represents the percentage of each user’s preferences to use to produce recommendations, the rest are compared to estimated preference values to evaluate.
 1 represent the percentage of users to use in evaluation (so here all users).*/
double score = evaluator.evaluate(builder,null, model,0.9,1);

System.out.println("Result: "+score);
}
}



Output:
Result: 0.8018675119933131

Note: if you run this test multiple times, you will get different results, because the splitting into trainingset and testset is done randomly.

All codes available at github.

Delay or Scheduled Message Delivery with RabbitMQ

In this tutorial,i will show you the logic of Delay or Scheduled message delivery with RabbitMQ. And  how to implement it through java. For Real world example/usecase of RabbitMQ,please go through this article.

Sometimes you don’t want messages in the queue to be read or delivered immediately. For example, while processing fax message if it fail due to network error,then no meaning of immediate retry,hence delay in this type of scenario will be useful.Fortunately, RabbitMQ 2.8+ introduced Dead Letter Exchanges (DLX), which allows us to simulate message scheduling.

In practice, if we wanted to enable retry on failure every 3 minutes, the flow would look like this:

1. Create Work Queue and bind it to Work Exchange.
2. Create Delay Queue and bind it to Delay Exchange.
3. Set x-dead-letter-exchange to Work Exchange.
4. Set x-message-ttl to 180000 ms (3 minutes) to Delay Queue .
5. Publish message to Work Queue.
6. Client reads message from Work Queue and attempts to process it.
7. If the message processing fails and client publishes to Delay Queue.
8. Messages stays in Delay Queue for 3 minutes.
9. When message ttl expires, it is re-queued to Work Queue via Work Exchange for another attempt at processing.
10. Repeat steps 4-7

Create the Work Queue:

private String WORK_QUEUE = "WorkQueue";

private String WORK_EXCHANGE = "WorkExchange"; 

//Create your connection factory for getting connection and channel

ConnectionFactory factory = new ConnectionFactory();

factory.setHost("localhost");

Connection connection = factory.newConnection();

Channel channel = connection.createChannel();

//declare Work Exchange and Work Queue,finally bind Work Queue to Work Exchange 

channel.exchangeDeclare(WORK_EXCHANGE, "direct", true);

channel.queueDeclare(WORK_QUEUE, true, false, false, null);

channel.queueBind(WORK_QUEUE, WORK_EXCHANGE,"RK", null);

Create the Delay Queue:

private String DELAY_QUEUE ="DelayQueue";

private String DELAY_EXCHANGE = "DelayExchange";

//Make Delay Queue's Dead Letter Exchange to Work Exchange,so that after message ttl expires the message are sent to Work Queue via Work Exchange.

Map<String, Object> args = new HashMap<String, Object>();

args.put("x-dead-letter-exchange", WORK_EXCHANGE);

args.put("x-message-ttl", 180000);

//declare Delay  Exchange and Delay Queue,finally bind Delay Queue to Delay Exchange 

channel.exchangeDeclare(DELAY_EXCHANGE, direct, true);

channel.queueDeclare(DELAY_QUEUE, true, false, false,args);

channel.queueBind(DELAY_QUEUE, DELAY_EXCHANGE, "RK", null);

Read from Work Queue:

QueueingConsumer consumer = new QueueingConsumer(channel);

 channel.basicConsume(WORK_QUEUE, true, consumer);

    while (true) {

      QueueingConsumer.Delivery delivery = consumer.nextDelivery();

      String message = new String(delivery.getBody());

         if (!processSomething(message)) {

        processLater(message);

    }

   }

Publish to Delay Queue on message processing failure:

String message = new String(delivery.getBody());

channel.basicPublish(DELAY_EXCHANGE, "", null, message.getBytes());

Notes:

It's worth mentioning that Delay Queue mechanishm gurantee that the message will be delivered atleast after the delay time,but not exactly after delay time is over.

Refer: http://www.rabbitmq.com/dlx.html and http://www.rabbitmq.com/ttl.html

Real World Example of RabbitMQ - Universal Message Queue

Last year i was developing a message queue based application  for our company to be used by different other products/application.Since it's purpose was to be very scalable and also to be used by various other application, so we named it Universal Message Queue (UMQ).

Why UMQ came to the picture?

Earlier database was used as queue for certain scenario. But Database is not a good choice for huge queuing functionality. A heavy loaded database with additional queuing functionality will affect the performance of an application. Database should not be used for queuing purpose because of following:

•  Database inherently does not support queue functionality
•  Using Database as queue increases load on the database, hence affect the overall application performance. 
• Implementing some new functionality of queue (such as priority, Delay, Tracking etc) through database are complex and not well proven.

To overcome the above issues, UMQ is good choice. UMQ is a generalized message queue system.
It is based on rest services. Beside eliminating the issues of database as a queuing system, UMQ
has the important additional features that are useful in different scenario for various application as
stated below.

Some of the major components  that were used for the UMQ are RabbitMQ and Redis.

Truly speaking we have done lots of poc & rnd with various opensource message Queue's before developing this internal product,but RabbitMQ was able to full-fill our above needs very well.

Some of the fetaures of RabbitMQ are very well, like Routing logic based on Exchange and Queue mechanism, Prioity(through its Plugin) ,Negative Acknowledgement(NACK) with/without  Requeing and Delay or Schedule Message delivery.

So in the next tutorial,i will give you the idea of Delay/Schedule message delivery along with how to implement the logic through java. 

Web Crawling and Data Mining with Apache Nutch

This tutorial series will how to do web crawling with Apache Nutch.
After you complete this tutorial,you will be able to successfully crawl data from most popular web site,and even build your own search engine with Apache Solr.

With fast-growing technologies such as social media, cloud computing, mobile applications, and big data, these are exciting, and challenging, times to be in computing.
One of the main challenges facing software architects is handling the massive volume of data consumed and produced by a huge, global user base. In addition, users expect online applications to always be available and responsive. To address the scalability and availability needs of modern web applications, we’ve seen a growing interest in specialized, non-relational data storage and processing technologies, collectively known as NoSQL (Not only SQL).

Apache Nutch:
Apache Nutch is a very robust and scalable tool for web crawling; it can be also integrated with the scripting language Python for web crawling. You can use it whenever your application contains huge data and you want to apply crawling on your data.And also you can integrate this with search engine like Apache Solr very easily.

Apache Solr:
Solr is a scalable, ready-to-deploy enterprise search engine that’s optimized to search large
volumes of text-centric data and return results sorted by relevance.
Scalable- Solr scales by distributing work (indexing and query processing) to multiple servers in a cluster.
Ready to deploy- Solr is open source, is easy to install and configure, and provides a preconfigured example to help you get started.
Optimized for search- Solr is fast and can execute complex queries in subsecond speed, often only tens of milliseconds.
Large volumes of documents- Solr is designed to deal with indexes containing many millions of documents.
Text-centric- Solr is optimized for searching natural-language text, like emails,web pages, resumes, PDF documents, and social messages such as tweets or blogs.


This tutorial series consists of two parts:

Part 1-  Build and Install Nutch 2.2 with MySQL

Part 2-  Crawling Naptol, Flipkart, Amazon, Jabong with Apache Nutch and Solr

Get ready to have some fun..!!

Web Crawling Naptol, Flipkart, Amazon, Jabong with Apache Nutch and Apache Solr

This tutorial will teach you to crawl data from popular online shopping portal like (Amazon, Flipkart, Naptol and Jabong) and index this crawl data into Apache Solr. Also you will learn how to crawl ajax enabled and secured (https) site with Apach Nutch.

Please go through the previous tutorial to set up Apache Nutch 2.2 with MySql.

Update the nutch-site.xml:
cd ${APACHE_NUTCH_HOME}/runtime/local/conf

Edit the nutch-site.xml to enable crawling through secure https:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>DemoWebCrawler</value>
</property>

<property>
<name>http.accept.language</name>
<value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.This allows selecting non-English language as default one to retrieve.It is a useful setting for search engines build for certain national group.
</description>
</property>

<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other information is available </description>
</property>

<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.
Currently the following stores are available: ….
</description>
</property>

<!-- Add this property ,so that nutch can crawl into secure https based websites -->
<property>
 <name>plugin.includes</name>
 <value>protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
</configuration>

Update regex-urlfilter.txt to overcome the block urls:

The regex-urlfilter blocks urls that have querystring parameters:

skip URLs containing certain characters as probable queries, etc.

-[?*!@=]

Modify that file so that urls with querystring parameters are crawled:

skip URLs containing certain characters as probable queries, etc.

-[*!@]

So comment (using #) those regex queries.More information on this check.

Now it lets crawl data/information from  Naptol(mobile phone), Amazon(books) , Flipkart(sport shoes) and Jabong(sport shoes).

Edit your seed.txt and paste the following:

http://www.naaptol.com/brands/nokia/mobile-phones.html

http://www.flipkart.com/mens-footwear/shoes/sports-shoes/pr?sid=osp,cil,nit,1cu&otracker=hp_nmenu_sub_men_0_Sports%20Shoes

http://www.amazon.in/s/ref=nb_sb_noss_2/278-5129563-3057638?url=search-alias%3Daps&field-keywords=machine%20learning

http://www.jabong.com/men/shoes/sports-shoes/?source=topnav_men

Start crawling by typing the following into the command line:

bin/nutch inject urls
bin/nutch generate -topN 20
bin/nutch fetch -all
bin/nutch parse -all
bin/nutch updatedb

Repeat the last four commands (generate, fetch, parse and updatedb) again.

Set up and index with Solr 
Use latest version of Solr 4 (im using 4.9),other version 4+ will work fine too. Untar it to to $HOME/apache­solr­4.X.X­XX. This folder will be now referred to as ${APACHE_SOLR_HOME}.

Download from this link and use it to replace ${APACHE_SOLR_HOME}/example/solr/collection1/conf/schema.xml .

From the terminal start solr:

cd ${APACHE_SOLR_HOME}/example

java -jar start.jar

You can check this is running by opening http://localhost:8983/solr in your web browser as hown below. Select collection1 from the core selector.

Leave that terminal running and from a different terminal type the following:

cd ${APACHE_NUTCH_HOME}/runtime/local/

bin/nutch solrindex http://localhost:8983/solr/ -reindex

You can now run queries using Solr versus your crawled content. Open http://localhost:8983/solr/#/collection1/query and assuming you have already crawled the above websites,type in the input box titled “q” or "fq" you can do a search by inputting

content: jabong OR content: nokia  
(similarly try out for others like shoes,books etc)

and you should see something like this:

Congratulation :) for making your first small search engine ready from the crawl data of popular online shopping websites.

Build and Install Nutch 2.2 with MySQL

This tutorial will teach you to build set up Apache Nutch (latest version -2.2) with MySql. Let's get started !

Install MySQL Server and MySQL Client using the Ubuntu software center or  sudo apt-get install mysql-server mysql-client  at the command line.

As MySQL defaults to latin we need to edit  sudo vi /etc/mysql/my.cnf  and under [mysqld] add

innodb_file_format=barracuda
innodb_file_per_table=true
innodb_large_prefix=true
character­set­server=utf8mb4
collation­server=utf8mb4_unicode_ci
max_allowed_packet=500M

The innodb options are to help deal with the small primary key size restriction of MySQL. The character and collation settings are to handle Unicode correctly.The max_allowed_packet settings is optional and only necessary for very large sizes. Restart your machine for the changes to take effect.

Check to make sure MySQL is running by typing  sudo netstat -tap | grep mysql  and you should see something like:
tcp 0 0 localhost:mysql *:* LISTEN

We need to set up the nutch database manually as the current Nutch/Gora/MySQL generated db schema defaults to latin.
Log into mysql at the command line using your previously set up MySQL id and password type

mysql -u xxxxx -p
then in the MySQL editor type the following:

CREATE DATABASE nutch DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci; 

use nutch;

and enter and then copy and paste the following altogether:

CREATE TABLE `webpage` (
`id` varchar(767) NOT NULL,
`headers` blob,
`text` longtext DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`markers` blob,
`parseStatus` blob,
`modifiedTime` bigint(20) DEFAULT NULL,
`prevModifiedTime` bigint(20) DEFAULT NULL,
`score` float DEFAULT NULL,
`typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`batchId` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`baseUrl` varchar(767) DEFAULT NULL,
`content` longblob,
`title` varchar(2048) DEFAULT NULL,
`reprUrl` varchar(767) DEFAULT NULL,
`fetchInterval` int(11) DEFAULT NULL,
`prevFetchTime` bigint(20) DEFAULT NULL,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime` bigint(20) DEFAULT NULL,
`retriesSinceFetch` int(11) DEFAULT NULL,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
ROW_FORMAT=COMPRESSED
DEFAULT CHARSET=utf8mb4;

Then type enter. You are done setting up the MySQL database for Nutch.

Set up Nutch 2.2 by downloading the apache­nutch­2.2­src.tar.gz version from
http://www.apache.org/dyn/closer.cgi/nutch/.
Untar the contents of the file you just downloaded to a folder we will refer to
going forward as ${APACHE_NUTCH_HOME}.

From inside the nutch folder ensure the MySQL dependency for Nutch is available by editing the following in ${APACHE_NUTCH_HOME}/ivy/ivy.xml
change

<dependency org=”org.apache.gora” name=”gora­core” rev=”0.3′′ conf=”*­>default”/>
to
<dependency org=”org.apache.gora” name=”gora­core” rev=”0.2.1′′ conf=”*­>default”/>

and uncomment the gora­sql

<dependency org=”org.apache.gora” name=”gora­sql” rev=”0.1.1­incubating” conf=”*­>default” />
and uncomment the mysql connector

<!– Uncomment this to use MySQL as database with SQL as Gora store. –>
<dependency org=”mysql” name=”mysql­connector­java” rev=”5.1.18′′ conf=”*­>default”/>

Also update the following jetty mortbay dependency with latest version (7.0.0.pre5),else you might get build failure:

<dependency org="org.mortbay.jetty" name="jetty" rev="7.0.0.pre5" conf="test->default" />
 <dependency org="org.mortbay.jetty" name="jetty-util" rev="7.0.0.pre5" conf="test->default" />

<dependency org="org.mortbay.jetty" name="jetty-client" rev="7.0.0.pre5" />

Edit the ${APACHE_NUTCH_HOME}/conf/gora.properties file either deleting or commenting out the Default SqlStore
Properties using #. Then add the MySQL properties below replacing xxxxx with the user and password you set up when installing MySQL earlier.

###############################
# MySQL properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=xxxxx
gora.sqlstore.jdbc.password=xxxxx

Edit the ${APACHE_NUTCH_HOME}/conf/gora­sql­mapping.xml file changing the length of the primarykey from 512 to 767 in both places.
<primarykey column=”id” length=”767′′/>

Configure ${APACHE_NUTCH_HOME}/conf/nutch­site.xml to put in a name in the value field under http.agent.name. It can be anything but cannot be left blank. You must specify Sqlstore.

<property>
<name>http.agent.name</name>
<value>DemoWebCrawler</value>
</property>
<property>
<name>http.accept.language</name>
<value>ja­jp, en­us,en­gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept­Language” request header field.This allows selecting non­English language as default one to retrieve.It is a useful setting for search engines build for certain national group.
</description>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf­8</value>
<description>The character encoding to fall back to when no other information
is available</description>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.Currently the following stores are available: ....
</description>
</property>

Install ant using the Ubuntu software center or  sudo apt-get install ant  at the command line.
From the command line  cd  to your nutch folder and  after you have  cd  to ${APACHE_NUTCH_HOME} simply type  ant runtime
This may take a few minutes to compile.

Start your first crawl by typing the lines below at the terminal (replace ‘http://nutch.apache.org/’ with whatever site you want to crawl):

Inject a URL into the DB

cd ${APACHE_NUTCH_HOME}/runtime/local

mkdir -p urls

echo 'http://nutch.apache.org/' > urls/seed.txt

Start crawling (you will want to create your own script later but manually just to see what is happening type the following into the command line:

bin/nutch inject urls
bin/nutch generate -topN 20
bin/nutch fetch -all
bin/nutch parse -all
bin/nutch updatedb

Repeat the last four commands (generate, fetch, parse and updatedb) again.

For the generate command, topN is the max number of links you want to actually parse each time. The first time there is only one URL (the one we injected from seed.txt) but after that there are many more. Note, however, Nutch keeps track of all links it encounters in the webpage table. It just limits the amount it actually parses to TopN so don’t be surprised by seeing many more rows in the webpage table than you expect by limiting with TopN.

Check your crawl results by looking at the webpage table in the nutch database.
mysql -u xxxxx -p
use nutch;
SELECT * FROM nutch.webpage LIMIT 10;

You should see the 10 rows/results of your crawl ( i have shown in mysql workbench):

Now that you have successfully set up Apache nutch with MySql and crawl few web site.Its time to do something more interesting stuff like,using this crawl data for indexing and searching.Follow the next tutorial.