Search Engine - Lucene

Why eBay will keep losing its business - product search sucks big time

2009-05-04T15:01:00.000-07:00

I wanted to buy a Sony battery charger for rechargable AA batteries. I went to a few of online purchase websites and did search for keyword "Sony battery charger". At buy.com the results were shown for the kind of chargers I was looking for and it did show one charger, which I really liked (Sony Power Battery Charger - with 4 AA Ni-MH Rechargeable Batteries).

I did the same search on eBay and it found 14,500+ products but everything on first page was of no relevance to me. In results, the first product is sony camera batter charger (New Battery & Charger For Sony Camera NP-BG1 NPBG1). What the hell!! God knows where did that come from? eBay is into online shopping business and they can't afford to have such rubbish search results.

I am not here to promote eBay business but I know eBay has got so many deals for every product and some people who know how to search it on eBay really enjoy shopping there. I guess eBay is using Lucene or similar search engine for product search. I am sure that eBay is losing its business due to the poor search relevance. I am a search engine programmer and had tough time doing query for what I am looking for. How does eBay expect non-geeks to search stuff on their site?

I find Craiglist search returning more relevant results then eBay does. No doubt that people would stop going to a website where they can't find things they are looking for.

May be getting a relevant result from millions of product data can be difficult, but thats why search engines are invented isn't it? The economy is tough and still eBay is thriving. First quarter results of comapny were good and now may be if John Donahoe want's to be a successful CEO he would try to address this simple problem (of searching products) before getting into more complex issues.

By the way when I used Google site search for ebay it didn't give satisfactory results either. try "site:ebay.com Sony battery charger" on google.

I tried a few other products search on eBay and compared with other online shopping portals and the results were no different. Where as Google's shopping results show absolutely relevant results as if they were reading my mind.

Did you know you can also get cashback discounts on your eBay purchases if you go through Microsoft's live.com Read More.

Search Engine Best Practices: Compass as a Lucene Abstraction

2009-02-23T16:23:00.001-08:00

If you are considering Lucene as a Search Engine library for your application then you would certainly want to have a look at Compass. Compass is a open source project, which should be considered before starting from scratch development using Lucene.

I have seen many people not feeling comfortable dealing with Lucene APIs, Compass provides a simple API for using Lucene. Here are the features I gathered from its latest documenation on [http://www.compass-project.org/]

It provides operations for save, and delete & query without knowing details of index handling.

Compass simplifies common usage patterns of Lucene such as google style search, index updates. It also has support for caching and index sharding (sub indexes).
In fact it has built in optimizations for concurrent commits and merges, which can be complex implementation if you are trying to do it on your own.

It provides a transactional API on top of the Search Engine supporting different transaction isolation levels.
It also provides a local transaction manager as well as integration with external transaction managers such as JTA (Sync and XA), Spring, and ORM ones.

It integrates seamlessly with most popular ORM frameworks allowing automatic mirroring, to the index, of the changes in data performed via the ORM tool. Compass has generic support for JPA as well as embedded support for Hibernate, OpenJPA, TopLink Essentials, and EclipseLink allow to add Compass using three simple steps.

It also integrates with Spring. Compass can be easily configured using Spring, integrates with Spring transaction management, has support for Spring MVC, and has Spring aspects built in for reflecting operations to the search engine.

It also simplifies the creation of distributed Lucene index by allowing to store the Lucene index in a database, as well as storing the index simply with Data Grid products such as GigaSpaces, Coherence and Terracotta.

For more details of Compass, have a look at the Compass web page here [http://www.compass-project.org/], If you are a Compass user/developer and facing some problem then you can post your problems in Compass forums at [http://forum.compass-project.org/index.jspa]. The community is really enthusiastic and you would get responses very quickly.

If you have some idea about best practices in search engine arena please share with us, and post a comment. We would be glad to discuss on it.

Lucene Index Maintenance (Insert, Update and Delete)

2008-10-10T23:42:00.001-07:00

Lucene is a powerfull search engine with great indexing API which can be used to build search feature for your web application.
In this post I am trying to address the maintenance aspects of a Lucene index.
Once the initial index is created here are the steps which you may need to perform

1. Inserting new documents
2. Updating existing documents
3. Deleting existing documents

Inserting new documents:
Lucene doesnt check for duplicate documents, so you need to build your own logic to identify whether a document is new document or a existing one.
Once identied, inserting is not different than creating the index we just need to take care of recreate flag,
which needs to be false so that the index directory is not overwritten (recreated). Adding the document can be done using IndexWriter.addDocument() call for each document you want to add.

Updating existing documents:
As I have already mentioned that Lucene is not going to check for duplicates, so in case of updating
if its an existing document then it can be done in two steps,
first delete the document and then add the new document to index.

Deleting existing documents:
Deleting can be easily done by IndexReader.deleteDocuments(Terms) and IndexWriter.deleteDocuments(Terms)

Problem deleting documents:

What I have observed is that if the document field is UN_TOKENIZED then these methods work well. But if the Document field is TOKENIZED then delete fails.

One good way to do this is to keep same field with a different name and with UN_TOKENIZED type index field, and while deleting use the same field Term to delete the document.

Deleting the index documents by number is simple and easy. It works well if you know the document number.

You need to reopen all the searchers which were already open.

Post a Message

Lucene: Asynchronous Index Writer for faster writing

2008-08-28T13:40:00.000-07:00

I was trying to do some Index writing speed improvement and thought of creating a asynchronous Lucene index writer. This writer provides a addDocument() method which can be called asynchronously by multiple threads.

Here are the few scenario where you can utilize this implementation

- Reading the data is slower then writing to the index. (Typical scenario where you read over network, or from database.)

- Reading can be divided in multiple logical parts which can be processed in separate threads.

- You are looking for asynchronous behavior to decouple reading and writing processes.

This implementation is a wrapper which utilizes core methods of IndexWriter class, and does not do any change to it except making it asynchronous. It utilizes Java's java.util.concurrent.BlockingQueue for storing the documents. It can be supplied with any implementation of this class using its constructor.

Below is the Java source of this implementation class

This class provides multiple constructors to have better control.

Few terms which are used here are as follows

Sleep Milliseconds On Empty: This is the sleep duration when writer finds nothing in queue and wants to wait for some data to come in queue. ()

Queue Size: This is the size of the queue which can be configured as a constructor parameter input.

package swiki.lucene.asynchronous;

import java.io.IOException;

import java.util.concurrent.ArrayBlockingQueue;

import java.util.concurrent.BlockingQueue;

import org.apache.lucene.document.Document;

import org.apache.lucene.index.CorruptIndexException;

import org.apache.lucene.index.IndexWriter;

/**

* @author swiki swiki

public class AsynchronousIndexWriter implements Runnable {

* A blocking queue of document to facilitate asynchronous writing.

private BlockingQueue documents;

* Instance of core index writer which does the actual writing task.

private IndexWriter writer;

* Thread which makes writing asynchronous

private Thread writerThread;

* We need to set this to false if the document addition is completed. This

* will not immediately stop the writing as there could be some documents in

* the queue. It completes once all documents are written and the queue is

* empty.

private boolean keepRunning = true;

* This flag is set to false once writer is done with the queue data

* writing.

private boolean isRunning = true;

* Duration in miliseconds for which the writer should sleep when it finds

* the queue empty and job is still not completed

private long sleepMilisecondOnEmpty = 100;

/**

* This method should be used to add documents to index queue. If the queue

* is full it will wait for the queue to be available.

* @param doc

* @throws InterruptedException

public void addDocument(Document doc) throws InterruptedException {

documents.put(doc);

}

public void startWriting() {

writerThread = new Thread(this, "AsynchronousIndexWriter");

writerThread.start();

}

/**

* Constructor with indexwriter as input. It Uses ArrayBlockingQueue with

* size 100 and sleepMilisecondOnEmpty is 100ms

* @param w

public AsynchronousIndexWriter(IndexWriter w) {

this(w, 100, 100);

}

/**

* Constructor with indexwriter and queue size as input. It Uses

* ArrayBlockingQueue with size queueSize and sleepMilisecondOnEmpty is

* 100ms

* @param w

* @param queueSize

public AsynchronousIndexWriter(IndexWriter w, int queueSize) {

this(w, queueSize, 100);

}

/**

* Constructor with indexwriter, queueSize as input. It Uses

* ArrayBlockingQueue with size queueSize

* @param w

* @param queueSize

* @param sleepMilisecondOnEmpty

public AsynchronousIndexWriter(IndexWriter w, int queueSize,

long sleepMilisecondOnEmpty) {

this(w, new ArrayBlockingQueue(queueSize), sleepMilisecondOnEmpty);

}

/**

* A implementation of BlockingQueue can be used

* @param w

* @param queueSize

* @param sleepMilisecondOnEmpty

public AsynchronousIndexWriter(IndexWriter w, BlockingQueue queue,

long sleepMilisecondOnEmpty) {

writer = w;

documents = queue;

this.sleepMilisecondOnEmpty = sleepMilisecondOnEmpty;

startWriting();

}

* (non-Javadoc)

* @see java.lang.Runnable#run()

public void run() {

while (keepRunning || !documents.isEmpty()) {

Document d = (Document) documents.poll();

try {

if (d != null) {

writer.addDocument(d);

} else {

/*

* Nothing in queue so lets wait

*/

Thread.sleep(sleepMilisecondOnEmpty);

}

} catch (ClassCastException e) {

e.printStackTrace();

throw new RuntimeException(e);

} catch (InterruptedException e) {

e.printStackTrace();

throw new RuntimeException(e);

} catch (CorruptIndexException e) {

e.printStackTrace();

throw new RuntimeException(e);

} catch (IOException e) {

e.printStackTrace();

throw new RuntimeException(e);

}

}

isRunning = false;

}

/**

* Stop the thread gracefully, wait until its done writing.

private void stopWriting() {

this.keepRunning = false;

try {

while (isRunning) {

//using the same sleep duration as writer uses

Thread.sleep(sleepMilisecondOnEmpty);

}

} catch (InterruptedException e) {

e.printStackTrace();

}

}

public void optimize() throws CorruptIndexException, IOException {

writer.optimize();

}

public void close() throws CorruptIndexException, IOException {

stopWriting();

writer.close();

}

Below is a sample class which demonstrates how we can use this class. Here are few things to note, asynchronous thread is started as soon as you instantiate using new AsynchronousIndexWriter(...)

package swiki.lucene.asynchronous;

import org.apache.lucene.analysis.standard.StandardAnalyzer;

import org.apache.lucene.document.Document;

import org.apache.lucene.document.Field;

import org.apache.lucene.index.IndexWriter;

import org.apache.lucene.store.Directory;

import org.apache.lucene.store.FSDirectory;

/**

* @author swiki swiki

public class TestAsyncWriter {

public static void main(String[] args) {

try {

Directory fsdir = FSDirectory.getDirectory("index");

IndexWriter w = new IndexWriter(fsdir, new StandardAnalyzer(), true);

AsynchronousIndexWriter writer = new AsynchronousIndexWriter(w);

* This call can be replaced by the logic of reading

* data using multiple threads

addDocumentsInMultipleThreads(writer);

writer.optimize();

writer.close();

} catch (Exception e) {

e.printStackTrace();

}

private static void addDocumentsInMultipleThreads(

AsynchronousIndexWriter writer) throws InterruptedException {

//add here the code for adding document from multiple threads.

Document doc = new Document();

doc.add(new Field("content","My Content", Field.Store.YES, Field.Index.UN_TOKENIZED));

writer.addDocument(new Document());

}

If you find this useful or have some suggestions for improvements please leave a comment and I will try to respond.

Can I use Lucene to index text in Chinese and other multi-byte character set languages?

2008-08-26T13:45:00.000-07:00

Yes, Lucene is not limited to English, or any other language. To index text, you need to use an appropriate Analyzer for the language of the text you are indexing. Lucene's default Analyzers work well for English. There are other Analyzers in Lucene Sandbox, including those for Chinese, Japanese and many other languages.

How can I index Powerpoint documents?

2008-08-26T13:39:00.000-07:00

To create a Lucene based index of Powerpoint documents you need to first parse them to extract text that you want to index. You can use the Jakarta Apache POI, to parse the Powerpoint document and extract the relevant text from it.

You should also store the location of the Powerpoint document on file system in one of index fields this way when you find the document it can be directly shown as a link to the user. Storing all text in a content field is a good idea as you may want to show text representation of same document. Also this will help people to different wild card searches.

Indexing Speed Factors

2008-08-14T14:49:00.001-07:00

If you are using Lucene in a non-trivial application, you will want to ensure optimal indexing performance. The bottleneck of a typical text-indexing application is the process of writing index files onto a disk. Therefore, we need to instruct Lucene to be smart about adding and merging segments while indexing documents.

When new documents are added to a Lucene index, they are initially stored in memory instead of being immediately written to the disk. This is done for performance reasons. The simplest way to improve Lucene's indexing performance is to adjust the value of IndexWriter's mergeFactor instance variable. This value tells Lucene how many documents to store in memory before writing them to the disk, as well as how often to merge multiple segments together. With the default value of 10, Lucene will store 10 documents in memory before writing them to a single segment on the disk. The mergeFactor value of 10 also means that once the number of segments on the disk has reached the power of 10, Lucene will merge these segments into a single segment. (There is a small exception to this rule, which I shall explain shortly.)

For instance, if we set mergeFactor to 10, a new segment will be created on the disk for every 10 documents added to the index. When the 10th segment of size 10 is added, all 10 will be merged into a single segment of size 100. When 10 such segments of size 100 have been added, they will be merged into a single segment containing 1000 documents, and so on. Therefore, at any time, there will be no more than 9 segments in each power of 10 index size.

The exception noted earlier has to do with another IndexWriter instance variable: maxMergeDocs. While merging segments, Lucene will ensure that no segment with more than maxMergeDocs is created. For instance, if we set maxMergeDocs to 1000, when we add the 10,000th document, instead of merging multiple segments into a single segment of size 10,000, Lucene will create a 10th segment of size 1000, and keep adding segments of size 1000 for every 1000 documents added.

The default value of maxMergeDocs is Integer.MAX_VALUE. In my experience, one rarely needs to change this value.

Now that I have explained how mergeFactor and maxMergeDocs work, you can see that using a higher value for mergeFactor will cause Lucene to use more RAM, but will let Lucene write data to disk less frequently, which will speed up the indexing process. A smaller mergeFactor will use less memory and will cause the index to be updated more frequently, which will make it more up-to-date, but will also slow down the indexing process. Similarly, a larger maxMergeDocs is better suited for batch indexing, and a smaller maxMergeDocs is better for more interactive indexing.

Lucene Search Engine: Optimization Techniques

2008-08-14T14:48:00.000-07:00

Most obvious optimization - always call writer.optimize() after index update. This will merge index into one segment, and only one Searcher will be created during search.
Use combined search field for all text fields instead (or on the top) of indexing them separately and searching with complex query like field1:query OR field2:query ... OR fieldN:query
Reducing number of field make indexing and search much faster. Use combined field instead or on the top of separate fields if needed
Do not use compound file format during indexing. writer.setUseCompoundFile(false); If you want to keep index in single file, turn it on when indexing is complete.
Do not create Searcher and Analyzer for each search. When you create Searcher, it reads small dictionary into memory to speed up term lookups. It takes time. Keep Searcher opened and reuse it for every search. When index changes make sure to recreate you Searcher.
Increase heap size for the java virtual machine
Use quotes “” only to perform exact phrase searches. field:term is much better than field:”term”

Merge Factor

Lucene’s IndexWriter has parameter called Merge Factor. This is a number of segments writer keeps in memory (RAMDirectory) before flushing them to disk. Lucene documentation claims that if you want to speed up indexing, you should increase Merge Factor. Though it sounds logical, there is not much to gain here. And drawback with using big Merge Factor is that there could be too many open files. Keeping it to default 10 or 20 turns out to be the most reasonable choice.

Lucene: A Powerfull & Free Search Engine by Apache

2008-05-02T14:28:00.000-07:00

Evaluating search engines

Other widely used open source search engines include Swish-E, Glimpse, libibex, freeWAIS, and iSearch. Like any software package, each is optimized for use in particular situations; it is often difficult to deploy these tools outside of their intended domains. Consider the following features when evaluating a search engine:

Incremental versus batch indexing: Some search engines only support batch indexing; once they create an index for a set of documents, adding new documents becomes difficult without reindexing all the documents. Incremental indexing allows easy adding of documents to an existing index. For some applications, like those that handle live data feeds, incremental indexing is critical. Lucene supports both types of indexing.

Data sources: Many search engines can only index files or Webpages. This handicaps applications where indexed data comes from a database, or where multiple virtual documents exist in a single file, such as a ZIP archive. Lucene allows developers to deliver the document to the indexer through a String or an InputStream, permitting the data source to be abstracted from the data. However, with this approach, the developer must supply the appropriate readers for the data.

Indexing control: Some search engines can automatically crawl through a directory tree or a Website to find documents to index. While this is convenient if your data is already stored in this manner, crawler-based indexers often provide limited flexibility for applications that require fine-grained control over the indexed documents. Since Lucene operates primarily in incremental mode, it lets the application find and retrieve documents.

File formats: Some search engines can only index text or HTML documents; others support a filter mechanism, which offers a simple alternative to indexing word processing documents, SGML documents, and other file formats. Lucene supports such a mechanism.

Content tagging: Some search engines treat a document as a single stream of tokens; others allow the specification of multiple data fields within a document, such as "subject," "abstract," "author," and "body." This permits semantically richer queries like "author contains Hamilton AND body contains Constitution." Lucene supports content tagging by treating documents as collections of fields, and supports queries that specify which field(s) to search.

Stop-word processing: Common words, such as "a," "and," and "the," add little value to a search index. But since these words are so common, cataloging them will contribute considerably to the indexing time and index size. Most search engines will not index certain words, called stop words. Some use a list of stop words, while others select stop words statistically. Lucene handles stop words with the more general Analyzer mechanism, to be described later, and provides the StopAnalyzer class, which eliminates stop words from the input stream.

Stemming: Often, a user desires a query for one word to match other similar words. For example, a query for "jump" should probably also match the words "jumped," "jumper," or "jumps." Reducing a word to its root form is called stemming. Lucene does not yet implement stemming, but you could easily add a stemmer through a more sophisticated Analyzer class.

Query features: Search engines support a variety of query features. Some support full Boolean queries; others support only and queries. Some return a "relevance" score with each hit. Some can handle adjacency or proximity queries -- "search followed by engine" or "Knicks near Celtics" -- others can only search on single keywords. Some can search multiple indexes at once and merge the results to give a meaningful relevance score. Lucene supports a wide range of query features, including all of those listed above. However, Lucene does not support the valuable Soundex, or "sounds like," query.

Concurrency: Can multiple users search an index at the same time? Can a user search an index while another updates it? Lucene allows users to search an index transactionally, even if another user is simultaneously updating the index.

Non-English support: Many search engines implicitly assume that English is the target language; this is evident in areas such as stop-word lists, stemming algorithms, and the use of proximity to match phrase queries. As Lucene preprocesses the input stream through the Analyzer class provided by the developer, it is possible to perform language-specific filtering.