In this context, the creation of a search engine is a three step process

Indexing. This is the fundamental step to all searching. Roughly described as follows: a document is read (parsed) and the text is then divided into tokens (this means really, words -- collections of letters separated by blank spaces). If this is the first document and the first token, the token is added to the "index". Further, to the token the number of the document (in this case 1) is added to the token. If the index already has tokens, then if the token is in the index, the document number of the current document is added to the token, otherwise the token with document number is added to the index. Consider an index as a collection of tokens (words for our purposes) and a vector of document numbers.

The process of tokenizing (extracting the tokens) is also often accompanied by a filtering and transformation process. In general it is useful to change all characters to lower case. Further it might be advantageous to label the words with their part of speech in the context in which found (POS labeling). So called "stop words" are dropped and not indexed. By stop words we mean words such as "I", "it"(but IT, information technology) , "the", which occur in every document and do not contribute in any way to the being able to fish this document out of a group. Sometimes it is advantages to reduce a word to its stem (grammatically). For example, an adjective big, bigger, biggest would all be reduced to "big", or a verb to go, went, gone, going would all be reduced to "go".

So the tokens and the associated vectors of document numbers are stored in the index. But that is not all. The documents represented as Lucene documents are also stored. This is not the Word, PDF or what have you, document, but an invocation of a Lucene class called document. When indexing, the user (programmer actually) decides what information is to be stored in the Lucene document, in our case for example, the name of the file in which the document is stored, the date it was created, last changed, as well as a sort of summary.

Searching. Searching is then the process of taking a key word or an expression (e.g., "a" AND "b" -- with short cut +a +b) and finding those documents in which it is present. There are many variations and the tools we have at hand allow simple key word searches (where there is just one key word), so called query searches (this allows a more complex syntax involving more than one key word. The query search also allows wild card searches. Query searches are the only type of search realized in this system.

Range searches, for example with dates and such, have not been realized, but could be. Also phrase searches and fuzzy searches (where tokens which in some way are "near" the desired word) are not realized -- Query searches support fuzzy searches and phrase searches to an extent. See the chapter on searching.

A search results in a collection of documents or in the case we have, references to documents.

Presentation. The step of presenting the information to the user involves deciding which information contained in the documents won is to be presented and how. Recall, that the information retrieved from a search is not the physical document, but the parsed information which is stored in the index. The user is offered an HTML file containing salient information and a reference to the document. This HTML file can be configured (appearance only) by the user. There are rough hints as to how to do this.

This application consists of two separate user interfaces, one to allow the user to index a collection of documents and the other to search that index and find the documents in it. The general usage is to create an index and then use the index as a basis for searching.

The program is started via a DOS batch file (which can be started by double clicking on the file name in the explore (file explorer). There are two batch files, index.bat and search.bat, with, not surprisingly, index.bat starts the indexer interface and search the search interface. For the potential Linux or other Unix user, I have included index.sh and search.sh as batch files for such systems. Do not edit these files with Windows editors! In a Windows environment, each line ends with CR and LF (carriage return, 0D, and line feed, 0A), whereas in a Unix environment, only LF is used.

I have added a possibility to start the program with a default index directory (that is the directory in which the index is to be written and the directory in which the search machine finds its index.)

The program is stored in the directory, C:\searchTest, for example. The directory contains one directory "data" and data contains several files. Some images, tkl.png and lucene.gif used for the entry screen, screen shots for this document, searchresults.css (more later about this), the two configuration files (indexer.defaults and search.defaults) and diverse documents, this included. The searchTest directory contains the following

As one can see this application is based on the work of others, all in the open source community. I would like to explicitly acknowledge this here. The source code of this application is available in the distribution.

New with version 1. The following file types can now can be indexed.

Technical issues on the realization of the file readers. It is possible in Java to bind a class (classes are sort of like programs) while the program is running. This means that for the indexer, we can decide which readers to use via the configuration file. At the time of this writing, I have written the following classes:

# The term filetype followed by the file suffix (if more than one, use comma(,) to separate
# and after the semicolon followed by the class name and path.
filetype.csv:com.theloutons.search.specialreaders.CSVReader
filetype.doc:com.theloutons.search.specialreaders.DOCReader
filetype.xls:com.theloutons.search.specialreaders.XLSReader
filetype.html,htm:com.theloutons.search.specialreaders.HTMLReader
filetype.pdf:com.theloutons.search.specialreaders.PDFReader
filetype.ppt:com.theloutons.search.specialreaders.PPTReader
filetype.rtf:com.theloutons.search.specialreaders.RTFReader
filetype.xml,xsl:com.theloutons.search.specialreaders.XMLReader
filetype.txt:com.theloutons.search.specialreaders.TXTReader
filetype.blk,me,:com.theloutons.search.specialreaders.TXTReader

     

Thus it is possible to choose which types of files one wishes to index as well as the reader. The reader simply extracts the textual information from the file and returns a byte stream (a string, or a series of characters) to the indexer. For any file type the user wishes not to use, put a hash (#) mark before it, as in the first two lines.

Details about the indexing process. At the moment the indexing procedure is limited to filtering out common English language stop words and setting all letters to small case. It would be possible to extend this to be able to choose different analyzing methods.

The decisions to be made on the part of the user are a) where to store the index and b) what to index.

The index must be stored in a completely different place from the collection of documents to be indexed. Also the program is at the moment conceived that it will index only one directory. The suggested procedure then would be to chose the target, the directory in which to write the index and then to chose the source, the directory containing the files to be indexed. The tell the system to begin -- push the button "index now".

A word of CAUTION once you start to create the index, any contents found in the directory chosen will be erased. A log book is written to the directory containing the index (log.txt). This contains a list of files which for some reason or another could not be indexed, as well as the number of files indexed and start and end time. Time is in "Java time", that is number of milliseconds since January 1, 1970, 00:00:00.000 GMT. I do not bother with the date. Further, a sort of copy of the indexer.defaults file so that one can later see what sort of files were indexed.

Figure 1. Indexer Application Screenshot

There is the potential for problems occurring and this section is intended to support the user who may encounter a serious problem. That is problem which stops the application and prevents indexing. The following is a list of the known problems and suggestions on how to handle them.

An "out of memory" exception is an exception I cannot catch and handle. The program stops. I have observed this with PDF and PPT files only.

There is a separate screen to update an index. To use this it is for practical purposes the same as creating an index. One must select the directory which contains the index. Should this directory not contain an index, a message with a note to correct this is given. The directory to containing the documents to be indexed (updated) must not be identical with the original document directory. This means that one can index one directory and at a later point in time add a second one to the original.

How does this work? The program first checks to make sure the index directory really contains an index. Then it recursively (as the indexer) parses the directory containing the documents to be indexed. If a document is not in the index it is added then and there. If it be present in the index the dates are compared and if the file be newer, the old one is removed and the newer version is added.

Figure 2. Index Update Screenshot

There is one type of search realized: a parsed search. Parsed in this case means that the string which is read in is analyzed. This is a very flexible search engine. That is a search can be started with several words using AND OR etc. (See below for complete instructions).

It would be possible to add different search functionality in the future.

Quick introduction to Lucene Parsed Query Search Syntax. The symbol back slash "\" is called in this context the escape symbol, and it is used to "escape characters". If you wish to search for any of the following symbols in your search string, they must be escaped:

\ + - !( ) : ^ ] { } ~ * ?

The use of parentheses is allowed to group boolean terms.

Two words such as bbbb cccc would be taken to mean bbbb OR cccc. If one writes (bbbb and cccc) the system looks for "bbbb" or "and" or "cccc". The word and is a stop word (for us) and therefor never present. Thanks to Peter for noting this!

Wild cards can also be used. By a wild card we mean the use of the symbol * (zero or more characters) ? zero or one character. The wild card character may not begin the search.

Figure 3. Search Interrface Screenshot

The results are presented as HTML files (actually XHTML) and the styles are done with CSS (Cascading Style Sheets). I have a default programmed and there is a possibility to read in a CSS file (searchresults.css) in the directory "data" just in the directory from which the search application starts. Here are a few hints as to how to change the CSS, and a reference to a document called "SelfHTML" by Stefan Münz. If you like this can be downloaded and set up on you PC. I also have it on my server (bes601). This is the best document I have seen on HTML.

In the case of our presentation the CSS is copied to the directory in which the search results are written and is imported into the HTML document. As mentioned above you may edit the file searchresults.css in the data directory to suit your needs or wishes. Here is a sample, an excerpt from a CSS file with each layer colored differently and with many options as examples.

/*The style for the search results*/
body {background-color:#7F7F7F; color:#000000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px; border:double;}
h1 {background-color:#FFFF00; color:#800000;font-family:'Century Schoolbook',serif; font-size:20pt;letter-spacing:3px;padding:10px}
h2 {background-color:#FFBF00; color:#800000;font-family:'Century Schoolbook',serif; font-size:18pt;letter-spacing:3px;}
h3 {background-color:#00FF3F; color:#800000;font-family:'Century Schoolbook',serif; font-size:16pt;letter-spacing:3px;padding:40px;}
h4{background-color:#FF7F00; color:#800000;font-family:'Century Schoolbook',serif; font-size:14pt;letter-spacing:3px;}
h5 {background-color:#00FF3F; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
a {background-color:#FF9933; color:#5940BF;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
p {background-color:#FF9F00; color:#2F2F2F;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
ul {background-color:#FF7F7F; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
ul li {background-color:#FFDF00; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px; 
      border-color:#000000; border-style:solid; border-width:thick; margin-bottom:3em}
ul li p {background-color:#C0C0C0; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
ul li ul {background-color:#FF7F00; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}
ul li ul li {background-color:#C0C000; color:#800000;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px; 
      border-color:#C000C0; margin-bottom:0em}
ul li ul li p {background-color:#F0F0F0; color:#800F00;font-family:'Century Schoolbook',serif; font-size:12pt;letter-spacing:3px;}

With the style sheet one can control the format and layout of the report. I have added a variety of colors to allow you the user to see the effect of each of the colors on the layout. Also it is possible to put "boxes" around items and padding between the text and the "box" (border) and the space between the box and the next item, the margin, can also be formatted.

Recall the basic structure of the output HTML file is the following

<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="Text" name="D.C. Type"/>
<style type="text/css">
@import url(searchresults.css);
</style>
</head>
<body text="#000000" bgcolor="#FFFFFF" link="#FF0000" alink="#FF0000" vlink="#FF0000">
<!-- repeated for each document the list -->
<ul> <!= the top level list, just once in document -->
     <li> <!-- the list item for each document one li for each document-->
        <ul> <!-- list of information for the document -->
            <li> <!-- list item for each type of information, title, synopisis, summay etc. -->
            </li>
        </ul>
     </li>
</ul>
</body>
</html>

The style tag is part of the head (header) tag along with various meta data one may chose to include in the file. The meta data are important for allowing web crawlers and such to classify the document. My experience is that these are often not present or very sloppily filled out. I have included a few, but this is hardly necessary for such a document. In any event the CSS style sheet is either a in a comment section of the style tag, or an external file, e.g., searchresults.css, which is referenced via the import directive (@import url(file name);. In HTML a comment starts with <!-- and ends with -->.

The current default is much simpler and plain. The above is just to support the user in modifying the CSS file to create a "personal" touch to the presentation.

Just a few hints. The tags used to present the data are ul (unordered list) and li (list item). I use them in a nested fashion. That is as below.

<ul>The documents
  <li>Document 1
    <ul>
     <li>File ref for doc 1<a href=.... >file ref </a></li>
     <li>Summary for document 1<p>.....</p></li>
    </ul>
    <ul> .... </ul>
.
.
.
</ul>

The applications are started by a DOS batch files, called index.bat and search.bat. Both of these are set so that the user can set an initial directory for the index. So for example from a DOS box (enter "start" "run" and cmd in the box in the run window.) one could enter the following command from the directory where the application is installed:

index C:\Lucene\IndexDir\full

It is also possible to have a shortcut with the above text as a command.

The same holds for the search.bat file. Actually this is more useful, as one indexes once and searches often.