Text retrieval systems - 4A: the Library

[JosAH]

Greetings,

the last two article parts described the design and implementation of the
text Processor which spoonfeeds paragraphs of text to the LibraryBuilder.
The latter object organizes, cleans up and stores the text being fed to it.
Finally the LibrayBuilder is able to produce a Library which is the topic
of this part of the article.

Introduction

A Library manages groups of books. Each book contains chapters which contain
paragraphs of text. The object can store itself in a compressed way given
any OutputStream. A static method can load Library objects given any Input-
Stream again. A Library can produce pieces of text, either entire books or
chapters or a single paragraph of text.

A Library can also produce lists of 'meta' data, i.e. it can produces a list
of group names of book or chapter titles; it can also produce simple statistics
such as the number of books in the library etc.

Basically a Library can produce Lists. The Lists are either lists of pure
text (Strings) or Lists of BookMarks. A BookMark represents a single paragraph
of text and has additional functionality as well.

Let's see how it's all done.

The Library object

In the previous part in the article we already saw a Library constructor being
invoked by the LibraryBuikder; here are the data parts and de constructor of
the Library object:

protected String    title;
protected WordMap   wordMap;
protected Section[] groups;    
protected Section[] books;
protected Section[] chapters;
protected String[]  paragraphs;
protected String[]  words;
protected String[]  punctuation;
 
protected NotesMap  notesMap;
 
public Library(String    title,
           WordMap   wordMap,
           Section[] groups,
           Section[] books, 
           Section[] chapters,
           String[]  paragraphs, 
           String[]  words, 
           String[]  punctuation) {
 
 
    this.title        = title;
    this.wordMap    = wordMap;
 
    this.groups        = groups;
    this.books      = books;
    this.chapters   = chapters;
    this.paragraphs = paragraphs;
    this.words      = words;
    this.punctuation= punctuation;
 
    this.notesMap   = new NotesMap();
}
 

This is quite a bit of code but it is just simple stuff; all the parameters of
the constructor are stored in the member variables. As already discussed, there
are Sections for groups, books and chapters. The paragraphs and punctuations
are strings. There's a A String[] array for the unique words and there's the
WordMap that glues it all together. See the previous article part for an
explanation of these arrays and objects.

Also a NotesMap is initialized. a NoteMap is used for additional text for every
single paragraph in the text. Here's its definition:

class NotesMap extends HashMap<Integer, String> {
 
    private static final long serialVersionUID = 4627674436919142999L;
 
    private boolean modified= false;
 
    public void setModified(boolean modified) {this.modified= modified; }
    public boolean isModified() { return modified; }
 
    public void put(int paragraph, String note) {
 
        String oldNote= get(paragraph);
 
        note= (note != null)?note.trim():"";
        note= (note.length() == 0)?null:note;
 
        if (note == null) 
            modified|= remove(paragraph) != null;
        else {
            put(paragraph, note);
            modified|= !note.equals(oldNote);
        }
    }
}
 

Basically a NotesMap is a simple Map that maps Integers (the index number of
a paragraph) to a simple String that stores the annotations for the paragraph.

It applies a bit of intelligence to check whether or not the map was modified.
Empty text is considered no text at all. The modified flag can be used to check
whether or not the Library object should be saved again when it isn't needed
anymore.

The Library object implements a little convenience method so you never have
to deal with a NoteMap object yourself directly:

public boolean isModified() { return notesMap.isModified(); }
 

The NotesMap is not a very complicated nor a very interesting class except for
one aspect: it implements the Serializable interface because its parent class
implements this interface. This is needed because this class must be saved or
loaded along with the Library object itself.

As we'll see later the Library class itself also implements the Serializable
interface.

BookMarks

As briefly mentioned in the introduction, a Library uses BookMarks to represent
one single paragraph of text. Here's its interface definition:

public interface BookMark {
 
    public String  getGroup();
    public String  getBook();
    public String  getChapter();
    public String  getParagraph();
 
    public int     getGroupNumber();
    public int     getBookNumber();
    public int     getChapterNumber();
    public int     getParagraphNumber();
 
    public int     getRelativeBook();
    public int     getRelativeChapter();
    public int     getRelativeParagraph();
 
    public void    putNote(String note);
    public String  getNote();
}
 

If the Library has produced a BookMark object for you, you can query that object
for several aspects of that text, i.e. you can get the name of the chapter the
paragraph is part of; you can also get the name of the book where the paragraph
can be found and you can find the name of the group where the book is stored.
You can also get the text of the paragraph of course.

Paragraphs, chapters, books and groups all have a number, starting at zero.
There are methods available to retrieve those absolute index numbers. There
are also methods available to retrieve relative numbers for those entities.
Suppose there are two chapters in the first book and three chapters in a
second book. The absolute chapter numbers are 0, 1, 2, 3, 4 and the relative
chapter numbers are 0, 1, 0, 1, 2, i.e. the relative chapter numbers start
at zero per book. A similar reasoning applies to books and paragraphs as well.

Note that for obvious reasons there is no relative group number available.

The last two methods set and get optional additional text given the paragraph
represented by this bookmark. These two methods together with the Library
delegator isModified() method completely shield the NotesMap class from you,
i.e. you never have to deal with it yourself.

Loading and Saving a Library

If you want to load a library given an InputStream you don't have that object
available yet (it isn't loaded yet). That's why the Library class implements
two static convenience methods: one method can load a Library given a file name,
the other method takes an InputStream from which it loads a Library object.

Here they are:

public static Library read(String name) 
            throws IOException, ClassNotFoundException {
 
    FileInputStream fis= null;
 
    try {
        return read(fis= new FileInputStream(name));
    }
    finally {
        try { fis.close(); } catch (IOException ioe) { }
    }
}
 
public static Library read(InputStream is) 
            throws IOException, ClassNotFoundException {
 
    return (Library)new ObjectInputStream(new InflaterInputStream(is)).
            readObject();
}
 

The first method just tries to open a FileInputStream given a file name and
delegates the hard part to the second method. The second method wraps two
other InputStreams around the given InputStream. The first wrapper/decorator
inflates the input stream. The inflated input is used by the ObjectInputStream
wrapper that does the actual object construction. Both methods can throw simple
IOExceptios when some reading fails and the can throw ClassNotFoundExceptions
if the InputStream didn't make any sense.

The second method doesn't close the stream passed to it because it doesn't own
it, i.e. maybe the caller of this method has different plans for this stream,
and closing it will prohibit it. The first method had created that OutputStrem
itself, so it owns it and so it closes the stream itself.

Of course you can only save a Library object if you have one. Therefore the
write methods are not static methods. The logic of those methods don't differ
much from the static load methods. Here they are:

public void write(String name) throws IOException {
 
    FileOutputStream fos= null;
 
    try {
        fos= new FileOutputStream(name);
        write(fos);
    }
    finally {
        try { fos.close(); } catch (IOException ioe) { }
    }
}
 
public void write(OutputStream os) throws IOException {
 
    DeflaterOutputStream dos;
    ObjectOutputStream oos= new ObjectOutputStream(dos= new DeflaterOutputStream(os));
 
    oos.writeObject(this);
    oos.flush();
    dos.finish();
}
 

The first method just creates a FileOutputStream given a file name and delegates
the hard work to the second method. Compare this with the static load method.
The other method wraps a DeflaterOutputStream around the OutputStream parameter.
This DeflaterOutputStream is wrapped in an ObjectOutputStream again and the
entire Library is written.

Also observe that the second method doesn't close the stream for identical
reasons as explained for the load methods (compare the logic again).

I noticed that this article part is getting quite long again, so I see you below in
the sequel of this article part.

kind regards,

Jos