Text retrieval systems - 2B: Text Processors

[JosAH]

Greetings,

Introduction

welcome back. It's time to do some real design: I want two have two 'things':

1) a 'LibraryBuilder ' that gradually builds the processed text and finally
builds the 'Library' itself.

2) a 'Processor' that processes the input text and spoonfeeds it to the first
object.

Processor

I want these two entities to be as general as possible. First I design the
wanted interfaces. Here is the Processor interface:

public interface Processor {
 
    public void process(String prefix) throws IOException;
 
    public Library getLibrary();
}
 

The prefix String can be any string; the Processor knows what to do with it.
The prefix can be a uri or a directory or whatever is needed to get to the
raw text.

The process() method does all the processing; and because things can go wrong
during processing it is allowed to throw an IOException which is the most
likely exception that can be thrown. I'll design sub classes thereof when
needed.

The second method gives me the end result: the Library. A Library is an ordinary
class that can retrieve text for me.

I want a Processor implementation to be as generic as possible, i.e. I don't
want to stick any particular King James bible knowledge into my Processor.

The Processor implementation will be an abstract class that does all the
organizational or 'conducting' work and leaves the particular King James bible
knowledge to a subclass. It implements abstract methods for that purpose.

LibraryBuilder

I use the same scenario for a LibraryBuilder:

public interface LibraryBuilder {
 
    public void preProcess();
    public void postProcess();
 
    public void setTitle(String title);
 
    public void buildGroup(String group);
    public void buildParagraph(String book, String chapter, 
                   int para, String text) throws IOException;
 
    public Library build();
}
 

The interface can't enforce it, but the intention is to call the preProcess()
method before anything else is done. After all processing is over and done
with, the postProcess() method is supposed to be called.

At the very end the LibraryBuilder is supposed to give me a Library object
when its build() method is invoked.

The Library class itself doesn't know which text it handles, i.e. it knows
nothing about King James bible texts, nor about CD collections or whatever.

The two remaining methods implement the text spoonfeeding:

1) buildGroup() builds a new group for its caller.
2) buildParagraph( ) builds a new paragraph given a book, chapter, paragraph
and the raw paragraph text. It may throw an IOException if needed.

As already can be seen, when the Processor.getLi brary() method is invoked it
delegates the job to the LibraryBuilder. build() method.

Here too, I want a LibraryBuilder to be as generic as possible so I implement
an abstract class that implements the LibraryBuilder interface. This class does
all the work that doesn't need any particular knowledge about the King James
bible text.

A special subclass should implement the abstract methods defined in the abstract
super class in which it can stick its specific King James bible text.

Class structure

This is the top level class structure:

// interfaces:
interface Processor { ... }
interface LibraryBuilder { ... }
// implementing classes:
abstract class AbstractProcessor() implements Processor { ... }
abstract class AbstractBuilder() implements LibraryBuilder { ... }
 

For this particular example project I have to implement two specific classes:

class KJProcessor extends AbstractProcessor { ... }
class KJBuilder extends AbstractBuillder { ... }
 

These two classes contain specific King James bible knowledge about the text
being processed and from which a Library is constructed by the builder.

AbstractProcess or

The AbstractProcess or does all the 'conducting' work for the raw text processing
job. It needs to be subclassed for the real job. Here is its first part:

public abstract class AbstractProcessor implements Processor {
 
    protected LibraryBuilder builder;
    protected String title;
 
    public AbstractProcessor(String title, LibraryBuilder builder) {
 
        this.builder= builder;
        this.title= title;
    }
    ...
 

An AbstractProcess or can be constructed given the title of the library and
a LibraryBuilder. The KJProcessor supplies the KBBuilder for its superclass
as well as the title String.

The abstract methods defined in this class are:

    ...
    protected abstract void preProcess();
    protected abstract void postProcess();
 
    protected abstract int getNofBooks();
    protected abstract String getBookTitle(String prefix, int book);
 
    protected abstract Reader getBookReader(String prefix, int book) 
                throws IOException;
 
    protected abstract void processBook(String title, BufferedReader br) 
                throws IOException;
    ...
 

Similar to the LibraryBuilder this object calls the preProcess() method before
processing starts. When the processing is done the postProcess() method is
invoked. The KJProcessor implements empty methods for these two abstract methods
because it doesn't need to do any special pre- or post processing.

The AbstractProcess or needs to know how many books are to be processed and it
needs the title of each book. That's what the next two methods are for and
they need to be implemented in a subclass of the AbstractProcess or class.

The getBookReader() method needs to return a Java Reader object that can read
from a book. The last method must process an entire book, given a Reader for
that book.

The last two methods can throw an IOException because anything input/output
related actions can go wrong.

Note that the subclass can invoke methods and read or alter member variables
in the builder directly, i.e. the coupling between the two is tight.

Here's the delegator method when a Library object is wanted:

    ...
    public Library getLibrary() { return builder.build(); }
    ...
 

Also see above: the AbstractProcess or simply invokes the builder.build() method
for the Library.

Now for some substantial conducting work. The next method in the AbstractProcess or
class is the implementation of the process() method defined in the Processor
interface:

    ...
    public void process(String prefix) throws IOException {
 
        builder.preProcess();
        builder.setTitle(title);
 
        this.preProcess();
 
        for (int i= 0, n= getNofBooks(); i < n; i++)
            processBook(prefix, i);
 
        this.postProcess();
 
        builder.postProcess();
    }
    ...
 

It calls the preProcess() methods on the builder and the subclass and it
passes the title to the builder.

Next it determines the number of books to be processed and processes each
book by invoking the processBook() method (see below).

When everything succeeds the postProcess() method is invoked on both the
subclass and the builder.

Here's the processBook() method implementation:

    ...
    public void processBook(String prefix, int book) throws IOException {
 
        BufferedReader br= null;
 
        try {
            br= new BufferedReader(getBookReader(prefix, book));
            processBook(getBookTitle(prefix, book), br);
        }
        finally {
            try { br.close(); } catch (IOException ioe) { }
        }
    }
 

This methods asks the subclass to return a Reader given a book. It wraps
a BufferedReader around the Reader and asks the subclass again to process
the current book. Finally the buffered reader is closed again, which closes
the wrapped reader itself.

I think this is enough design and implementation for this week. Next week I'll
show how the LibraryBuilder is designed and implemented. It's more work than
this Processor implementation.

After that I'll show the KJProcessor and KJBuilder classes; they handle the
nitty-gritty String processing work and are basically the implementations
of the abstract methods defined in their parent classes and a few ugly methods
that must come up with consistent text (see the last week's article part).

I'll add all the code as an attachment in some of the following article parts so
you can play with it or maybe actually apply it in a useful way. It doesn't
hurt to actually read the source code. If you find bugs feel free to correct me.

See you next week and

kind regards,

Jos