Finding messages in huge mboxes

[Bastiaan Welmers]

Hi,

I wondered if anyone has ever met this same mbox issue.

I'm having the following problem:

I need find messages in huge mbox files (50MB or more).
The following way is (of course?) not very usable:

fp = open("mbox", "r")
archive = mailbox.UnixMailbox(fp)
i=0
while i < message_number_needed:
i+=1
archive.next()

needed_message = archive.next()

Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

i=0
j=0
while 1:
i+=1
fp.seek(-i, SEEK_TO_END=2)
line = fp.readline()
if not line:
break
if line[:5] == 'From ':
j+=1
if j == total_messages - message_number_needed:
archive.seekp = fp.tell()
message = archive.next()
# message found

But also seems to be slow and CPU consuming.

Anyone who has a better idea?

Regards,

Bastiaan Welmers

[Miklós]

What about putting it into a database like MySQL? <pyWink>

Miklós
"Bastiaan Welmers" <ha****@welmers.net> wrote in message
news:40*********************@news.xs4all.nl...

Hi,

I wondered if anyone has ever met this same mbox issue.

I'm having the following problem:

I need find messages in huge mbox files (50MB or more).
The following way is (of course?) not very usable:

fp = open("mbox", "r")
archive = mailbox.UnixMailbox(fp)
i=0
while i < message_number_needed:
i+=1
archive.next()

needed_message = archive.next()

Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

i=0
j=0
while 1:
i+=1
fp.seek(-i, SEEK_TO_END=2)
line = fp.readline()
if not line:
break
if line[:5] == 'From ':
j+=1
if j == total_messages - message_number_needed:
archive.seekp = fp.tell()
message = archive.next()
# message found

But also seems to be slow and CPU consuming.

Anyone who has a better idea?

Regards,

Bastiaan Welmers

[Diez B. Roggisch]

> Anyone who has a better idea?

AFAIK MUAs usually use a mbox.index-file for faster access. The index is
computed once, and updated whenever a new message is added. You could
create this index quite easily yourself by looping over the mbox and
pickling a list of tell'ed positions. If you also store the creation-date
of the index and the filesize of the mbox-file, you should be able to
create a function that will update the index whenever the underlying mbox
has changed. Another approach would be to perform index-creation on regular
bases using cron.

Regards,

Diez

[Donn Cave]

In article <40*********************@news.xs4all.nl>,
Bastiaan Welmers <ha****@welmers.net> wrote:
....

I need find messages in huge mbox files (50MB or more). .... Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

readline() is not your friend here. I suggest that
you read large blocks of data, like 8192 bytes for
example, and search them iteratively. Like,
next = block.find('\nFrom ', prev + 1)

This will give you the location of each message in
the current block, so you can split the block up
into a list of messages. (There will be an extra
chunk of data at the beginning of each block, before
the first "From " - recycle that onto the end of the
next block.)

Since file object buffering is at best useless in this
application, I would use posix.open, posix.lseek and
posix.read. Taking this approach, I find that reading
the last 10 messages in a 100 Mb folder takes 0.05 sec.

Donn Cave, do**@u.washington.edu

[David M. Cooke]

At some point, Donn Cave <do**@u.washington.edu> wrote:

In article <40*********************@news.xs4all.nl>,
Bastiaan Welmers <ha****@welmers.net> wrote:
...

I need find messages in huge mbox files (50MB or more).

...

Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

readline() is not your friend here. I suggest that
you read large blocks of data, like 8192 bytes for
example, and search them iteratively. Like,
next = block.find('\nFrom ', prev + 1)

Unless, of course, you read '\nFr', then 'om ' in the next block...

I can't think of a simple way around this (except for reading by
lines). Concating the last two together means having to keep track of
what you've seen in the last block. Maybe picking off the last line
from the last block (using line.rfind('\n')), and concatenating that
to the beginning of the next.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca

[Donn Cave]

Quoth co**********@physics.mcmaster.ca (David M. Cooke):
| At some point, Donn Cave <do**@u.washington.edu> wrote:
|> In article <40*********************@news.xs4all.nl>,
|> Bastiaan Welmers <ha****@welmers.net> wrote:
|> ...
|>> I need find messages in huge mbox files (50MB or more).
|> ...
|>> Especially because I often need messages at the end
|>> of the MBOX file.
|>> So I tried the following (scanning messages backwards
|>> on found "From " lines with readline())
|>
|> readline() is not your friend here. I suggest that
|> you read large blocks of data, like 8192 bytes for
|> example, and search them iteratively. Like,
|> next = block.find('\nFrom ', prev + 1)
|
| Unless, of course, you read '\nFr', then 'om ' in the next block...
|
| I can't think of a simple way around this (except for reading by
| lines). Concating the last two together means having to keep track of
| what you've seen in the last block. Maybe picking off the last line
| from the last block (using line.rfind('\n')), and concatenating that
| to the beginning of the next.

I'm reading from the end backwards, so the fragment is block[:start].
Append that to the block before it, and each block always will end at
a message boundary. If you start in the middle, you have to deal with
an extra boundary problem. If reading forward from the beginning, it
would be about as simple.

If I have overlooked some obvious problem with this, it wouldn't be
the first time, but I think it's as simple as it could be. The only
inelegance to it is that you have to scan the fragment at least twice
(one extra time for each time it's added to a new block.)

Donn Cave, do**@drizzle.com

[Miki Tebeka]

Hell Bastiaan,

I need find messages in huge mbox files (50MB or more).
...
Anyone who has a better idea?

I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.

--- last.py ---
#!/usr/bin/env python
from os import popen
from sys import argv

# Find last "From:" line
last = popen("grep -n 'From:' %s | tail -1" % argv[1]).read()
last = int(last.split(":")[0])
# Find total number of lines
size = popen("wc -l %s" % argv[1]).read()
size = int(size.split()[0].strip())
# Print the message
print popen("tail -%d %s" % (size - last, argv[1])).read()
--- last.py ---
Tool less than 1sec on my computer on a 11MB mailbox.

HTH.
Miki

[Cameron Laird]

In article <4f**************************@posting.google.com >,
Miki Tebeka <mi*********@zoran.com> wrote:

Hell Bastiaan,

I need find messages in huge mbox files (50MB or more).
...
Anyone who has a better idea?

I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.

--- last.py ---
#!/usr/bin/env python
from os import popen
from sys import argv

# Find last "From:" line
last = popen("grep -n 'From:' %s | tail -1" % argv[1]).read()
last = int(last.split(":")[0])
# Find total number of lines
size = popen("wc -l %s" % argv[1]).read()
size = int(size.split()[0].strip())
# Print the message
print popen("tail -%d %s" % (size - last, argv[1])).read()
--- last.py ---
Tool less than 1sec on my computer on a 11MB mailbox.

[Erno Kuusela]

Bastiaan Welmers <ha****@welmers.net> writes:

Especially because I often need messages at the end
of the MBOX file.
So I tried the following (scanning messages backwards
on found "From " lines with readline())

i=0
j=0
while 1:
i+=1
fp.seek(-i, SEEK_TO_END=2)
line = fp.readline()
if not line:
break
if line[:5] == 'From ':
j+=1
if j == total_messages - message_number_needed:
archive.seekp = fp.tell()
message = archive.next()
# message found

But also seems to be slow and CPU consuming.

something like this might work. the loop below scanned a 115MB mailbox
in about 1 second on a 1.2ghz k7. extracts the next-to-last message,
but you get the idea. if you don't want to read the file into cache,
you could adapt it to start with a smaller mmapped chunk from the end
of the file and enlarge it until you find what you want.
import os, re, mmap, sys
from cStringIO import StringIO
import email

fd = os.open(sys.argv[1], os.O_RDONLY)
size = os.fstat(fd).st_size
print size
buf = mmap.mmap(fd, size, access=mmap.ACCESS_READ)
message_offsets = []
for m in re.finditer(r'(?s)\n\nFrom', buf):
message_offsets.append(m.start())

msgfp = StringIO(buf[message_offsets[-2] + 2:message_offsets[-1] + 2])
msg = email.message_from_file(msgfp)
print msg['to']

-- erno

[Bastiaan Welmers]

Miki Tebeka wrote:

Hell Bastiaan,

I need find messages in huge mbox files (50MB or more).
...
Anyone who has a better idea?

I find that sometime using the unix little utilties (which are
available for M$ as well) gives very good performance.

Sounds as a very good idea. Tanks.

/Bastiaan

[Bastiaan Welmers]

Miklós wrote:

What about putting it into a database like MySQL? <pyWink>

Too much work to archieve this. It's just a Mailman archieve mbox
which has to be opened. So then I have to rewrite
pipermail archiever.

/Bastiaan

[Bastiaan Welmers]

Diez B. Roggisch wrote:

Anyone who has a better idea?

AFAIK MUAs usually use a mbox.index-file for faster access. The index is
computed once, and updated whenever a new message is added. You could
create this index quite easily yourself by looping over the mbox and
pickling a list of tell'ed positions. If you also store the creation-date
of the index and the filesize of the mbox-file, you should be able to
create a function that will update the index whenever the underlying mbox
has changed. Another approach would be to perform index-creation on
regular bases using cron.

Also good idea. It's a mailman archieve so then I have
to hack mailman for creating an index file besides the
mbox file.

/Bastiaan