Question for a "super-man"!! (pop3, mails...)

[WebRod]

Hi everybody,

I just would like to download the new mails I received on my POP3 server
(and keep a copy on the server)
I use PHP 4.3 and IMAP functions (they are compatible with both POP3 and
IMAP servers).

When I use OUTLOOK, no problem, in less than 5 seconds, it answers me:
" you have XX new messages"

But, I can't manage to do the same thing within the same time in PHP!!!

I believe it's a IMAP functions problem.

First, I thought that each new mail was the last message returned by the
IMAP functions.
For exemple, when I connect to the POP3 servers, I get the total number of
messages.
Let's say I have 600 messages, the 600th messages is supposed to be the last
message.
So, here is the process:
1- X=total number of messages
2- I download the Xth message
3- Is it a NEW message?
YES-> download it, X=X-1, then go to step 2
NO -> Go to step 4
4- End

So, if I have already downloaded the 600th message, obviously, i know I have
already dowloaded ALL the mails, and I can say immediately "no new message".

BUT, this is wrong with some servers (like ovh.com).
They do not sort the emails like this!!
If I send a new mail to myself, it will not be the last message!!! It will
appear for example in the middle of the list!!!!!!!!!!!!!!!!!!!!!!

So the only solution is:
1- Download the whole list of emails!!!!!!!!
2- compare EACH mail on the server with the locals mails!!
3- If the mail on the server is not on my local machine, then I download it

Suppose I have no new mail, I read 600 mails for nothing!!
So it takes more than 1 minute just to say: "no mail"!

Do you have another idea?

Some important points with POP3:
- no uid!! Even the field MESSAGE_ID is not usedul. Sometimes it is empty!!
- udate is sometimes empty!! (0)
- date can have several formats
- when you use the SORT imap function (or search) it download the whole
list!! So it makes a lot of time!!
How can OUTLOOK know the number of new mail in less than 5 seconds (even
with OVH)????
Can I do the same thing with PHP scripts?????

thanks for your help

Rod

[Markku Uttula]

WebRod wrote:

First, I thought that each new mail was the last message returned by
the IMAP functions.
Nope, this ain't the case. First thing to remember with POP3 is that
you can *never* assume the messages are in the same order on two
subsequent connections - the standard doesn't specify the order in any
way.
For exemple, when I connect to the POP3 servers, I get the total
number of messages.
Yes. And there's actually more than one way to accomplish that :)
Let's say I have 600 messages, the 600th messages is supposed to be
the last message.
So, here is the process: [SNIP] So, if I have already downloaded the 600th message, obviously, i
know
I have already dowloaded ALL the mails, and I can say immediately
"no
new message".
As I said, you can't count on this.
Some important points with POP3:
- no uid!! Even the field MESSAGE_ID is not usedul. Sometimes it is
empty!!
This isn't actually the case. POP3 has extended command UIDL, which
returns you a list of unique identifiers (just like LIST returns the
sizes of messages). After this, you only need to check the UID with
stored UIDs of the messages you've already downloaded. This is quite
fast even with thousands of messages.

The downside is that not all servers provide UIDL-command since it's
not required by the standard. If this is the case, the only solution
is to download message headers (using command like TOP) only and
comparing them to headers already received. This takes somewhat longer
(but not as long as it'd take to download the whole messages) and
isn't as reliable as using UIDs.
How can OUTLOOK know the number of new mail in less than 5 seconds
(even with OVH)????

For example (O=Outlook, S=Server, indented for readability):

S: +OK Welcome
O: USER example
S: +OK
O: PASS example
S: +OK
O: UIDL
S: +OK List follows
1 abcdef
2 abcdeg
3 abcdeh
4 abcdei
5 abcdej

[Markku Uttula]

One more thing... the user contribution at
http://www.php.net/manual/en/function.imap-uid.php isn't too good, put
it gives good pointers on how to proceed. To be honest, I don't
believe it works to begin with - fgets shouldn't be used on a protocol
like POP3 where we can't know beforehand how much data we should
receive :(

--
Markku Uttula

[Markku Uttula]

Markku Uttula wrote:

One more thing... the user contribution at
http://www.php.net/manual/en/function.imap-uid.php isn't too good,
put
it gives good pointers on how to proceed. To be honest, I don't
believe it works to begin with - fgets shouldn't be used on a
protocol
like POP3 where we can't know beforehand how much data we should
receive :(

Oops - my bad... fgets only does reading up to newline. Sorry to bash
the user contribution.

--
Markku Uttula

[WebRod]

many thanks for all your messages.
Actually, irelaised that is should not try to do that with imap functions.

I now use POP PEAR functions and it does exactly what you are saying.

So many thanks.
The problem with this function is that the mails are not formatted.
I get some text like this:
=?ISO-8859-1?Q?eBay_-_F=E9licitations=A0!_Achat_?
Anyway, i use mb_decode_mimeheader and the problem is solved.
A last one, sometimes i get "_" instead of spaces!!
Before or even after i use mb_decode_mimeheader.

Any idea?
many thanks!

"Markku Uttula" <ma***********@disconova.com> a écrit dans le message de
news: 41***********************@news.yhteys.mtv3.fi...

WebRod wrote:

First, I thought that each new mail was the last message returned by
the IMAP functions.

Nope, this ain't the case. First thing to remember with POP3 is that you
can *never* assume the messages are in the same order on two subsequent
connections - the standard doesn't specify the order in any way.

For exemple, when I connect to the POP3 servers, I get the total
number of messages.

Yes. And there's actually more than one way to accomplish that :)

Let's say I have 600 messages, the 600th messages is supposed to be
the last message.
So, here is the process:

[SNIP]

So, if I have already downloaded the 600th message, obviously, i know
I have already dowloaded ALL the mails, and I can say immediately "no
new message".

As I said, you can't count on this.

Some important points with POP3:
- no uid!! Even the field MESSAGE_ID is not usedul. Sometimes it is
empty!!

This isn't actually the case. POP3 has extended command UIDL, which
returns you a list of unique identifiers (just like LIST returns the sizes
of messages). After this, you only need to check the UID with stored UIDs
of the messages you've already downloaded. This is quite fast even with
thousands of messages.

The downside is that not all servers provide UIDL-command since it's not
required by the standard. If this is the case, the only solution is to
download message headers (using command like TOP) only and comparing them
to headers already received. This takes somewhat longer (but not as long
as it'd take to download the whole messages) and isn't as reliable as
using UIDs.

How can OUTLOOK know the number of new mail in less than 5 seconds
(even with OVH)????

For example (O=Outlook, S=Server, indented for readability):

S: +OK Welcome
O: USER example
S: +OK
O: PASS example
S: +OK
O: UIDL
S: +OK List follows
1 abcdef
2 abcdeg
3 abcdeh
4 abcdei
5 abcdej
.
at this point Outlook can check if it has some or all of the messages 1
through 5 and retrieve those it hasn't.

Can I do the same thing with PHP scripts?????

Certainly. I'd suggest reading RFC-1939 as starters
http://www.faqs.org/rfcs/rfc1939.html - that way you should get to know
how POP3 really works. It's not really much to read (when compared to
something like IMAP, which is a pain in the ...)

After that, check http://www.php.net/manual/en/function.imap-uid.php - to
be more precise, check the last user contributed comment on the page.
After you've read the RFC, you see the comment is meant to only retrieve
UID of *one* message (which of course is very unefficient, since you can
get them all during one session). Also, remember what I said about the
messages not necessarily being in the same order on next connection - you
need to retrieve the messages you want (if you're not only counting them)
during the same session... ie. do not issue QUIT before you're done.

And last but not least - though this propably is quite obvious - you need
to remember that it's imperative you store locally the UIDs of the
messages you've already "received"/"checked"/whatever. Otherwise this
won't work.

--
Markku Uttula