preg_match(_all) & big strings

[Muumac]

I have problem with large textfiles! When I load over 4MB xml and then
try to preg_match something in this I get always FALSE!
I have <File>....</File> tags in XML. Between tags is files contents
BASE64 encoded! When xml contains only one big over 4MB file in it,
preg_match and preg_match_all doesnt find it! When included file size is
nelow 4MB, preg functions works perfectly!
But PHP manual says that there is no size limitations to strings?!?!

Is this somekind BUG or why doesnt preg work over 4MB matches??? Looks
like problem is when patern-matched string is over 4MB, not the string
where to search!?!?!

How to get work this function? I have portal what should work with large
XML files (contains one ore more BASE64 coded files with unlimited
sizes). How can I get contents of those tags from XML???

/Sorry my bad english, my native language is estonian/

/Muumac

[steve]

"Muumac" wrote:

I have problem with large textfiles! When I load over 4MB xml and then
try to preg_match something in this I get always FALSE!
I have <File>....</File> tags in XML. Between tags is
files contents
BASE64 encoded! When xml contains only one big over 4MB file in it, preg_match and preg_match_all doesnt find it! When included file size is
nelow 4MB, preg functions works perfectly!
But PHP manual says that there is no size limitations to strings?!?!
Is this somekind BUG or why doesnt preg work over 4MB matches??? Looks
like problem is when patern-matched string is over 4MB, not the string
where to search!?!?!

How to get work this function? I have portal what should work with
large
XML files (contains one ore more BASE64 coded files with unlimited sizes). How can I get contents of those tags from XML???

/Sorry my bad english, my native language is estonian/

/Muumac

If all else fails, why don’t you try ereg functions. I think preg
and ereg are written by different developers, so they probably have
different limitations.

If that fails, the other option is to read the file and parse it into
an array, and do a regex by looping thru the array.

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-preg_mat...ict145502.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=487022

[Andy Hassall]

On Wed, 01 Sep 2004 20:25:03 +0300, Muumac <ma*****@hot.ee> wrote:

I have problem with large textfiles! When I load over 4MB xml and then
try to preg_match something in this I get always FALSE!
I have <File>....</File> tags in XML. Between tags is files contents
BASE64 encoded! When xml contains only one big over 4MB file in it,
preg_match and preg_match_all doesnt find it!
First question is why are you trying to parse XML with regular expressions,
instead of an XML parser? You can't create a single regular expression capable
of parsing arbitrary XML as they aren't capable of maintaining enough state
information. They'll work for simple cases, but isn't that defeating part of
the purpose of XML?

http://www.php.net/manual/en/ref.xml.php uses Expat, which streams through the
XML file in manageable chunks, and may be more appropriate for your usage. I
think you'd just set up a handler and watch for <File> elements, and set up a
character data handler for the contents.
When included file size is
nelow 4MB, preg functions works perfectly!
But PHP manual says that there is no size limitations to strings?!?!
Well, given that preg_replace is part of the PCRE library, it might be worth
looking directly in the PCRE docs, which say:

<http://www.pcre.org/pcre.txt>

The maximum length of a subject string is the largest posi-
tive number that an integer variable can hold. However, PCRE
uses recursion to handle subpatterns and indefinite repeti-
tion. This means that the available stack space may limit
the size of a subject string that can be processed by cer-
tain patterns.

OK, so that appears to say for very simple expressions that you can probably
handle about 2GB of data, but that goes down depending on how complex your
expression is - possibly going down a lot, since it's based on "stack space",
if it's referring to the normal stack space that the OS provides processes.
What pattern are you using?
Is this somekind BUG or why doesnt preg work over 4MB matches??? Looks
like problem is when patern-matched string is over 4MB, not the string
where to search!?!?!
Hang on - I think I'm reading that incorrectly, but are you saying you're
supplying a 4MB _pattern_? A pattern that large is pretty absurd, and then
there's this from the docs:

The maximum length of a compiled pattern is 65539 (sic)
bytes if PCRE is compiled with the default internal linkage
size of 2. If you want to process regular expressions that
are truly enormous, you can compile PCRE with an internal
linkage size of 3 or 4 (see the README file in the source
distribution and the pcrebuild documentation for details).
If these cases the limit is substantially larger. However,
the speed of execution will be slower.
How to get work this function? I have portal what should work with large
XML files (contains one ore more BASE64 coded files with unlimited
sizes). How can I get contents of those tags from XML???

Another point to look out for is PHP's memory_limit directive - if you're
loading the whole lot into memory and processing it you're likely to hit that
limit. That's probably another point where the XML parser functions above will
win out, as you can read it in bits and write it off into a file rather than
having the whole lot in memory at one time.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>
<http://www.andyhsoftware.co.uk/space> Space: disk usage analysis tool