cElementTree clear semantics

[Igor V. Rafienko]

Hi,
I am trying to understand how cElementTree's clear works: I have a
(relatively) large XML file, that I do not wish to load into memory.
So, naturally, I tried something like this:

from cElementTree import iterparse
for event, elem in iterparse("data.xml"):
if elem.tag == "schnappi":
count += 1
elem.clear()

.... which resulted in caching of all elements in memory except for
those named <schnappi> (i.e. the process' memory footprint grew more
and more). Then I though about clear()'ing all elements that I did not
really need:

from cElementTree import iterparse
for event, elem in iterparse("data.xml"):
if elem.tag == "schnappi":
count += 1
elem.clear()

.... which gave a suitably small memory footprint, *BUT* since
<schnappi> has a number of subelements, and I subscribe to
'end'-events, the <schnappi> element is returned after all of its
subelements have been read and clear()'ed. So, I see indeed a
<schnappi> element, but calling its getiterator() gives me completely
empty subelements, which is not what I wanted :(

Finally, I thought about keeping track of when to clear and when not
to by subscribing to start and end elements (so that I would collect
the entire <schnappi>-subtree in memory and only than release it):

from cElementTree import iterparse
clear_flag = True
for event, elem in iterparse("data.xml", ("start", "end")):
if event == "start" and elem.tag == "schnappi":
# start collecting elements
clear_flag = False
if event == "end" and elem.tag == "schnappi":
clear_flag = True
# do something with elem
# unless we are collecting elements, clear()
if clear_flag:
elem.clear()

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

Thanks in advance,

ivr
--
"...but it's HDTV -- it's got a better resolution than the real world."
-- Fry, "When aliens attack"

[D H]

Igor V. Rafienko wrote:

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

Try emailing the author for support.

[Reinhold Birkenfeld]

D H wrote:

Igor V. Rafienko wrote:

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

Try emailing the author for support.

I don't think that's needed. He is one of the most active members
of c.l.py, and you should know that yourself.

Reinhold

[D H]

Reinhold Birkenfeld wrote:

D H wrote:

Igor V. Rafienko wrote:

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

Try emailing the author for support.

I don't think that's needed. He is one of the most active members
of c.l.py, and you should know that yourself.

I would recommend emailing the author of a library when you have a
question about that library. You should know that yourself as well.

[Reinhold Birkenfeld]

D H wrote:

Reinhold Birkenfeld wrote:

D H wrote:

Igor V. Rafienko wrote:

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?
Try emailing the author for support.

I don't think that's needed. He is one of the most active members
of c.l.py, and you should know that yourself.

I would recommend emailing the author of a library when you have a
question about that library. You should know that yourself as well.

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold

[D H]

Reinhold Birkenfeld wrote:

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold
Reinhold Birkenfeld also wrote: If I had wanted to say "you have opinions? fuck off!", I would have said
"you have opinions? fuck off!".

Take your own advice asshole.

[D H]

D H wrote:

Reinhold Birkenfeld wrote:

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold

Reinhold Birkenfeld also wrote:

> If I had wanted to say "you have opinions? fuck off!", I would have said
>"you have opinions? fuck off!".

Take your own advice asshole.

[Reinhold Birkenfeld]

D H wrote:

Reinhold Birkenfeld wrote:

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold

Reinhold Birkenfeld also wrote:

> If I had wanted to say "you have opinions? fuck off!", I would have said
>"you have opinions? fuck off!".

Take your own advice asshole.

QED. Irony tags for sale.

Reinhold

[Reinhold Birkenfeld]

D H wrote:

D H wrote:

Reinhold Birkenfeld wrote:

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold

Reinhold Birkenfeld also wrote:

> If I had wanted to say "you have opinions? fuck off!", I would have said
>"you have opinions? fuck off!".

Take your own advice asshole.

And what's that about?

Reinhold

[D H]

Reinhold Birkenfeld wrote:

D H wrote:

D H wrote:

Reinhold Birkenfeld wrote:
Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Reinhold
Reinhold Birkenfeld also wrote:
> If I had wanted to say "you have opinions? fuck off!", I would have said
>"you have opinions? fuck off!".
Take your own advice asshole.

And what's that about?

I think it means you should fuck off, asshole.

[Reinhold Birkenfeld]

D H wrote:

Reinhold Birkenfeld wrote:

D H wrote:

D H wrote:

Reinhold Birkenfeld wrote:
>Well, if I had e.g. a question about Boo, I would of course first ask
>here because I know the expert writes here.
>
>Reinhold
Reinhold Birkenfeld also wrote:
> If I had wanted to say "you have opinions? fuck off!", I would have said
>"you have opinions? fuck off!".
Take your own advice asshole.

And what's that about?

I think it means you should fuck off, asshole.

I think you've made that clear.

*plonk*

Reinhold

PS: I really wonder why you get upset when someone except you mentions boo.

[Fredrik Lundh]

Igor V. Rafienko wrote:

Finally, I thought about keeping track of when to clear and when not
to by subscribing to start and end elements (so that I would collect
the entire <schnappi>-subtree in memory and only than release it):

from cElementTree import iterparse
clear_flag = True
for event, elem in iterparse("data.xml", ("start", "end")):
if event == "start" and elem.tag == "schnappi":
# start collecting elements
clear_flag = False
if event == "end" and elem.tag == "schnappi":
clear_flag = True
# do something with elem
# unless we are collecting elements, clear()
if clear_flag:
elem.clear()

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

the iterparse/clear approach works best if your XML file has a
record-like structure. if you have toplevel records with lots of
schnappi records in them, iterate over the records and use find
(etc) to locate the subrecords you're interested in:

for event, elem in iterparse("data.xml"):
if event.tag == "record":
# deal with schnappi subrecords
for schappi in elem.findall(".//schnappi"):
process(schnappi)
elem.clear()

the collect flag approach isn't that bad ("twice as slow" doesn't
really say much: "raw" cElementTree is extremely fast compared
to the Python interpreter, so everything you end up doing in
Python will slow things down quite a bit).

to make your application code look a bit less convoluted, put the
logic in a generator function:

# in library
def process(filename, annoying_animal):
clear = True
start = "start"; end = "end"
for event, elem in iterparse(filename, (start, end)):
if elem.tag == annoying_animal:
if event is start:
clear = False
else:
yield elem
clear = True
if clear:
elem.clear()

# in application
for subelem in process(filename, "schnappi"):
# do something with subelem

(I've reorganized the code a bit to cut down on the operations.
also note the "is" trick; iterparse returns the event strings you
pass in, so comparing on object identities is safe)

an alternative is to use the lower-level XMLParser class (which
is similar to SAX, but faster), but that will most likely result in
more and tricker Python code...

</F>

[D H]

Reinhold Birkenfeld wrote:

D H wrote:

Reinhold Birkenfeld wrote:

D H wrote:
D H wrote:
>Reinhold Birkenfeld wrote:
>
>
>
>>Well, if I had e.g. a question about Boo, I would of course first ask
>>here because I know the expert writes here.
>>
>>Reinhold
>
>
>Reinhold Birkenfeld also wrote:
>
>>If I had wanted to say "you have opinions? fuck off!", I would have said
>>"you have opinions? fuck off!".
>
>
>Take your own advice asshole.
And what's that about?

I think it means you should fuck off, asshole.

I think you've made that clear.

*plonk*

Reinhold

PS: I really wonder why you get upset when someone except you mentions boo.

You're the only one making any association between this thread about
celementree and boo. So again I'll say, take your own advice and fuck off.

[Fredrik Lundh]

Doug Holton wrote:

You're the only one making any association between this thread about
celementree and boo.

really? judging from the Original-From header in your posts, your internet
provider is sure making the same association...

</F>

[Igor V. Rafienko]

[ Fredrik Lundh ]

[ ... ]

the iterparse/clear approach works best if your XML file has a
record-like structure. if you have toplevel records with lots of
schnappi records in them, iterate over the records and use find
(etc) to locate the subrecords you're interested in: (...)

The problem is that the file looks like this:

<data>
<schnappi>
<color>green</color>
<friends>
<friend>
<id>Lama</id>
<color>white</color>
</friend>
<friend>
<id>mother schnappi</id>
<color>green</color>
</friend>
</friends>
<food>
<id>human</id>
<id>rabbit</id>
</food>
</schappi>
<schnappi>
<!-- something interesting -->
</schnappi>
<!-- 60,000 more schnappis -->
</data>

.... and there is really nothing above <schnappi>. The "something
interesting" part consists of a variety of elements, and calling
findall for each of them although possible, would probably be
unpractical (say, distinguishing <friend>'s colors from <schnappi's>).

Conceptually I need a "XML subtree iterator", rather than an XML
element iterator. <schnappi>-elements are the ones having a complex
internal structure, and I'd like to be able to speak of my XML as a
sequence of Python objects representing <schnappi>s and their internal
structure.

[ ... ]

(I've reorganized the code a bit to cut down on the operations. also
note the "is" trick; iterparse returns the event strings you pass
in, so comparing on object identities is safe)

Neat trick.

Thank you for your input,

ivr
--
"...but it's HDTV -- it's got a better resolution than the real world."
-- Fry, "When aliens attack"

[D H]

Fredrik Lundh wrote:

Doug Holton wrote:

You're the only one making any association between this thread about
celementree and boo.

really? judging from the Original-From header in your posts, your internet
provider is sure making the same association...

You seriously need some help.

[Paul Boddie]

Reinhold Birkenfeld wrote:

D H wrote:

I would recommend emailing the author of a library when you have a
question about that library. You should know that yourself as well.

Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Regardless of anyone's alleged connection with Boo or newsgroup
participation level, the advice to contact the package
author/maintainer is sound. It happens every now and again that people
post questions to comp.lang.python about fairly specific issues or
packages that would be best sent to mailing lists or other resources
devoted to such topics. It's far better to get a high quality opinion
from a small group of people than a lower quality opinion from a larger
group or a delayed response from the maintainer because he/she doesn't
happen to be spending time sifting through flame wars amidst large
volumes of relatively uninteresting/irrelevant messages.

Paul

[Fredrik Lundh]

Igor V. Rafienko wrote:

The problem is that the file looks like this:

<data>

... lots of schnappi records ...

okay. I think your first approach

from cElementTree import iterparse

for event, elem in iterparse("data.xml"):
if elem.tag == "schnappi":
count += 1
elem.clear()

is the right one for this case. with this code, the clear call will
destroy each schnappi record when you're done with it, so you
will release all memory allocated for the schnappi elements.

however, you will end up with a single toplevel element that
contains a large number of empty subelements. this is usually
no problem (it'll use a couple of megabytes), but you can get
rid of the dead schnappis too, if you want to. see the example
that starts with "context = iterparse" on this page

http://effbot.org/zone/element-iterparse.htm

for more information.

</F>

[Reinhold Birkenfeld]

Paul Boddie wrote:

Reinhold Birkenfeld wrote:

D H wrote:

> I would recommend emailing the author of a library when you have a
> question about that library. You should know that yourself as well.
Well, if I had e.g. a question about Boo, I would of course first ask
here because I know the expert writes here.

Regardless of anyone's alleged connection with Boo or newsgroup
participation level

Which was sort of an ironic <wink> from my side. I did not expect "D H"
to go overboard on this.
the advice to contact the package author/maintainer is sound.
Correct. But if the post is already in the newsgroup and the author is known
to write there extensively, it sounds ridiculous to say "contact the author".
It happens every now and again that people
post questions to comp.lang.python about fairly specific issues or
packages that would be best sent to mailing lists or other resources
devoted to such topics. It's far better to get a high quality opinion
from a small group of people than a lower quality opinion from a larger
group or a delayed response from the maintainer because he/she doesn't
happen to be spending time sifting through flame wars amidst large
volumes of relatively uninteresting/irrelevant messages.

Hey, the flame war stopped before it got interesting ;)

Reinhold

[Grant Edwards]

On 2005-09-25, D H <no@spam> wrote:

Igor V. Rafienko wrote:

This gave me the desired behaviour, but:

* It looks *very* ugly
* It's twice as slow as version which sees 'end'-events only.

Now, there *has* to be a better way. What am I missing?

Try emailing the author for support.

I don't think that's needed. He is one of the most active
members of c.l.py, and you should know that yourself.

I would recommend emailing the author of a library when you
have a question about that library. You should know that
yourself as well.

Why??

For the things I "support", I much prefer answering questions
in a public forum. That way the knowledge is available to
everybody, and it reduces the number of e-mailed duplicate
questions. Most of the gurus I know (not that I'm attempting
to placing myself in that category) feel the same way. ESR
explained it well.

Quoting from http://www.catb.org/~esr/faqs/smart-...ons.html#forum

You are likely to be ignored, or written off as a loser, if
you:

[...]

* post a personal email to somebody who is neither an
acquaintance of yours nor personally responsible for
solving your problem

[...]

In general, questions to a well-selected public forum are
more likely to get useful answers than equivalent questions
to a private one. There are multiple reasons for this. One
is simply the size of the pool of potential respondents.
Another is the size of the audience; hackers would rather
answer questions that educate a lot of people than questions
which only serve a few.

--
Grant Edwards grante Yow! I'm a GENIUS! I
at want to dispute sentence
visi.com structure with SUSAN
SONTAG!!

[Fredrik Lundh]

Paul Boddie wrote:

Regardless of anyone's alleged connection with Boo or newsgroup
participation level, the advice to contact the package
author/maintainer is sound. It happens every now and again that people
post questions to comp.lang.python about fairly specific issues or
packages that would be best sent to mailing lists or other resources
devoted to such topics. It's far better to get a high quality opinion
from a small group of people than a lower quality opinion from a larger
group or a delayed response from the maintainer because he/she doesn't
happen to be spending time sifting through flame wars amidst large
volumes of relatively uninteresting/irrelevant messages.

well, for the record, I strongly recommend people to post questions in
public forums. google is far more likely to pick up answers from mailing
list archives and newsgroups than from the "I really should do something
about all the mails in my support folder" part of my brain.

it's often a good idea to spend a little time looking for the right forum
(the xml-sig is a good place for elementtree-related questions), but
posting a question about a widely used Python library to c.l.python is
never wrong.

</F>

[D H]

Grant Edwards wrote:

On 2005-09-25, D H <no@spam> wrote:

Igor V. Rafienko wrote:
>This gave me the desired behaviour, but:
>
>* It looks *very* ugly
>* It's twice as slow as version which sees 'end'-events only.
>
>Now, there *has* to be a better way. What am I missing?

Try emailing the author for support.

I don't think that's needed. He is one of the most active
members of c.l.py, and you should know that yourself.
I would recommend emailing the author of a library when you
have a question about that library. You should know that
yourself as well.

Why??

Please tell me I don't have to explain to you why the author of a 3rd
party library might be able to answer a question specific to that library.

For the things I "support", I much prefer answering questions
in a public forum.

Right, which is exactly why we have sourceforge, tigris, google groups,
and numerous other free resources where you can set up a mailing list to
publicly ask and answer questions about your software.
Of course it may not get you as many paypal hits as spamming larger
forums with your support questions will, but it is the decent thing to do.

[D H]

Fredrik Lundh wrote:

Paul Boddie wrote:

Regardless of anyone's alleged connection with Boo or newsgroup
participation level, the advice to contact the package
author/maintainer is sound. It happens every now and again that people
post questions to comp.lang.python about fairly specific issues or
packages that would be best sent to mailing lists or other resources
devoted to such topics. It's far better to get a high quality opinion
from a small group of people than a lower quality opinion from a larger
group or a delayed response from the maintainer because he/she doesn't
happen to be spending time sifting through flame wars amidst large
volumes of relatively uninteresting/irrelevant messages.

well, for the record, I strongly recommend people to post questions in
public forums. google is far more likely to pick up answers from mailing
list archives and newsgroups than from the "I really should do something
about all the mails in my support folder" part of my brain.

You run your own server and get plenty of paypal donations. Why not run
your own mailing list for support? If not, see sourceforge or google
groups: http://groups.google.com/groups/create?lnk=l&hl=en

[Grant Edwards]

On 2005-09-25, D H <no@spam> wrote:

I would recommend emailing the author of a library when you
have a question about that library. You should know that
yourself as well.
Why??

Please tell me I don't have to explain to you why the author
of a 3rd party library might be able to answer a question
specific to that library.

Of course not. And when that author reads this group, why not
post questions here so that everybody can benefit from the
information?

For the things I "support", I much prefer answering questions
in a public forum.

Right, which is exactly why we have sourceforge, tigris,
google groups,

Exactly how do you think c.l.p on google groups differs from
c.l.p on the rest of Usenet?
and numerous other free resources where you can set up a
mailing list to publicly ask and answer questions about your
software. Of course it may not get you as many paypal hits as
spamming larger forums with your support questions will,
WTF are you on about? What the hell is a "Paypal hit"? How is
posting a single, on-topic, question to c.l.p "spamming"?
but it is the decent thing to do.

You sir, are a loon.

<plonk>

--
Grant Edwards grante Yow! Where do your SOCKS
at go when you lose them in
visi.com th' WASHER?

[Neil Hodgson]

Paul Boddie:

Regardless of anyone's alleged connection with Boo or newsgroup
participation level, the advice to contact the package
author/maintainer is sound. It happens every now and again that people
post questions to comp.lang.python about fairly specific issues or
packages that would be best sent to mailing lists or other resources
devoted to such topics. It's far better to get a high quality opinion
from a small group of people than a lower quality opinion from a larger
group or a delayed response from the maintainer because he/she doesn't
happen to be spending time sifting through flame wars amidst large
volumes of relatively uninteresting/irrelevant messages.

As the author of a widely used component (Scintilla) I feel public
fora should be preferred over private mail since
* The effort in answering is spread over more people.
* The author will only have experience in a narrow range of usage and
the query is likely to match some other user's experience.
* The author may be out of touch or busy.
* The author will make mistakes which can be picked up by other
participants. I'd estimate that 10% of the answers I give are wrong or
useless, sometimes due to misunderstanding the query and sometimes due
to confusion over how the component works.
* Public fora are archived and searchable.

Neil

[D H]

Grant Edwards wrote:

On 2005-09-25, D H <no@spam> wrote:

I would recommend emailing the author of a library when you
have a question about that library. You should know that
yourself as well.

Why??
Please tell me I don't have to explain to you why the author
of a 3rd party library might be able to answer a question
specific to that library.

Of course not. And when that author reads this group, why not
post questions here so that everybody can benefit from the
information?

When did I ever argue against that? I was suggesting a resource to use
to support your own software. I think you have assumed that I suggested
posting here about celementree was off-topic. Tell me where I ever said
that. I said to the first guy that he should ask the author for help,
meaning that he could get help that way as well. Furthermore, I believe
that is the more efficient way to get help, by contacting the author
directly, especially if that author is too lazy to set up their own
support forum or list.

For the things I "support", I much prefer answering questions
in a public forum.

Right, which is exactly why we have sourceforge, tigris,
google groups,

Exactly how do you think c.l.p on google groups differs from
c.l.p on the rest of Usenet?

Who the fuck said that? Are you pulling this shit out of your ass?

and numerous other free resources where you can set up a
mailing list to publicly ask and answer questions about your
software. Of course it may not get you as many paypal hits as
spamming larger forums with your support questions will,

WTF are you on about? What the hell is a "Paypal hit"? How is
posting a single, on-topic, question to c.l.p "spamming"?

Fredrik Lundh gets money via paypal on his site where his software is
located. That's what I meant. Where did I say this particular post is
a spam? Again, your ass? where do you get this shit?

but it is the decent thing to do.

You sir, are a loon.

You're a funny ass.

[François Pinard]

[D H]

I think it means you should fuck off, asshole.

Such language looks very inappropriate to me in a public list, and
builds a bad and long-lasting prejudice against those resorting to it.
Flamers should rather, for their own image, email each other privately
on their matters (yet D.H. is not providing his/her email address in
the last messages I've seen). Flamers being in the strong need of an
audience may easily find special forums for people enjoying flames.

As for the Python list, let's keep it the calm and enjoyable place it
usually is. Resist the temptation of feeding flames, rather ignore or
merely learn to "killfile" the (happily few) abusers we got.

Enough said for now! Keep happy, all of you. :-)

--
François Pinard http://pinard.progiciels-bpi.ca

[Paul Boddie]

Fredrik Lundh wrote:

well, for the record, I strongly recommend people to post questions in
public forums.
I wasn't saying that such advice was wrong, though. I suppose the
quality or responsiveness of advice is going to vary somewhat depending
on a number of factors...
it's often a good idea to spend a little time looking for the right forum
(the xml-sig is a good place for elementtree-related questions), but
posting a question about a widely used Python library to c.l.python is
never wrong.

I think the popularity of the library is probably the biggest factor,
and given that you've released much more popular stuff than I have, I'd
imagine that a lot more people would be able to offer advice about
ElementTree than anything I've written. My software's niche status
leads me to encourage people to contact me directly, but it'd surely be
a great experience to be one of many in a community fielding questions
about it in public.

That said, choosing the right forum can give better results: how many
Zope-specific posts do you see on comp.lang.python these days?

Paul