473,560 Members | 3,037 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Iterparse and ElementTree confusion

Hi

Im trying to parse a large(150MB) xml file in order to extract specific
required records.

import sys
from elementtree.Ele mentTree import ElementTree

root = ElementTree(fil e=big.xml')

This works fine for smaller versions of the same xml file but...when i
attempted the above my PC goes to lala land, theres much HDD grinding
followed by "windows runnign low on virtual memory" popup after
10-15mins. Then just more grinding...for an hour before i gave up

XML file format:
<root>
<rubbish1>
Aug 17 '05 #1
4 4646
Further to the above....

I pass my found element 'Product' to ElementTree's handy print all
function

def print_all(eleme nt):
root = element
#Create an iterator
iter = root.getiterato r()
#Iterate
for element in iter:
#First the element tag name
print "Element:", element.tag
#Next the attributes (available on the instance itself using
#the Python dictionary protocol
if element.keys():
print "\tAttribut es:"
for name, value in element.items() :
print "\t\tName: '%s', Value: '%s'"%(name, value)
#Next the child elements and text
print "\tChildren :"
#Text that precedes all child elements (may be None)
if element.text:
text = element.text
text = len(text) > 40 and text[:40] + "..." or text
print "\t\tText:" , repr(text)
if element.getchil dren():
#Can also use: "for child in element.getchil dren():"
for child in element:
#Child element tag name
print "\t\tElemen t", child.tag
#The "tail" on each child element consists of the text
#that comes after it in the parent element content, but
#before its next sibling.
if child.tail:
text = child.tail
text = len(text) > 40 and text[:40] + "..." or text
print "\t\tText:" , repr(text)

if the element i pass to the above is from

root = ElementTree(fil e='somefile.xml ')
iter = root.getiterato r()
#Iterate
for element in iter:
if element.tag == 'Product':
print_all(eleme nt)

I print all of my required element's attributes, children, children's
children, etc

However if i pass the element i get from iterparse

for event, elem in iterparse(filen ame):
if elem.tag == "Products":
print_all(elem)
else:
elem.clear()

i only get the attributes for <product> and a list of its children, no
further iteration into the children and the children's children

what am i missing?

Aug 17 '05 #2
pa***********@g mail.com wrote:
def parse_for_produ cts(filename):

for event, elem in iterparse(filen ame):
if elem.tag == "Products":
root = ElementTree(ele m)
print_all(root)
else:
elem.clear()

My problem is that if i pass the 'elem' found by iterparse then try to
print all attributes, children and tail text i only get
elem.tag....ele m.keys returns nothing as do all of the other previously
useful elementtree methods.

Am i right in thinking that you can pass an element into ElementTree?
How might i manually iterate through <product>...</product> grabbing
everything?


by default, iterparse only returns "end" events, which means that the
iterator will visit the Products children before you see the Products
element itself. with the code above, this means that the children will
be nuked before you get around to process the parent.

depending on how much rubbish you have in the file, you can do

for event, elem in iterparse(filen ame):
if elem.tag == "Products":
process(elem)
elem.clear()

or

for event, elem in iterparse(filen ame):
if elem.tag == "Products":
process(elem)
elem.clear()
elif elem.tag in ("Rubbish1", "Rubbish2") :
elem.clear()

or

inside = False
for event, elem in iterparse(filen ame, events=("start" , "end")):
if event == "start":
# we've seen the start tag for this element, but not
# necessarily the end tag
if elem.tag == "Products":
inside = True
else:
# we've seen the end tag
if elem.tag == "Products":
process(elem)
elem.clear()
inside = False
elif not inside:
elem.clear()

for more info, see

http://effbot.org/zone/element-iterparse.htm

</F>

Aug 17 '05 #3
"when i attempted [to load 150MB xml file] my PC goes to lala land,
theres much HDD grinding followed by "windows runnign low on virtual
memory" popup after 10-15mins. Then just more grinding...for an hour
before i gave up"

I have had great success using SAX to parse large XML files. If you use
care your memory use will peak at a low number no matter how much XML
you chew through.

Aug 17 '05 #4
Ofcourse!

Thankyou so much. All is well, using the Iterparse takes a mere 3mins
to find the last product in the xml and this is probably due to my
checking 10000 products for a specific.

<looks at shoes>I feel rather honoured to have 'the' effbot reply to my
humble post....</looks at shoes>

Thanks again

Paul

Aug 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
3254
by: Stewart Midwinter | last post by:
I want to parse a file with ElementTree. My file has the following format: <!-- file population.xml --> <?xml version='1.0' encoding='utf-8'?> <population> <person><name="joe" sex="male" age="49"></person> <person><name="hilda" sex="female" age="33"></person> <person><name="bartholomew" sex="male" age="17"> </person> </population>
1
3077
by: Greg Wilson | last post by:
I'm trying to convert from minidom to ElementTree for handling XML, and am having trouble with entities in DTDs. My Python script looks like this: ---------------------------------------------------------------------- #!/usr/bin/env python import sys, os from elementtree import ElementTree
1
3046
by: mirandacascade | last post by:
O/S: Windows 2K Vsn of Python: 2.4 Currently: 1) Folder structure: \workarea\ <- ElementTree files reside here \xml\ \dom\
15
5404
by: Steven Bethard | last post by:
I'm having trouble using elementtree with an XML file that has some gbk-encoded text. (I can't read Chinese, so I'm taking their word for it that it's gbk-encoded.) I always have trouble with encodings, so I'm sure I'm just screwing something simple up. Can anyone help me? Here's the interactive session. Sorry it's a little verbose, but...
0
1758
by: Greg Aumann | last post by:
I am trying to write some python code for a library that reads an XML-like language from a file into elementtree data structures. Then I want to be able to read and/or modify the structure and then be able to write it out either as XML or in the original format. I really want the api for the XML-like language to be the same as the elementtree...
2
1956
by: mirandacascade | last post by:
Situation is this: 1) I have inherited some python code that accepts a string object, the contents of which is an XML document, and produces a data structure that represents some of the content of the XML document 2) The inherited code is somewhat 'brittle' in that some well-formed XML documents are not correctly processed by the code; the...
2
3761
by: Stuart McGraw | last post by:
I have a broad (~200K nodes) but shallow xml file I want to parse with Elementtree. There are too many nodes to read into memory simultaneously so I use iterparse() to process each node sequentially. Now I find i need to get and save the input file line number of each node. Googling turned up a way to do it by subclassing...
5
9977
by: saif.shakeel | last post by:
#!/usr/bin/env python from elementtree import ElementTree as Element tree = et.parse("testxml.xml") for t in tree.getiterator("SERVICEPARAMETER"): if t.get("Semantics") == "localId": t.set("Semantics", "dataPackageID")
13
5043
by: George Sakkis | last post by:
It seems xml.etree.cElementTree.iterparse() is not unicode aware: .... print elem.text .... Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 64, in __iter__ UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-15: ordinal not in range(128)
0
7632
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7552
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7841
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8066
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
6191
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5175
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3585
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1169
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
877
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.