Reading 5MB xml with python

[realin]

hiya all,

However, the title says it all but still i want to explain it further. I am trying to fetch a xml file which is lying at different size. The file size is like >5MB. So as soon as i write my code, the browser keeps on waiting for the response and in the end it freezes or displays time out.
What can be the work around for this ?
I am using google's app engine, so not exactly core python. And i would like to mention that i started learning python just half an hour back, i am a programmer of php, ruby on rails and little bit of java .. so its app engine is more like servlet has get/post methods..

Please help me with it, is it possible using threads ? spliting the file or what ?

Thanks & Regards
Realin !

[Laharl]

Which XML API are you using, DOM or SAX? For large files (on the order of what you're dealing with), I believe SAX is the better choice.

You could use a thread to read through the file in the background, but it's going to be reasonably slow no matter what...

[realin]

Which XML API are you using, DOM or SAX? For large files (on the order of what you're dealing with), I believe SAX is the better choice.

You could use a thread to read through the file in the background, but it's going to be reasonably slow no matter what...

hiya laharl,

thanks for the quick reply.. I am using DOM, i will give a try to SAX. Time is no constraint, so what do you suggest now ?

I am not a die hard fan of threads though, i want my code to be much more simpler, if you think SAX can do it without having to use thread, then i would be more than happy :)

waiting for ur reply
thanks :)

[realin]

hiya all,

I m unable to get a code to fetch XML from a URL link using SAX.
Can anyone please help me in that .. its really urgent

please

[Laharl]

I've never actually used SAX (only xml.dom.minidom), but were I to guess, you'd use something from url or some other networking library to get the XML file from the server and then SAX to process it.

[realin]

I've never actually used SAX (only xml.dom.minidom), but were I to guess, you'd use something from url or some other networking library to get the XML file from the server and then SAX to process it.

hiya

sorry for late update, was outta town actually. I had to use urllib for doing the same, but google app engine wont allow me to download some bulk size files, so i had to drop the idea of importing data thru XML.

However, i found a nice utility called bulkuploader using CSV files. I converted XMLs into CSV using php and then using bulkuploader utility, i just pushed the data in Datastore.

thanks for your help, its always good to see you guys coming up and helping :)

cheer!!