473,808 Members | 2,882 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Huge xml files

Hi Group,

I have an xml file wich contains a rootelement and a subelement with a
filename and a subelement with bindata.
In each xml file there is only one entry for each element. Like this:

<root>
<filename>somef ile.exe</filename>
<bindata>bindat a in base64 here</bindata>
</root>

I have to parse the content in the bindata element to a Websphere MQ
queue, so I read the xml file into memory with xmlserializer and
filestream and put it in the queue. This works pretty fine.

On the other side of the queue I get the queuedata and collects the
data as an object. This works very well also.

But now and then there are xml files with huge amount of bindata. When
they exceed about 350mb I get an out of memory error. So I don't know
how to handle this. My idea is to somehow break the bindata into
smaller peaces. But I do not know how to partially load the data of
one element (bindata). Is that possible and how?
With this problem solved - is there some way to tell the message queue
that this is only a portion of an element?

- rick -
Jan 28 '08 #1
1 2061
Hi,
I think this is more a question for a Websphere MQ group, it depend of how
they can manage huge amount of data.
You can always read part of the file (maybe you will have to treat it as a
text file instead of a XML).

--
Ignacio Machin
http://www.laceupsolutions.com
Mobile & warehouse Solutions.
"rickbear" <ri********@gma il.comwrote in message
news:53******** *************** ***********@s12 g2000prg.google groups.com...
Hi Group,

I have an xml file wich contains a rootelement and a subelement with a
filename and a subelement with bindata.
In each xml file there is only one entry for each element. Like this:

<root>
<filename>somef ile.exe</filename>
<bindata>bindat a in base64 here</bindata>
</root>

I have to parse the content in the bindata element to a Websphere MQ
queue, so I read the xml file into memory with xmlserializer and
filestream and put it in the queue. This works pretty fine.

On the other side of the queue I get the queuedata and collects the
data as an object. This works very well also.

But now and then there are xml files with huge amount of bindata. When
they exceed about 350mb I get an out of memory error. So I don't know
how to handle this. My idea is to somehow break the bindata into
smaller peaces. But I do not know how to partially load the data of
one element (bindata). Is that possible and how?
With this problem solved - is there some way to tell the message queue
that this is only a portion of an element?

- rick -

Jan 28 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
1579
by: Anders Søndergaard | last post by:
Hi, I'm trying to process a large filesystem (+20 million files) and keep the directories along with summarized information about the files (sizes, modification times, newest file and the like) in an instance hierarchy in memory. I read the information from a Berkeley Database. I'm keeping it in a Left-Child-Right-Sibling instance structure, that I operate on recursively.
0
1236
by: hakhan | last post by:
Hello, I need to store huge(+/- 100MB) data. Furthermore, my GUI application must select data portions from these huge data files in order to do some post-processing. I wonder in which format I should put my data in? XML or just a (relational) database? Or should I use an XML database (native or xml-enabled?)??? I am a little bit confused .... If I'd put the data in XML files, then loading the entire XML tree in memory(DOM) would...
5
3335
by: Laiverd.COM | last post by:
Currently working on a site that requires multiple different styles for all kind of things (tables, link colors etc.) and am suddenly ;-) wondering what would be the wise approach: have all these different styles in one huge stylesheet, or separate into smaller css files and include only those that are necessary. Target audience alos includes dialup visitors. Would anyone be willing to share his/her opinion on this? Looking for the pro's...
10
1815
by: Mario Rodriguez | last post by:
HI people, there is an alternative way to upload files than the ASP.NET native way? Is OK for small files but It doesn't seems to be convenient for an application where there is a lot of paralell uploadings of huge files (50MB-100MB) Any idea? open source components ? thanks
7
4541
by: Joe | last post by:
I have an upload file operation in the web application. UploadForm.asp is the form, and UploadAction.asp is the form processing. //UploadForm.asp <FORM NAME="InputForm" ACTION="UploadAction.asp" METHOD="POST" enctype=multipart/form-data> <input type="file" name="fileName"> //etc ... </FORM>
2
7213
by: Zam | last post by:
Hello World, Under Windows 2003 Server. IIS6. The following code working fine for small files, and for files with size about few megabytes. If I am trying to send HUGE file -- about 700 Megabyte -- w3wp.exe eat all memory (I have 1024 megabytes) and all CPU time without positive result.
12
5367
by: Jeff Calico | last post by:
I have 2 XML data files that I want to extract data from simultaneously and transform with XSLT to generate a report. The first file is huge and when XSLT builds the DOM tree in memory, it runs out of space. I only need a few branches of elements from the original XML, so I am seeking a recomended way of building a DOM for XSLT of only the elements that I need. I'm writing a Java application that invokes Xalan, and reading up on SAX...
2
4832
by: MichiMichi | last post by:
My asp.net application has a directory which is filled up with over 17 files (email) a second. After a while this sums up into a huge amount of files. I usually count files with the GETFILES method. This works perfectly of there are only a few hundert files in the folder ********* ..... Dim files() As String
0
1368
by: ranganadh | last post by:
Dear Group members, I am new to LINQ, pls help on the deeling with huge amount of data with the C# stand Alone application. I have two file, which contains more then 2 lacs lines in every file suppose file1 like ...
9
13240
by: NvrBst | last post by:
Whats the best way to count the lines? I'm using the following code at the moment: public long GetNumberOfLines(string fileName) { int buffSize = 65536; int streamSize = 65536; long numOfLines = 0; byte bArr = new byte;
0
9721
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9600
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10631
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10374
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10374
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10114
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
6880
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
4331
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3859
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.