473,320 Members | 1,910 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

2gb Limit

My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.

Comments?

Thanks,
Dec 1 '06 #1
4 1925

Andrew Robinson wrote:
My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.
You should probably ask this question in the group

microsoft.public.dotnet.xml

or, better yet, search the archives there. That's where the XML gurus
hang out....

Dec 1 '06 #2
On Fri, 1 Dec 2006 14:52:54 -0800, "Andrew Robinson" <ne****@nospam.nospam>
wrote:
>My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.

Comments?

Thanks,
You can do it as long as you don't try to load all of it at once. You're on
target with your last sentence. If the data is in lines of text it's a piece of
cake. If not exchange the word lines with chunks. Read a line, convert, save,
get next line and do it again until you reach the end. It's the way we did it
in the early times when all we had was 16 or 32k of memory.
Good luck with your project,

Otis Mukinfus
http://www.arltex.com
http://www.tomchilders.com
Dec 2 '06 #3
One of the purposes of XML is transferring or stream the small amount of
data over the wire. I do not think the intention of XML is to handle
something like 1GB or 2GB of data. Use SQL server to handle such a volume
instead.

chanmm

"Andrew Robinson" <ne****@nospam.nospamwrote in message
news:OE**************@TK2MSFTNGP04.phx.gbl...
My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it
can be done even with XP or Vista 64. Am I correct? I assume that I can
read from a file that is larger than 2GB with a stream but I will need to
read, convert and write using much smaller chuncks of data.

Comments?

Thanks,
Dec 3 '06 #4

chanmm wrote:
One of the purposes of XML is transferring or stream the small amount of
data over the wire.
As you said: _one_ of the purposes of XML... not the _only_ purpose of
XML.
I do not think the intention of XML is to handle something like 1GB or 2GB of data.
In my experience, this is absolutely not true. XML is used for many
things, and one of them may be to transmit a large amount of data from
place to place. For example, it's perfectly reasonable to send a
product catalog from one company to another as an XML document, and
that could run into multi-gigabytes, especially if there are base-64
encoded images in the document.

Where I do agree with you is that XML is not a good format for storing
and maintaining large amounts of data. A database is much better suited
to that. However, there are people who disagree with me even on that
point.

Dec 4 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Afkamm | last post by:
Hi, :) The preg_replace function... preg_replace(pattern, replacement, subject ) How on earth do you get the limit value to work with arrays? In my code both the pattern and replacement...
3
by: CajunCoiler \(http://www.cajuncoiler.tk\) | last post by:
I've seen no reference to this in the MSDN library, so I ask here... what is the known upper limit for the RichTextbox control? And when this limit is exceeded, what error is generated?
3
by: Jay K | last post by:
Hi, I have multiple queries like this: SELECT col1, col2, col3, col4 FROM table1, table2 where table1.col1 = table2.col1 and table1.col2 = 1 ORDER BY col3 desc LIMIT 5 and
2
by: Urban | last post by:
hi, I have a question. Maybe You know the equivalent to command LIMIT from MySQL I couldn`t find something like this in MS SQL PS I try to display 10 records begining form e.g. 4 sort by id...
4
by: emily_g107 | last post by:
Hi, I need to limit results in the following query type: ...
0
by: D. Dante Lorenso | last post by:
I need to know that original number of rows that WOULD have been returned by a SELECT statement if the LIMIT / OFFSET where not present in the statement. Is there a way to get this data from PG ?...
2
by: elein | last post by:
Yes, I vacuumed. Reproduced on both 7.3.2 and 7.5. Brain dead java beans want order by clauses in views that they use. my view is: select .... from bigtable b left join lookuptable l order...
4
by: Bill | last post by:
Hi, I would be grateful if someone could clarify my rather confused ideas of the 10 connection limit on XP/2000 when its being used as a server. (I realise that XP is really a client op sys with...
1
by: lawrence k | last post by:
Want to replace the limit clause in a query, but can't get it right. What's wrong with this: $pattern = "(.*)limit (.*)"; $replacement = '$1'; $replacement .= "LIMIT $limit"; $replacement .=...
3
by: sadanjan | last post by:
Hi , Appreciate if someone can clarify if database Share Memory Limit (2 GB ) in Unix 32 bit boxes is the top limit for all the databases put together in a database or is it for each of the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.