473,320 Members | 2,054 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

streaming large strings

I'm working on an application that needs to move very large strings
around (XML files, several megabytes in length). I can't do anything
about the fact they're available to my program as Strings. It's just
the way the API I'm using works.

There are a couple of things I need to do with those strings: pass them
to a SAXParser (which expects an InputStream), and write them to files.
Problem is, no matter what I to do, the file ends up truncated (pretty
much always in the same location, toward the end of the file).

I've tried lots of things, including passing the string to a
StringReader and buffering it. I've also tried things like:

byte[] bytes = xmlDoc.getBytes();
InputStream bytestream = new ByteArrayInputStream(bytes);

and then writing it out to a Buffered writer with:

while ((nextChar = xmlDoc.read()) != -1)

Pretty much everything I do results in a truncated file.

Can anyone suggest a good general approach to dealing with large strings
(that need to turn into streams for writing or other reasons)?

Thanks,

Steve

--
Stephen Ramsay
Jul 17 '05 #1
3 4830
Stephen Ramsay wrote:
I'm working on an application that needs to move very large strings
around (XML files, several megabytes in length). I can't do anything
about the fact they're available to my program as Strings. It's just
the way the API I'm using works.

There are a couple of things I need to do with those strings: pass them
to a SAXParser (which expects an InputStream), and write them to files.
Problem is, no matter what I to do, the file ends up truncated (pretty
much always in the same location, toward the end of the file).
Are you losing the last line of the file? Perhaps you're algorithm
doesn't handle the final line if it doesn't end with a line separator.

I've tried lots of things, including passing the string to a
StringReader and buffering it.
That sounds like a good strategy.
I've also tried things like:

byte[] bytes = xmlDoc.getBytes();
InputStream bytestream = new ByteArrayInputStream(bytes);
I'd avoid fooling around with bytes if at all possible.

and then writing it out to a Buffered writer with:

while ((nextChar = xmlDoc.read()) != -1)

Pretty much everything I do results in a truncated file.
Are you sure the string has the entire contents?

Have you tried using the Writer.write(String) method?

Are you sure you are closing the file stream properly (use a finally
block and call flush() for extra paranoia points).

Can anyone suggest a good general approach to dealing with large strings
(that need to turn into streams for writing or other reasons)?

Thanks,

Steve

--
XML is the programmer's duct tape.
Jul 17 '05 #2
On 2005-02-07, Raymond DeCampo <no****@twcny.rr.com> wrote:
Are you sure you are closing the file stream properly (use a finally
block and call flush() for extra paranoia points).


That was it. Kind of a freshman error, I suppose, but I guess I've
never really understood streams properly. Thank you so much for this
fantastic list of things to check.

Am I right in thinking that the proper (an sufficiently paranoid) way to
close a stream is with something like:

} finally {
try {
buffer.close();
buffer.flush();
} catch (IOException ioe) {
...
}
}

Thanks again.

Steve

--
Stephen Ramsay
web: http://cantor.english.uga.edu/
Jul 17 '05 #3
Stephen Ramsay wrote:
On 2005-02-07, Raymond DeCampo <no****@twcny.rr.com> wrote:
Are you sure you are closing the file stream properly (use a finally
block and call flush() for extra paranoia points).

That was it. Kind of a freshman error, I suppose, but I guess I've
never really understood streams properly. Thank you so much for this
fantastic list of things to check.

Am I right in thinking that the proper (an sufficiently paranoid) way to
close a stream is with something like:

} finally {
try {
buffer.close();
buffer.flush();
} catch (IOException ioe) {
...
}
}


You'll want to call flush() before calling close().

Ray

--
XML is the programmer's duct tape.
Jul 17 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Erick | last post by:
Hello, I've been looking for a while for an answer, but so far I haven't been able to turn anything up yet. Basically, what I'd like to do is to use re.finditer to search a large file (or a file...
3
by: A.M-SG | last post by:
Hi, I have a ASP.NET aspx file that needs to pass large images from a network storage to client browser. The requirement is that users cannot have access to the network share. The aspx file...
1
by: Vanga Sasidhar | last post by:
Hi All, I am developing a program in Visual Basic .NET 2002. There are some videos in AVI format. For the protection of these AVI files i encrypted my files with the use of DES algorithm in the...
6
by: | last post by:
Hi all, is there a better way to stream binary data stored in a table in sql 2005 to a browser in .net 2.0? Or is the code same as in .net 1.1? We noticed that in certain heavy load scenarios,...
5
by: pmakoi | last post by:
dear all this might be a piece of cake for some of you out there but it is causing me a lot of stress given the fact that there is not enogh documentation out there regarding this topic I am...
4
by: =?Utf-8?B?TWljaGFlbA==?= | last post by:
Hello, I need some help. I am trying to write my own web server for serving up large files on my own network. I am using FileStream, BinaryReader, TcpClient, and Sockets. I am trying to break...
1
by: michaelpyles | last post by:
IIS6 (x86) - Classic ASP application - I am downloading files via calls to BinaryWrite from a C++ COM component. The ASP DLL doesn't seem to do a good job limiting memory usage based on the...
1
by: Faisal Shafiq | last post by:
I want to upload a file direct to the Silverlight Streaming Service from a Web Client such as silverlight application. As per our product requirement we want to upload a .WMV file directly from...
11
by: Alexander Adam | last post by:
Hi there! I've got a few (more generic) questions for those C++ experts out there. 1. I am trying to store any kind of data (custom PODs, numbers, strings etc.) within some kind of stream...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.