Need for efficient method for File Parsing and copying

[Tom]

I am trying to update another developers code, and am stuck in a
cludge.

It works like this:

A user uploads a file from the web page. Our code reads from a
NetworkStream to a Filestream, and writes out a temporary file. Then
the file is parsed (The file contains HTML data + the actual binary
file that was transmitted). As part of the parsing, once the main
binary file is to be parsed out, it is written with a new Filestream,
byte by byte to a new location on disk.

This was great when the file was 24Mb, but now it is 70Mb, and our
windows CE device can not hold 2 copies of this large file. Not to
mention that the additional copy takes a while to perform.

To me it seams that there are two methods to solve this, the first
being to read the networkstream realtime until the actual binary file
shows up, then filestream directly. The second method, which to me
seems simpler, is to just take the large temp file, and truncate off
the non-binary data.

By the way, I can not use a memorystream, as I have alocated most of
the device memory to storage, thus there is not enough memory room.

Anyone have any advice??

Tom Kuhn

[Peter Duniho]

On Wed, 27 Aug 2008 10:08:47 -0700, Tom <ku****@gmail.comwrote:

[...]
To me it seams that there are two methods to solve this, the first
being to read the networkstream realtime until the actual binary file
shows up, then filestream directly. The second method, which to me
seems simpler, is to just take the large temp file, and truncate off
the non-binary data.

Well, the second method would definitely be simpler if the file system
allowed you to trunacte a file from the beginning.

But, it doesn't. You'd have to rewrite the entire file in-place, shifting
the binary data down within the file, and then truncate the end (which you
can do).

"Simpler" is in the eye of the beholder, but personally I'd just go with
your first suggestion: process the incoming stream as its read, writing
only the data you need to the file.

If you've already got code that takes the FileStream, parses that and
emits another FileStream, it should be simple enough to change that code
to just take the NetworkStream as input instead. Ideally, you would have
written the code so that it actually just takes a Stream and not a
FileStream. In that case, there's practically nothing to change. Just
pass a different Stream to the code.

But if not, it would just be a matter of changing the FileStream
variable(s) to a Stream variable(s). Again, ideally you would not have
used any of the members specific to FileStream, but if so it should not be
that hard to rework the code so that they aren't (either just don't do
whatever it was you were doing with them, or provide the same information
some other way as needed).

Of course, the simplest approach is to somehow redefine the problem so
that you can just upgrade the storage space to accomodate the larger data
size. :)

Pete

[Ignacio Machin ( .NET/ C# MVP )]

>
A user uploads a file from the web page.

Upload or download?
From the description below it seems that you are getting the file from
a webserver to a Win CE device.

>*Our code reads from a
NetworkStream to a Filestream, and writes out a temporary file. *Then
the file is parsed (The file contains HTML data + the actual binary
file that was transmitted). *As part of the parsing, once the main
binary file is to be parsed out, it is written with a new Filestream,
byte by byte to a new location on disk.

>
This was great when the file was 24Mb, but now it is 70Mb, and our
windows CE device can not hold 2 copies of this large file. *Not to
mention that the additional copy takes a while to perform.

You can delete the old file. If this is not possible cause you need at
least one working copy. then you can try to zip it . download the new
file (it should also be zipped) and only when the new file is
downloaded ok then y ou delete the old one and unzip the new one.

Note that with files that big it might take a LONG time to download &
process all the info. Maybe it's time to change the delivery method of
your app.

Anyone have any advice??

As I said above, maybe it's time to rethink your strategic. 70MB only
ni data is a LOT especially when some devices have only 128MB

[Tom]

Zipping... Now thats an idea that for some reason never crossed my
mind! The file that we are copying is a full Win CE Flash file system
image and is 63Mb unzipped... Well I zipped it and WOW, only 12Mb.
Since we alreeady handle unzipping in the device, it should be no
problem to go this route. Good Idea.

As for what we were doing, the file was transfered via HTTP POST from
a users computer. The networkStream grabbed to POST data and built a
temp file with the file along with the post data in it. I was trying
to real0time analyse the data and find the beginning of the actual
file in a byte by byte process reading from the stream.

I think this zip solution will work out well.

Thanks,
Tom


On Aug 28, 12:18 pm, "Ignacio Machin ( .NET/ C# MVP )"
<[e-mail]:

A user uploads a file from the web page.

Upload or download?
From the description below it seems that you are getting the file from
a webserver to a Win CE device.

Our code reads from a
NetworkStream to a Filestream, and writes out a temporary file. Then
the file is parsed (The file contains HTML data + the actual binary
file that was transmitted). As part of the parsing, once the main
binary file is to be parsed out, it is written with a new Filestream,
byte by byte to a new location on disk.

This was great when the file was 24Mb, but now it is 70Mb, and our
windows CE device can not hold 2 copies of this large file. Not to
mention that the additional copy takes a while to perform.

You can delete the old file. If this is not possible cause you need at
least one working copy. then you can try to zip it . download the new
file (it should also be zipped) and only when the new file is
downloaded ok then y ou delete the old one and unzip the new one.

Note that with files that big it might take a LONG time to download &
process all the info. Maybe it's time to change the delivery method of
your app.

Anyone have any advice??

As I said above, maybe it's time to rethink your strategic. 70MB only
ni data is a LOT especially when some devices have only 128MB

[Cheeso]

On Aug 28, 1:36*pm, Tom <[e-mail]:

>
I think thiszipsolution will work out well.

Thanks,
Tom

How do you do zipping / unzipping on the device?