469,344 Members | 6,496 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,344 developers. It's quick & easy.

large string manipulation

morning all,
can someone help me find the most efficient way of manipulating a large
file.

i need to replace special characters in a large file and multiple
string.replace functions are causing memory errors.

doug
Jun 7 '06 #1
3 1514
"douglas wittner" <dw******@teaminteractive.com> wrote:
can someone help me find the most efficient way of manipulating a large
file.


An efficient way for modifications that will change the length of the
file (i.e. possibly insert or delete in the middle) is to stream from
one file to another, iteratively using a buffer in memory, and working
on the buffer before streaming it out to the second file. It requires
careful buffer management to be sure that the boundaries between buffers
don't affect the outcome. For example, if searching for "foo", be
careful that one buffer doesn't end with "f" and the next one start with
"oo".

For modifications which don't change the length of the file, iteratively
reading into a smallish buffer, searching, and then using
FileStream.Seek & Write to perform updates will work reasonably well -
but not as well as a memory-mapped file, but that would be more awkward
in C#, since there is no memory-mapped file primitive.

-- Barry

--
http://barrkel.blogspot.com/
Jun 7 '06 #2
i have the reading down...while reading into a buffer, how do i perrform
the search for a character?

thanks for your help

"Barry Kelly" <ba***********@gmail.com> wrote in message
news:tb********************************@4ax.com...
"douglas wittner" <dw******@teaminteractive.com> wrote:
can someone help me find the most efficient way of manipulating a large
file.


An efficient way for modifications that will change the length of the
file (i.e. possibly insert or delete in the middle) is to stream from
one file to another, iteratively using a buffer in memory, and working
on the buffer before streaming it out to the second file. It requires
careful buffer management to be sure that the boundaries between buffers
don't affect the outcome. For example, if searching for "foo", be
careful that one buffer doesn't end with "f" and the next one start with
"oo".

For modifications which don't change the length of the file, iteratively
reading into a smallish buffer, searching, and then using
FileStream.Seek & Write to perform updates will work reasonably well -
but not as well as a memory-mapped file, but that would be more awkward
in C#, since there is no memory-mapped file primitive.

-- Barry

--
http://barrkel.blogspot.com/

Jun 7 '06 #3
"douglas wittner" <dw******@teaminteractive.com> wrote:
i have the reading down...while reading into a buffer, how do i perrform
the search for a character?


Either load it up into a string or character array (with an Encoding
class - and watch for split lead and trail bytes for UTF8 encoding, for
example) or scan through it with a loop, or write a more sophisticated
search such as Boyer-Moore etc.

-- Barry

--
http://barrkel.blogspot.com/
Jun 8 '06 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Roland Hall | last post: by
29 posts views Thread by zoro | last post: by
4 posts views Thread by moondaddy | last post: by
4 posts views Thread by WaterWalk | last post: by
10 posts views Thread by Peter Duniho | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.