469,626 Members | 1,167 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,626 developers. It's quick & easy.

append() to each row of a text file

Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

... I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
....

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

Oct 16 '07 #1
3 1793
sylvaticus wrote:
Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??
There is no such thing as "each row" when files are concerned. Each
file is a stream of characters. Only your interpretation of some
specific character (like the "newline" \n for example) makes the file
*appear* to contain "rows" (we usually call them "lines").
>
Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

.. I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??
No. The usual approach is writing a new file where "each line" is
replicated and "expanded" to contain new information.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask
Oct 16 '07 #2
On 2007-10-16 16:46, sylvaticus wrote:
Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

.. I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))
Files of that size is really nothing much for a modern computer, not
even if there are 40,000 of them. In fact, you could probably load them
all into memory without much trouble (though I think that it would be
faster to just open them one by one).
is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??
While such an approach would be possible it would require reading the
whole file into a buffer large enough to hold the final result, and
would involve a lot of copying making it very slow. By operating on one
line at at time you get much better memory usage and a much better design.

--
Erik Wikström
Oct 16 '07 #3
On Oct 16, 6:01 pm, Erik Wikström <Erik-wikst...@telia.comwrote:
On 2007-10-16 16:46, sylvaticus wrote:
Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??
Instead of the classical way:
VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2
.. I would like to use the much more redeable:
year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...
where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

Files of that size is really nothing much for a modern computer, not
even if there are 40,000 of them. In fact, you could probably load them
all into memory without much trouble (though I think that it would be
faster to just open them one by one).
is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

While such an approach would be possible it would require reading the
whole file into a buffer large enough to hold the final result, and
would involve a lot of copying making it very slow. By operating on one
line at at time you get much better memory usage and a much better design.

--
Erik Wikström
Thanks both.. I will try it opening each one at a time, and if it will
be too slow I'll go for the "normal" approach...

Oct 16 '07 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Charles Ranch | last post: by
2 posts views Thread by JMCN | last post: by
1 post views Thread by Mark | last post: by
2 posts views Thread by Gaby Sandoval | last post: by
12 posts views Thread by vj | last post: by
1 post views Thread by Semajthewise | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.