append() to each row of a text file

[sylvaticus]

Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

... I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
....

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

[Victor Bazarov]

sylvaticus wrote:

Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

There is no such thing as "each row" when files are concerned. Each
file is a stream of characters. Only your interpretation of some
specific character (like the "newline" \n for example) makes the file
*appear* to contain "rows" (we usually call them "lines").

>
Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

.. I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

No. The usual approach is writing a new file where "each line" is
replicated and "expanded" to contain new information.

V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask

[=?UTF-8?B?RXJpayBXaWtzdHLDtm0=?=]

On 2007-10-16 16:46, sylvaticus wrote:

Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

.. I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

Files of that size is really nothing much for a modern computer, not
even if there are 40,000 of them. In fact, you could probably load them
all into memory without much trouble (though I think that it would be
faster to just open them one by one).

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

While such an approach would be possible it would require reading the
whole file into a buffer large enough to hold the final result, and
would involve a lot of copying making it very slow. By operating on one
line at at time you get much better memory usage and a much better design.

--
Erik Wikström

[sylvaticus]

On Oct 16, 6:01 pm, Erik Wikström <[e-mail]:

On 2007-10-16 16:46, sylvaticus wrote:

Hello,
is it possible to append some data on a text output file appending
the data to each row, and without use too much computational I/O ??

Instead of the classical way:

VARX VARY VARZ...
year1 x1 y1 z1
year2 x2 y2 z2

.. I would like to use the much more redeable:

year1 year2 ..
VARX x1 x2 ...
VARY y1 y2 ...
VARZ z1 z2 ...
...

where the 1,2 series are wrote at different times.. a sort of a
appendByRow() function.. and I would avoid of loading the whole file
in memory and then printing it line-by-line, as I have to do it on
approx 40,000 (few KB) files.. and I am worry that the second approach
would take years :-)))

Files of that size is really nothing much for a modern computer, not
even if there are 40,000 of them. In fact, you could probably load them
all into memory without much trouble (though I think that it would be
faster to just open them one by one).

is it possible to do it in a efficient way (e.g. seeking to the right
place, delete the newline and appending there) ??

While such an approach would be possible it would require reading the
whole file into a buffer large enough to hold the final result, and
would involve a lot of copying making it very slow. By operating on one
line at at time you get much better memory usage and a much better design.

--
Erik Wikström

Thanks both.. I will try it opening each one at a time, and if it will
be too slow I'll go for the "normal" approach...