Best way to write a file n-bytes long

[Tony C]

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

thanks

[Sybren Stuvel]

Tony C enlightened us with:

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

Why not create a string that is 1 KB long and write that n*1024 times?

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?

[Raymond Hettinger]

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.
The C coded string methods are not slow.
In Py2.3, 'x' * n calls str.__mul__ which is implemented
using the C library's memset() function.

It is faster still to use itertools.repeat:

st = itertools.repeat('X', 1048576)

Of course, your particular use case is I/O bound
(meaning that string construction isn't the
cause of your performance issues).
st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

Raymond Hettinger

[Daniel Klein]

On 26 Aug 2003 15:04:12 -0700, ca*******@yahoo.com (Tony C) wrote:

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

Pythons file objects are automatically treated as streams. So you could do
something like this:

st = 'X'
somebignumber = 1048576
fh = open('junk','wb')
for n in xrange(somebignumber):
fh.write(st)
fh.close()

Daniel Klein

[Michael Porter]

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

thanks

Another alternative is to seek to required position and write (at least) one
byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.

[Greg Brunet]

"Michael Porter" <mp*****@despammed.com> wrote in message
news:3f*********************@news.dial.pipex.com.. .

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or 1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Another alternative is to seek to required position and write (at least) one byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.

I interpreted his 'arbitrary data' to mean the same thing & came up with
the same solution (which should be the fastest way to do it in C as
well!). Anyway, doing some rough timing also seems to show that
creating the string was only taking about 10% of the time that writing
it is, and that the seek solution is about 10x faster that creating the
string & 100x faster that writing the string! It shows once again that
it pays to find out where the bottleneck is before trying to optimize
the wrong area!

--
Greg

[Michael Peuser]

"Dialtone" <di************************@aruba.it> schrieb im Newsbeitrag
news:87************@vercingetorix.caesar.org...

ca*******@yahoo.com (Tony C) writes:

[...]

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

For little files (less than 2 or 3 Mb I think) your code is the
fastest I can think of. But growing There is a new version which is a
lot faster

All this is valid in a very limited scope only.
- Consider limited RAM (lets say 128 MB) - you will be extremely slowed down
by thrashing.
- Consider on-the-fly compression (Windows NTFS): The 1000 MB testdate will
be reduced to 1 Byte!

Kindly
Michael P