469,271 Members | 832 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,271 developers. It's quick & easy.

Best way to write a file n-bytes long

Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

thanks
Jul 18 '05 #1
6 16187
Tony C enlightened us with:
Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.


Why not create a string that is 1 KB long and write that n*1024 times?

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Jul 18 '05 #2

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...
Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.
The C coded string methods are not slow.
In Py2.3, 'x' * n calls str.__mul__ which is implemented
using the C library's memset() function.

It is faster still to use itertools.repeat:

st = itertools.repeat('X', 1048576)

Of course, your particular use case is I/O bound
(meaning that string construction isn't the
cause of your performance issues).
st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

Raymond Hettinger
Jul 18 '05 #3
On 26 Aug 2003 15:04:12 -0700, ca*******@yahoo.com (Tony C) wrote:
Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()


Pythons file objects are automatically treated as streams. So you could do
something like this:

st = 'X'
somebignumber = 1048576
fh = open('junk','wb')
for n in xrange(somebignumber):
fh.write(st)
fh.close()

Daniel Klein

Jul 18 '05 #4

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...
Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or
1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.

Python's write only allows a string to be passed in.

Sure, I could create a string that is n Megabytes long, and pass it to
write, but it seems as though there should be a better way ???
String manipulation is typically considered slow.

st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()

thanks


Another alternative is to seek to required position and write (at least) one
byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.

Jul 18 '05 #5
"Michael Porter" <mp*****@despammed.com> wrote in message
news:3f*********************@news.dial.pipex.com.. .

"Tony C" <ca*******@yahoo.com> wrote in message
news:8d**************************@posting.google.c om...
Does Python have a function that is analogous to C's write() or
fwrite()-

that is , I want to write a file (of arbitrary data) that is 100K, or 1MB (or more) bytes long..

Both write() and fwrite() in C allow the user to specify the size of
the data to be written.
Another alternative is to seek to required position and write (at least) one byte...

reqSize = 1048576
fh = open('junk', 'wb')
fh.seek(reqSize - 1)
fh.write('\0')
fh.close()

Mike.


I interpreted his 'arbitrary data' to mean the same thing & came up with
the same solution (which should be the fastest way to do it in C as
well!). Anyway, doing some rough timing also seems to show that
creating the string was only taking about 10% of the time that writing
it is, and that the seek solution is about 10x faster that creating the
string & 100x faster that writing the string! It shows once again that
it pays to find out where the bottleneck is before trying to optimize
the wrong area!

--
Greg

Jul 18 '05 #6

"Dialtone" <di************************@aruba.it> schrieb im Newsbeitrag
news:87************@vercingetorix.caesar.org...
ca*******@yahoo.com (Tony C) writes:

[...]
st="X" * 1048576
fh=open("junk","wb")
fh.write(st)
fh.close()


For little files (less than 2 or 3 Mb I think) your code is the
fastest I can think of. But growing There is a new version which is a
lot faster


All this is valid in a very limited scope only.
- Consider limited RAM (lets say 128 MB) - you will be extremely slowed down
by thrashing.
- Consider on-the-fly compression (Windows NTFS): The 1000 MB testdate will
be reduced to 1 Byte!

Kindly
Michael P

Jul 18 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Herve MAILLARD | last post: by
7 posts views Thread by Jimbo | last post: by
5 posts views Thread by Andrew S. Giles | last post: by
12 posts views Thread by Nettan | last post: by
3 posts views Thread by gordon | last post: by
3 posts views Thread by Nemisis | last post: by
7 posts views Thread by Gladen Blackshield | last post: by
9 posts views Thread by Brian Cryer | last post: by
10 posts views Thread by Brendan Miller | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by suresh191 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.