Hi,
another question: What's the most efficient way of copying data between
two file-like objects?
f1.write(f2.read()) doesn't seem to me as efficient as it might be, as a
string containing all the contents of f2 will be created and thrown away.
In the case of two StringIO objects, this means there's a point when the
contents is held in memory three times.
Reading and writing a series of short blocks to avoid a large copy buffer
seems ugly to me, and string objects will be created and thrown away all
the time. Do I have to live with that?
(In C, I would do the same thing, only without having to create and throw
away anything while overwriting a copy buffer, and being used to doing
everything the pedestrian way, anyway.)
--
Thomas 5 9586
Thomas Lotze wrote: another question: What's the most efficient way of copying data between two file-like objects?
f1.write(f2.read()) doesn't seem to me as efficient as it might be, as a string containing all the contents of f2 will be created and thrown away.
You could try f1.writelines(f2).
Steve
Thomas Lotze wrote: f1.write(f2.read()) doesn't seem to me as efficient as it might be, as a string containing all the contents of f2 will be created and thrown away.
if f2 isn't too large, reading lots of data in one operation is often the most
efficient way (trust me, the memory system is a lot faster than your disk)
if you don't know how large f2 can be, use shutil.copyfileobj: help(shutil.copyfileobj)
Help on function copyfileobj in module shutil:
copyfileobj(fsrc, fdst, length=16384)
copy data from file-like object fsrc to file-like object fdst
In the case of two StringIO objects, this means there's a point when the contents is held in memory three times.
to copy stringio objects, you can use f1 = StringIO(f2.getvalue()). why you
would want/need to do this is more than I can figure out, though...
</F>
Fredrik Lundh wrote: if f2 isn't too large, reading lots of data in one operation is often the most efficient way (trust me, the memory system is a lot faster than your disk)
Sure.
if you don't know how large f2 can be, use shutil.copyfileobj:
>>> help(shutil.copyfileobj) Help on function copyfileobj in module shutil:
copyfileobj(fsrc, fdst, length=16384) copy data from file-like object fsrc to file-like object fdst
This sounds like what I was looking for. Thanks for the pointer.
However, the following doesn't seem like anything is being copied: from StringIO import StringIO from shutil import copyfileobj s = StringIO() s2 = StringIO() s.write('asdf') copyfileobj(s, s2) s2.getvalue()
''
to copy stringio objects, you can use f1 = StringIO(f2.getvalue()).
But this should have the same problem as using read(): a string will be
created on the way which contains all the content.
why you would want/need to do this is more than I can figure out, though...
Because I want to manipulate a copy of the data and be able to compare it
to the original afterwards.
Another thing I'd like to do is copy parts of a StringIO object's content
to another object. This doesn't seem possible with any shutil method. Any
idea on that?
What one can really wonder, I admit, is why the difference between holding
data two or three times in memory matters that much, especially if the
latter is only for a short time. But as I'm going to use the code that
handles the long string as a core component to some application, I'd like
to make it behave as well as possible.
--
Thomas
Thomas Lotze wrote: if you don't know how large f2 can be, use shutil.copyfileobj:
>>> help(shutil.copyfileobj) Help on function copyfileobj in module shutil:
copyfileobj(fsrc, fdst, length=16384) copy data from file-like object fsrc to file-like object fdst
This sounds like what I was looking for. Thanks for the pointer. However, the following doesn't seem like anything is being copied:
from StringIO import StringIO from shutil import copyfileobj s = StringIO() s2 = StringIO() s.write('asdf') copyfileobj(s, s2) s2.getvalue()
copyfileobj copies from the current location, and write leaves the file
pointer at the end of the file. a s.seek(0) before the copy fixes that.
to copy stringio objects, you can use f1 = StringIO(f2.getvalue()).
But this should have the same problem as using read(): a string will be created on the way which contains all the content.
getvalue() returns the contents of the f2 file as a string, and f1 will use that
string as the buffer. there's no extra copying.
Because I want to manipulate a copy of the data and be able to compare it to the original afterwards.
why not just use a plain string (or a list of strings)? your focus on StringIO sounds
like a leftover from some C library you've been using in an earlier life ;-)
Another thing I'd like to do is copy parts of a StringIO object's content to another object. This doesn't seem possible with any shutil method. Any idea on that?
use a plain string and slicing. (if you insist on using StringIO, use seek and read)
What one can really wonder, I admit, is why the difference between holding data two or three times in memory matters that much, especially if the latter is only for a short time. But as I'm going to use the code that handles the long string as a core component to some application, I'd like to make it behave as well as possible.
use plain strings, so you know what you're doing.
</F>
Fredrik Lundh wrote: copyfileobj copies from the current location, and write leaves the file pointer at the end of the file. a s.seek(0) before the copy fixes that.
Damn, this cannot be read from the documentation, and combined with the
fact that there's no length parameter for a portion to copy either, I
thought copying would mean copying all.
getvalue() returns the contents of the f2 file as a string, and f1 will use that string as the buffer. there's no extra copying.
Oh, good to know. Then StringIO(f2.getvalue()) or StringIO(f2.read())
would be the way to go. Because I want to manipulate a copy of the data and be able to compare it to the original afterwards.
why not just use a plain string (or a list of strings)? your focus on StringIO sounds like a leftover from some C library you've been using in an earlier life ;-)
Because the data can be a lot, and modifying long strings means a lot of
slicing and copying partial strings around, if I understand right.
Modifying a StringIO buffer is possible in-place. Plus, it's easier to
teach an algorithm that works on a StringIO to use a file instead, so I
may be able to avoid reading stuff into memory altogether in certain
places without worrying about special cases.
use a plain string and slicing. (if you insist on using StringIO, use seek and read)
OK.
--
Thomas This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Alex Vinokur |
last post by:
Copying files : input to output
===============================
C/C++ Performance Tests
=======================
Using C/C++ Program Perfometer
http://sourceforge.net/projects/cpp-perfometer...
|
by: carmen |
last post by:
Is it possible to show some movement in the screen while copying a file?
I tried to use a timer that change the position of an image but when I start
the copy of a file from diskette all is stopped...
|
by: John Smith |
last post by:
Hi folks,
I know how to place text into the user's clipboard:
Clipboard.SetDataObject("My Copied Text");
but how do I place a file in there?
So, if I have a file C:\test.txt, how can I place...
|
by: Martin Ho |
last post by:
I am running into one really big problem.
I wrote a script in vb.net to make a copy of folders and subfolder to
another destination:
- in 'from.txt' I specify which folders to copy
- in...
|
by: zMisc |
last post by:
Is it possible to copy a table from one schema to another schema by just
copying the frm file to the directory for the new schema?
What is the best way to create a new database with all the...
|
by: John |
last post by:
Hi all,
My application updates a sql server 2005 express database prior to copying
it with the result being the "in use by another process" and I cannot copy
it as a result. I've posted the code...
|
by: berwiki |
last post by:
I am trying to copy a table to another SQL 2000 Database, but I
continually get errors.
When I right-click, choose All-Tasks, Export-Data and go through the
DTS settings, I get an 'Unspecified...
|
by: Edhy Rijo [Progytech] |
last post by:
Hi All,
I am learning VB.NET and building small application that will do the following:
1.. Copy all files from a DVD to a specific folder in the hard drive and overwrite all files always.
The...
|
by: pooba53 |
last post by:
My VB .NET application (created in VS 2003) resides in c:\Program Files
\Test\ and there's a data.mdb (Access) file within this directory.
I have a feature in my program that allows the user to...
|
by: Tom |
last post by:
I am trying to update another developers code, and am stuck in a
cludge.
It works like this:
A user uploads a file from the web page. Our code reads from a
NetworkStream to a Filestream, and...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |