Hey all,
I have a for loop which included the line:
items_ren.append(join(newPath,renamedItem))
I changed it to this:
items_ren.append([join(newPath,renamedItem), False])
And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!
Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA
BTW, the 'join' function is this:
def join(newPath,renamedItem):
return unicode(os.path.join(newPath,renamedItem)) 3 1108
On Dec 21, 3:29 pm, "ianaré" <ian...@gmail.comwrote:
Hey all,
I have a for loop which included the line:
items_ren.append(join(newPath,renamedItem))
I changed it to this:
items_ren.append([join(newPath,renamedItem), False])
And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!
Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA
BTW, the 'join' function is this:
def join(newPath,renamedItem):
return unicode(os.path.join(newPath,renamedItem))
Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)
9.5 seconds wOOt!
However, if anyone has a faster way, let me know!
ianaré wrote:
>I changed it to this: items_ren.append([join(newPath,renamedItem), False])
And processing speed is now much much slower. For 5780 items the old function would take 9.5 seconds (there is other stuff going on obviously), after changing that single line, speed is now 55 seconds in the same conditions!
have you checked the process size? short lists are relatively expensive
to create, both performance-wise, and more importantly, memory-wise.
try using tuples instead.
or better, move the flags to a separate data structure (either a list,
or, if possible, a set or dictionary keyed on filenames).
</F>
ianaré wrote:
On Dec 21, 3:29 pm, "ianaré" <ian...@gmail.comwrote:
>Hey all,
I have a for loop which included the line:
items_ren.append(join(newPath,renamedItem))
I changed it to this: items_ren.append([join(newPath,renamedItem), False])
And processing speed is now much much slower. For 5780 items the old function would take 9.5 seconds (there is other stuff going on obviously), after changing that single line, speed is now 55 seconds in the same conditions!
Can anyone give me some pointers please? I would like to keep the same structure if possible, as it allows the code in other places to be much faster. TIA
BTW, the 'join' function is this: def join(newPath,renamedItem): return unicode(os.path.join(newPath,renamedItem))
Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)
9.5 seconds wOOt!
However, if anyone has a faster way, let me know!
Maybe
import itertools as it
items_ren = zip(items_ren, it.repeat(False))
Or if you want to start earlier on
def gen_items():
for ...
yield newPath, renamedItem
items_ren = zip(it.imap(unicode, it.starmap(os.path.join, gen_items())),
it.repeat(False))
Probably buggy, but you should get the idea...
Peter This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: newmaid |
last post by:
Hi all
just put together this little bit of code... I am sure it is heavily flawed:(... but can someone please point out to me the reason why it becomes slower after every loop. thanks!!!
**...
|
by: Alexzive |
last post by:
Hello there :) ,
I am a python newbie and need to run following code for a task in an
external simulation programm called "Abaqus" which makes use of python
to access the mesh (ensamble of nodes...
|
by: guillaume weymeskirch |
last post by:
Hello everybody,
To test the python 2.5 garbage collector, I wrote a trivial script
allocating dummy objects of various sizes, then forgetting them in a loop.
The garbage collector seems...
|
by: tejsupra |
last post by:
Hello Everyone,
I need to read a .csv file which has a size of 2.26 GB . And I wrote a
Python script , where I need to read this file. And my Computer has 2
GB RAM Please see the code as...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
| |