473,289 Members | 2,087 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Re: "Faster" I/O in a script

On Jun 2, 2:08*am, "kalakouentin" <kalakouen...@yahoo.comwrote:
Do you know a way to actually load my data in a more
"batch-like" way so I will avoid the constant line by line reading?
If your files will fit in memory, you can just do

text = file.readlines()

and Python will read the entire file into a list of strings named
'text,' where each item in the list corresponds to one 'line' of the
file.
Jun 27 '08 #1
2 1037
mi***********@gmail.com wrote:
On Jun 2, 2:08 am, "kalakouentin" <kalakouen...@yahoo.comwrote:

> Do you know a way to actually load my data in a more
"batch-like" way so I will avoid the constant line by line reading?

If your files will fit in memory, you can just do

text = file.readlines()

and Python will read the entire file into a list of strings named
'text,' where each item in the list corresponds to one 'line' of the
file.
No that won't help. That has to do *all* the same work (reading blocks
and finding line endings) as the iterator PLUS allocate and build a list.

Better to just use the iterator.

for line in file:
...

Gary Herron
--
http://mail.python.org/mailman/listinfo/python-list
Jun 27 '08 #2
Gary Herron wrote:
mi***********@gmail.com wrote:
>On Jun 2, 2:08 am, "kalakouentin" <kalakouen...@yahoo.comwrote:

>> Do you know a way to actually load my data in a more
"batch-like" way so I will avoid the constant line by line reading?

If your files will fit in memory, you can just do

text = file.readlines()

and Python will read the entire file into a list of strings named
'text,' where each item in the list corresponds to one 'line' of the
file.

No that won't help. That has to do *all* the same work (reading blocks
and finding line endings) as the iterator PLUS allocate and build a list.
Better to just use the iterator.

for line in file:
...
Actually this *can* be much slower. Suppose I want to search a file to
see if a substring is present.

st = "some substring that is not actually in the file"
f = <50 MB log file>

Method 1:

for i in file(f):
if st in i:
break

--0.472416 seconds

Method 2:

Read whole file:

fh = file(f)
rl = fh.read()
fh.close()

--0.098834 seconds

"st in rl" test --0.037251 (total: .136 seconds)

Method 3:

mmap the file:

mm = mmap.mmap(fh.fileno(), 0, mmap.MAP_SHARED, mmap.PROT_READ)
"st in mm" test --3.589938 (<-- see my post the other day)

mm.find(st) --0.186895

Summary:

If you can afford the memory, it can be more efficient (more than 3
times faster in this example) to read the file into memory and process
it at once (if possible).

Mmapping the file and processing it at once is roughly as fast (I didnt
measure the difference carefully), but has the advantage that if there
are parts of the file you do not touch you don't fault them into memory.
You could also play more games and mmap chunks at a time to limit the
memory use (but you'd have to be careful with mmapping that doesn't
match record boundaries).

Kris
Jun 27 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Alex Ang | last post by:
I have written the following VBScript program. It is stored into a file "map_drive.vbs". It successfully mapped to a network drive \\server1\data. Dim WshNetwork Set WshNetwork =...
2
by: Craig Stadler | last post by:
mysql 4.0.22 (win32) Can anyone recommend best practices for the fastest way to remove large numbers of rows at once? I am diving my deletes into chunks (1000 rows at a time) but this still is...
2
by: Alexandre Ferrieux | last post by:
Hi, In a recent thread I discovered why the "for line in f" idiom was not suitable for live sources (pipes, sockets, tty). The reason is that it uses buffering on input, blocking on a full...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.