473,320 Members | 1,982 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

reading file objects in chunks

Hi,

I'm looking for something that will give me an iterator to a
file-(like)-object. I have large files with only a single line in it
that have fixed length fields like, record length is 26bytes, dataA is
10 bytes, dataB is 16 bytes.

Now when I made my parsing stuff but can't find anything that will let
me read those file efficiently (guess I'm just thinking too
complicated). I'd like to have something like:

f = file("datafile.dat", buffering=26)

for chunk in f.read_in_chunks():
compute_data(chunk)

f.iter() looked promising at first but somehow it doesn't do "the
right thing"(tm). also itertools doesn't quite seem to be what I want.
Maybe I just need coffee but right now I'm in the dark.

I'd really like something nicer than

chunksize = 26
f = file("datafile.dat", buffering=chunksize)

chunk = f.read(chunksize)
while len(chunk) == chunksize:
compute_data(chunk)
f.read(chunksize)

I just don't feel comfortable with it for some reason I can't explain...

thanks
martin

--
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours
Nov 12 '07 #1
1 3668
On Mon, 12 Nov 2007 17:47:29 +0100, Martin Marcher wrote:
I'd really like something nicer than

chunksize = 26
f = file("datafile.dat", buffering=chunksize)

chunk = f.read(chunksize)
while len(chunk) == chunksize:
compute_data(chunk)
f.read(chunksize)

I just don't feel comfortable with it for some reason I can't explain...
chunksize = 26
f = open('datafile.dat', 'rb')
for chunk in iter(lambda: f.read(chunksize), ''):
compute_data(chunk)
f.close()

Ciao,
Marc 'BlackJack' Rintsch
Nov 12 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: sashan | last post by:
Is the way to use DOM for an xml file as follows: 1) Read the file into a string 2) Call xml.dom.minidom.parseString(string)
6
by: Dietrich Epp | last post by:
Are there any good modules for reading a bitstream? Specifically, I have a string and I want to be able to get the next N bits as an integer. Right now I'm using struct.unpack and bit operations,...
3
by: David M. Wilson | last post by:
Hello! Is there a simple way of reading from a file object up to a specific byte value? I would like to do this without reading one character at a time, or reading in chunks and holding a...
6
by: tkpmep | last post by:
I have a text file with many hundreds of lines of data. The data of interest to me, however, resides at the bottom of the file, in the last 20 lines. Right now, I read the entire file and discard...
3
by: syntax | last post by:
hi, i want to read a file using fread() in equal chunks through a for loop so that at the last call i dont get error..which way, i should read it? let me give an example, suppose i have 100...
3
by: Nick | last post by:
I have found a class that compresses and uncompresses data but need some help with how to use part of it below is the deflate method which compresses the string that I pass in, this works OK. At...
1
by: Gustaf | last post by:
I'm converting a program I made in Python once to C#, and while I'm at it, I want to do some performance improvements. The program takes a set of mailbox files (in readable formats, like Eudora and...
1
by: adhag | last post by:
Hi I have an app that uses xpath to read an xml document. The problem is a 70meg file uses 1.5 gig of memory. What I really need is to read only chunks of the file at a given time and cannot...
16
by: WATYF | last post by:
Hi there... I have a huge text file that needs to be processed. At the moment, I'm loading it into memory in small chunks (x amount of lines) and processing it that way. I'd like the process to be...
3
by: Willy Stevens | last post by:
Hello, In my application I have to read sometimes quite big chunk of binary data. I have a buffer which default size is 32000 bytes. But how could I read binary data that exceeds 32000 bytes?...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.