473,554 Members | 2,290 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

accessing parts of large files with File.seek()

I'm having a problem with the File object's seek() method.
Specifically, I cannot use it to seek to a location in a binary file
that is greater than 2^31 (2147483648). This seems unnecessarily
limiting, as it is common these days to have files larger than 2 GB.

Is there some LargeFile object out there that I can use to read my
file, which is approximately 3.3 GB in size?

Python version (freshly built from source this morning):
Python 2.5.1 (r251:54863, Aug 8 2007, 09:23:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin

Thanks,

Mike

Aug 8 '07 #1
2 3913
On Aug 8, 11:46 am, "mhearne808[insert-at-sign-here]gmail[insert-dot-
here]com" <mhearne...@gma il.comwrote:
I'm having a problem with the File object's seek() method.
Specifically, I cannot use it to seek to a location in a binary file
that is greater than 2^31 (2147483648). This seems unnecessarily
limiting, as it is common these days to have files larger than 2 GB.

Is there some LargeFile object out there that I can use to read my
file, which is approximately 3.3 GB in size?

Python version (freshly built from source this morning):
Python 2.5.1 (r251:54863, Aug 8 2007, 09:23:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin

Thanks,

Mike
I use large files quite a bit, but in Windows & Linux. Here's a quick
excerpt of a local ubuntu-linux session...

jaime@elizabeth :/var/virtualbox/VDI$ ls -l
total 1682536
-rw-rwx--- 1 jaime vboxusers 40960 2007-07-20 21:41 windows
xp.vdi
-rw-rwx--- 1 jaime vboxusers 3591387136 2007-07-20 21:53 winxp.vdi
jaime@elizabeth :/var/virtualbox/VDI$ ipython
Python 2.5.1 (r251:54863, May 2 2007, 16:27:44)
Type "copyright" , "credits" or "license" for more information.

IPython 0.7.3 -- An enhanced Interactive Python.
? -Introduction to IPython's features.
%magic -Information about IPython's 'magic' % functions.
help -Python's own help system.
object? -Details about 'object'. ?object also works, ?? prints more.

In [1]: f = file('winxp.vdi ')

In [2]: f.seek(35913871 32)

In [3]: f.read()
Out[3]: '\x00\x00\x00\x 00'

In [4]:

What exception are you receiving? (Not that I can offer any advice,
as I don't have a Mac to use for testing.)...

jw

Aug 9 '07 #2
On Aug 8, 7:37 pm, "programmer...@ gmail.com" <programmer...@ gmail.com>
wrote:
On Aug 8, 11:46 am, "mhearne808[insert-at-sign-here]gmail[insert-dot-

here]com" <mhearne...@gma il.comwrote:
I'm having a problem with the File object's seek() method.
Specifically, I cannot use it to seek to a location in a binary file
that is greater than 2^31 (2147483648). This seems unnecessarily
limiting, as it is common these days to have files larger than 2 GB.
Is there some LargeFile object out there that I can use to read my
file, which is approximately 3.3 GB in size?
Python version (freshly built from source this morning):
Python 2.5.1 (r251:54863, Aug 8 2007, 09:23:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Thanks,
Mike

I use large files quite a bit, but in Windows & Linux. Here's a quick
excerpt of a local ubuntu-linux session...

jaime@elizabeth :/var/virtualbox/VDI$ ls -l
total 1682536
-rw-rwx--- 1 jaime vboxusers 40960 2007-07-20 21:41 windows
xp.vdi
-rw-rwx--- 1 jaime vboxusers 3591387136 2007-07-20 21:53 winxp.vdi
jaime@elizabeth :/var/virtualbox/VDI$ ipython
Python 2.5.1 (r251:54863, May 2 2007, 16:27:44)
Type "copyright" , "credits" or "license" for more information.

IPython 0.7.3 -- An enhanced Interactive Python.
? -Introduction to IPython's features.
%magic -Information about IPython's 'magic' % functions.
help -Python's own help system.
object? -Details about 'object'. ?object also works, ?? prints more.

In [1]: f = file('winxp.vdi ')

In [2]: f.seek(35913871 32)

In [3]: f.read()
Out[3]: '\x00\x00\x00\x 00'

In [4]:

What exception are you receiving? (Not that I can offer any advice,
as I don't have a Mac to use for testing.)...

jw
I think I've figured out what the problem must be (not at Mac anymore,
so will have to test tomorrow). According to this link:
http://docs.python.org/lib/posix-large-files.html

I probably need to compile in large file support on my Mac. I get it
for free on my Ubuntu linux box...

FYI, it was an OverFlow Error.

Aug 9 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
2916
by: Jason Murry | last post by:
I have a camera system (Axis) which stores JPG via FTP 1-10fps. There is also a motion jpg live stream. I am trying to store these images either in JPG or in video format so they can be reviewed at a later date. I would need to be able to pull a date-time range from the list. This means 86,400 - 2,160,000 images (around 110Kb a piece)...
2
11731
by: Amy L. | last post by:
Is there a way through .net to read a very large text file (400MB+) backwards line by line. In system.io the filestream class has a "seek" method but the only read method requires you to know how many bytes to read in. My problem is that the line length of this log file is not constant so there is no easy way to read one line in. The only...
8
2888
by: Darsant | last post by:
I'm currently reading 1-n number of binary files, each with 3 different arrays of floats containing about 10,000 values a piece for a total of about 30,000 values per file. I'm looking for a way to load them all into memory. I've tried using vector pushback with reserving, but it was horribly slow. The current method I am using is upon...
6
8449
by: Rolf Schroedter | last post by:
(Sorry for cross-posting). I need to access large files > 2GByte (Linux, WinXP/NTFS) using the standard C-library calls. Till today I thought I know how to do it, namely for Win32: Use open(), read(), _itelli64(), _lseeki64() with type __int64 Linux/Cygwin: #define _FILE_OFFSET_BITS 64 Use open(), read(), lseek() with type off_t
4
2943
by: Almad | last post by:
Hello, I wonder how do I create reasonable thumbnails from JPEG with PIL. My code: logging.debug('Downloading image %s' % id) uri = ''.join(, '?p=', str(id)]) uf = urlopen(uri).read() f = tmpfile()
6
6588
by: Yi Xing | last post by:
Hi, I need to read specific lines of huge text files. Each time, I know exactly which line(s) I want to read. readlines() or readline() in a loop is just too slow. Since different lines have different size, I cannot use seek(). So I am thinking of building an index for the file for fast access. Can anybody give me some tips on how to do...
2
5441
by: Kevin Ar18 | last post by:
I posted this on the forum, but nobody seems to know the solution: http://python-forum.org/py/viewtopic.php?t=5230 I have a zip file that is several GB in size, and one of the files inside of it is several GB in size. When it comes time to read the 5+GB file from inside the zip file, it fails with the following error: File "...\zipfile.py",...
6
2452
by: =?Utf-8?B?RENX?= | last post by:
Hello all: I have a situation where I need to read a text file containing several million rows (insurance eligibility files). In additional to sequential operations, I also need to support a 'seek' on the file. The file itself is not in a fixed-field format and each line could be different lengths. I obviously don't want to simply start...
21
2631
by: pilafi | last post by:
Hallo guys!!I have a problem and i would like your help.I have to make a reader for a platform,where reads csv files.So, i made this read and its possible to read now csv files.But the problem is that the size of some csv files is 300MB the largest one.In that case my program breaks!I include with my question the reading part of my code and a test...
0
7612
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7536
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7814
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8053
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7895
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6163
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5165
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
1147
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
853
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.