473,612 Members | 2,129 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

reading in lines from a file -FAST!

Hi
I have a file containing 168092 lines (each line a single word) and when
I use

for line in f:
s = s + line

it takes for ages to read it all in - so long in fact that it makes the
program unusable. Is there any way to do something like C's fread in
Python so that I can just slurp in 1.7MB of data at one go, rather than
reading line by line?

Thanks,
Rajarshi
Jul 18 '05 #1
3 5194
Rajarshi Guha wrote:
Hi
I have a file containing 168092 lines (each line a single word) and when
I use

for line in f:
s = s + line

it takes for ages to read it all in - so long in fact that it makes the
program unusable. Is there any way to do something like C's fread in
Python so that I can just slurp in 1.7MB of data at one go, rather than
reading line by line?

Thanks,
Rajarshi


Assuming f is an open File Object, you can use:

string = f.read()

Which reads the entire content of f into string. Notice, though, that
this solution isn't scalable: reading an entire file to memory becomes
messier the larger the file, whereas reading it one line at a time works
pretty much the same no matter how big the file is.

Adam

Jul 18 '05 #2
Why not use

s=openfile.read ()
s=string.split( s,'\n')

just a thought
sree

Jul 18 '05 #3
In article <pa************ *************** *******@preside ncy.com>,
Rajarshi Guha <ra******@presi dency.com> wrote:
Hi
I have a file containing 168092 lines (each line a single word) and when
I use

for line in f:
s = s + line

it takes for ages to read it all in - so long in fact that it makes the
program unusable. Is there any way to do something like C's fread in
Python so that I can just slurp in 1.7MB of data at one go, rather than
reading line by line?


How about:
s = f.read()
(assuming f is a file object)

-Mark
Jul 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2193
by: Boris Boutillier | last post by:
HI all, I came across a strange behaviour with read method, and I'm not sure if this is a filesystem problem or a misuse of this method. I do the simple following thing : f = open ('myFile','r') content = f.read() f.close() When I use this lines really intensively sometimes, at random the content
14
4499
by: Job Lot | last post by:
I have tab delimited text file which gets populated on daily basis via automated process. New entry is written at the bottom. I need to create a utility which makes a copy of this file with 10 most recent entries. When I read line using StreamReader object it starts from the top, so looping though lines and keeping track of the line is not helpful. Is there anyway to start reading from the bottom or if anyone could suggest some other...
7
2778
by: jamait | last post by:
Hi all, I m trying to read in a text file into a datatable... Not sure on how to split up the information though, regex or substrings...? sample: Col1 Col2 Col3 Col4 A0012430 REKAL TVÄTTMEDEL EKOMAX 0,5L ST 75.9000
3
2278
by: Steven Burn | last post by:
The application; Service on my webserver that allows a user to upload their HOSTS file for functions to verify the contents are still valid. Uses; 1. XMLHTTP (MSXML2) 2. FileSystemObject 3. CrazyBeavers Upload control (couldn't get the Dundas one to work)
20
33028
by: sahukar praveen | last post by:
Hello, I have a question. I try to print a ascii file in reverse order( bottom-top). Here is the logic. 1. Go to the botton of the file fseek(). move one character back to avoid the EOF. 2. From here read a character, print it, move the file pointer (FILE*) to 2 steps back (using fseek(fp, -2, SEEK_CUR)) to read the previous character. This seems to be ok if the file has a single line (i.e. no new line character). The above logic...
6
3772
by: KevinD | last post by:
assumption: I am new to C and old to COBOL I have been reading a lot (self teaching) but something is not sinking in with respect to reading a simple file - one record at a time. Using C, I am trying to read a flatfile. In COBOL, my simple file layout and READ statement would look like below. Question: what is the standard, simple coding convention for reading in a flatfile - one record at a time?? SCANF does not work because of...
11
6063
by: Girish Sahani | last post by:
I wrote the following code to concatenate every 2 keys of a dictionary and their corresponding values. e.g if i have tiDict1 = tiDict1 = {'a':,'b':} i should get tiDict2={'ab':} and similarly for dicts with larger no. of features. Now i want to check each pair to see if they are connected...element of this pair will be one from the first list and one from the second....e.g for 'ab' i want to check if 1 and 3 are connected,then 1 and...
2
1834
by: fool | last post by:
Dear group, I am a beginner in php and I was little bit experience in C language. I want to read a file's content and delete the first line of the file and display those lines which has got deleted. The content of the external file , which is in .cvs format is: -------------------------------------------------------------------
3
2821
by: The Cool Giraffe | last post by:
Regarding the following code i have a problem. void read () { fstream file; ios::open_mode opMode = ios::in; file.open ("some.txt", opMode); char *ch = new char; vector <charv; while (!file.eof ()) { do {
2
4090
by: rka77 | last post by:
Hi, I am trying to make a Python2.6 script on a Win32 that will read all the text files stored in a directory and print only the lines containing actual data. A sample file - Set : 1 Date: 10212009 12 34 56 25 67 90 End Set ******** Set: 2 Date: 10222009
0
8105
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8605
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8565
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8415
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7039
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4045
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4109
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2550
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1413
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.