I have a large string containing lines of text separated by '\n'. I'm
currently using text.splitlines (True) to break the text into lines, and
I'm iterating over the resulting list.
This is very slow (when using 400000 lines!). Other than dumping the
string to a file, and reading it back using the file iterator, is there a
way to quickly iterate over the lines?
I tried using newpos=text.fin d('\n', pos), and returning the chopped text
text[pos:newpos+1], but this is much slower than splitlines.
Any ideas?
Thanks
Jeremy 7 3202
Jeremy Sanders wrote: I have a large string containing lines of text separated by '\n'. I'm currently using text.splitlines (True) to break the text into lines, and I'm iterating over the resulting list.
This is very slow (when using 400000 lines!). Other than dumping the string to a file, and reading it back using the file iterator, is there a way to quickly iterate over the lines?
I tried using newpos=text.fin d('\n', pos), and returning the chopped text text[pos:newpos+1], but this is much slower than splitlines.
Any ideas?
Maybe [c]StringIO can be of help. I don't know if it's iterator is lazy. But
at least it has one, so you can try and see if it improves performance :)
--
Regards,
Diez B. Roggisch
On Fri, 25 Feb 2005 17:14:24 +0100, Diez B. Roggisch wrote: Maybe [c]StringIO can be of help. I don't know if it's iterator is lazy. But at least it has one, so you can try and see if it improves performance :)
Excellent! I somehow missed that module. StringIO speeds up the iteration
by a factor of 20!
Thanks
Jeremy
Jeremy,
How did you get the string in memory in the first place?
If you read it from a file, perhaps you should change to
reading it from the file a line at the time and use
file.readline as your iterator.
fp=file(inputfi le, 'r')
for line in fp:
...do your processing...
fp.close()
I don't think I would never read 400,000 lines as a single
string and then split it. Just a suggestion.
Larry Bates
Jeremy Sanders wrote: I have a large string containing lines of text separated by '\n'. I'm currently using text.splitlines (True) to break the text into lines, and I'm iterating over the resulting list.
This is very slow (when using 400000 lines!). Other than dumping the string to a file, and reading it back using the file iterator, is there a way to quickly iterate over the lines?
I tried using newpos=text.fin d('\n', pos), and returning the chopped text text[pos:newpos+1], but this is much slower than splitlines.
Any ideas?
Thanks
Jeremy
On Fri, 25 Feb 2005 10:57:59 -0600, Larry Bates wrote: How did you get the string in memory in the first place?
They're actually from a generated python script, acting as a saved file
format, something like:
interpret("""
lots of lines
""")
another_command ()
Obviously this isn't the most efficient format, but it's nice to
encapsulate the data and the script into one file.
Jeremy
Hi,
Using finditer in re module might help. I'm not sure it is lazy nor
performant. Here's an example :
=== BEGIN SNAP
import re
reLn = re.compile(r"""[^\n]*(\n|$)""")
sStr = \
"""
This is a test string.
It is supposed to be big.
Oh well.
"""
for oMatch in reLn.finditer(s Str):
print oMatch.group()
=== END SNAP
Regards,
Francis Girard
Le vendredi 25 Février 2005 16:55, Jeremy Sanders a écrit*: I have a large string containing lines of text separated by '\n'. I'm currently using text.splitlines (True) to break the text into lines, and I'm iterating over the resulting list.
This is very slow (when using 400000 lines!). Other than dumping the string to a file, and reading it back using the file iterator, is there a way to quickly iterate over the lines?
I tried using newpos=text.fin d('\n', pos), and returning the chopped text text[pos:newpos+1], but this is much slower than splitlines.
Any ideas?
Thanks
Jeremy
By putting them into another file you can just use
..readline iterator on file object to solve your
problem. I would personally find it hard to work
on a program that had 400,000 lines of data hard
coded into a structure like this, but that's me.
-Larry
Jeremy Sanders wrote: On Fri, 25 Feb 2005 10:57:59 -0600, Larry Bates wrote:
How did you get the string in memory in the first place?
They're actually from a generated python script, acting as a saved file format, something like:
interpret(""" lots of lines """) another_command ()
Obviously this isn't the most efficient format, but it's nice to encapsulate the data and the script into one file.
Jeremy
Jeremy Sanders wrote: On Fri, 25 Feb 2005 17:14:24 +0100, Diez B. Roggisch wrote:
Maybe [c]StringIO can be of help. I don't know if it's iterator is
lazy. But at least it has one, so you can try and see if it improves
performance :) Excellent! I somehow missed that module. StringIO speeds up the
iteration by a factor of 20!
Twenty?? StringIO.String IO or cStringIO.Strin gIO???
I did some "timeit" tests using the code below, on 400,000 lines of 53
chars (uppercase + lowercase + '\n').
On my config (Python 2.4, Windows 2000, 1.4 GHz Athlon chip, not short
of memory), cStringIO took 0.18 seconds and the "hard way" took 0.91
seconds. Stringio (not shown) took 2.9 seconds. FWIW, moving an
attribute look-up in the (sfind = s.find) saves only about 0.1 seconds.
python -m timeit -s "import itersplitlines as i; d =
i.mk_data(40000 0)" "i.test_csio(d) "
10 loops, best of 3: 1.82e+005 usec per loop
python -m timeit -s "import itersplitlines as i; d =
i.mk_data(40000 0)" "i.test_gen (d)"
10 loops, best of 3: 9.06e+005 usec per loop
A few questions:
(1) What is your equivalent of the "hard way"? What [c]StringIO code
did you use?
(2) How did you measure the time?
(3) How long does it take *compile* your 400,000-line Python script?
!import cStringIO
!
!def itersplitlines( s):
! if not s:
! yield s
! return
! pos = 0
! sfind = s.find
! epos = len(s)
! while pos < epos:
! newpos = sfind('\n', pos)
! if newpos == -1:
! yield s[pos:]
! return
! yield s[pos:newpos+1]
! pos = newpos+1
!
!def test_gen(s):
! for z in itersplitlines( s):
! pass
!
!def test_csio(s):
! for z in cStringIO.Strin gIO(s):
! pass
!
!def mk_data(n):
! import string
! return (string.lowerca se + string.uppercas e + '\n') * n This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: qwweeeit |
last post by:
Hi all,
I am writing a script to visualize (and print)
the web references hidden in the html files as:
'<a href="web reference"> underlined reference</a>'
Optimizing my code, I found that an essential step is:
splitting on a word (in this case 'href').
I am asking if there is some alternative (more pythonic...):
# SplitMultichar.py
|
by: pmatos |
last post by:
Hi all,
I have 2 char* iterators str and end and I'm doing as follows:
string id_str(str, end);
const char * id = id_str.c_str();
but these has 2 problems afaik. One, I'm generating a string as an
intermediate step to get a char*, which seems useless. Two, I don't
know when id gets destroyed or when the chars to where id points to are
cleaned. I could now use strcopy to copy id to a freshly allocated
|
by: Dr. StrangeLove |
last post by:
Greetings,
Let say we want to split column 'list' in table lists
into separate rows using the comma as the delimiter.
Table lists
id list
1 aa,bbb,c
2 e,f,gggg,hh
3 ii,kk
4 m
|
by: Trint Smith |
last post by:
Ok,
My program has been formating .txt files for input into sql server and
ran into a problem...the .txt is an export from an accounting package
and is only supposed to contain comas (,) between fields in a
table...well, someone has been entering description fields with comas
(,) in the description and now it is splitting between one
field...example:
"santa clause mushrooms, pens, cups and dolls"
I somehow need to NOT split anything...
|
by: Opettaja |
last post by:
I am new to c# and I am currently trying to make a program to retrieve
Battlefield 2 game stats from the gamespy servers. I have got it so I
can retrieve the data but I do not know how to cut up the data to
assign each value to its own variable. So right now I am just saving
the data to a txt file and when I look in the text file all the data is
there.
Not sure if this matters but when I open the text file in Word pad
(Rich Text) It...
| |
by: scroopy |
last post by:
Hi,
I've always used std::string but I'm having to use a 3rd party library
that returns const char*s. Given:
char* pString1 = "Blah ";
const char* pString2 = "Blah Blah";
How do I append the contents of pString2 to pString? (giving "Blah
Blah Blah")
|
by: AG |
last post by:
Hi,
I have a vector that represent memory in my code. I would like to split it into two smaller vector, without
copying it. I want the split to be "in-place", so that modifications on the two smaller vectors would affect the
original one.
Exemple :
int N=10;
int i;
|
by: CoreyWhite |
last post by:
/* WORKING WITH STRINGS IN C++ IS THE BEST WAY TO LEARN THE LANGUAGE
AND TRANSITION FROM C. C++ HAS MANY NEW FEATURES THAT WORK TOGETHER
AND WHEN YOU SEE THEM DOING THE IMPOSSIBLE AND MAKING COMPACT COHERENT
CODE THAT WORKS WITH STRINGS, IT ALL BEGINS TO MAKE SINCE*/
/* The basics of C++ are Classes, that build Types. Which are used to
create quick and dirty routines in the smallest possible space. The
Classes & Routines uses the...
|
by: shadow_ |
last post by:
Hi i m new at C and trying to write a parser and a string class.
Basicly program will read data from file and splits it into lines then
lines to words. i used strtok function for splitting data to lines it
worked quite well but srttok isnot working for multiple blank or
commas. Can strtok do this kind of splitting if it cant what should i
use .
Unal
|
by: Eyes Of Madness |
last post by:
I'm doing a program for a class of mine and I am having trouble splitting my strings up. I know you can do something like:
a = '012345'
a
returns 012
but I am inputing strings of varying length and I cant just do the above notation. I need to split the string into groups of 3 in order to work. Any help would be much appreciated.
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |