input record seperator (equivalent of "$|" of perl)

[les_ander]

Hi,
I know that i can do readline() from a file object.
However, how can I read till a specific seperator?
for exmple,
if my files are

name
profession
id
#
name2
profession3
id2

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?
thanks

[Fredrik Lundh]

le*******@yahoo.com wrote:

I know that i can do readline() from a file object.
However, how can I read till a specific seperator?

for exmple,
if my files are

name
profession
id
#
name2
profession3
id2

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?

not really; you have to do it manually.

if the file isn't too large, consider reading all of it, and splitting on the
separator:

for record in file.read().split(separator):
print record # process record

if you're using a line-oriented separator, like in your example, tools like
itertools.groupby can be quite handy:

from itertools import groupby

def is_separator(line):
return line[:1] == "#"

for sep, record in groupby(file, is_separator):
if not sep:
print list(record) # process record

or you could just spell things out:

record = []
for line in file:
if line[0] == "#":
if record:
print record # process record
record = []
else:
record.append(line)
if record:
print record # process the last record, if any

</F>

[Keith Dart]

le*******@yahoo.com wrote:

Hi,
I know that i can do readline() from a file object.
However, how can I read till a specific seperator?
for exmple,
if my files are

name
profession
id
#
name2
profession3
id2

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?
thanks

I don't think so. But in the pyNMS package
(http://sourceforge/net/projects/pynms) there is a module called
"expect", and a class "Expect". With that you can wrap a file object and
use the Expect.read_until() method to do what you want.

--
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
Keith Dart <kd***@kdart.com>
public key: ID: F3D288E4
================================================== ===================

[Doug Holton]

le*******@yahoo.com wrote:

Hi,
I know that i can do readline() from a file object.
However, how can I read till a specific seperator?
for exmple,
if my files are

name
profession
id
#
name2
profession3
id2

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?
thanks

To actually answer your question, there is no equivalent to $| in python.

You need to hand code your own record parser, or else read in the whole
contents of the file and use the string split method to chop it up into
fields.

[M.E.Farmer]

What about a generator and xreadlines for those really large files:

py>def recordbreaker(recordpath, seperator='#'):
.... rec = open(recordpath ,'r')
.... xrecord = rec.xreadlines()
.... a =[]
.... for line in xrecord:
.... sep = line.find(seperator)
.... if sep != -1:
.... a.append(line[:sep])
.... out = ''.join(a)
.... a =[]
.... a.append(line[sep+1:])
.... yield out
.... else:
.... a.append(line)
.... if a:
.... yield ''.join(a)
.... rec.close()
....
py>records = recordbreaker('/tmp/myrecords.txt')
py>for item in records:
.... print item

M.E.Farmer

le*******@yahoo.com wrote:

Hi,
I know that i can do readline() from a file object.
However, how can I read till a specific seperator?
for exmple,
if my files are

name
profession
id
#
name2
profession3
id2

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?
thanks

[Fredrik Lundh]

"M.E.Farmer" wrote:

What about a generator and xreadlines for those really large files:

when you loop over a file object, Python uses a generator and a xreadlines-
style buffering system to read data as you go. (if you check the on-line help,
you'll notice that xreadlines itself is only provided for compatibility reasons).

or in other words, the examples I posted a couple of hours ago uses no more
memory than your version.

</F>

[M.E.Farmer]

Fredrik,
Thanks didn't realize that about reading a file on a for loop. Slick!
By the way the code I posted was an attempt at not building a
monolithic memory eating list like you did to return the values in your
second example.
Kinda thought it would be nice to read them as needed instead of all at
once.
I dont have itertools yet. That module looks like it rocks.
thanks for the pointers,
M.E.Farmer

[Scott David Daniels]

M.E.Farmer wrote:

I dont have itertools yet. That module looks like it rocks.
thanks for the pointers,
M.E.Farmer

If you have python 2.3 or 2.4, you have itertools.
--Scott David Daniels
Sc***********@Acm.Org

[M.E.Farmer]

Yea I should have mentioned I am running python 2.2.2.
Can it be ported to python 2.2.2?
Till they get python 2.4 all up and running....I'll wait a bit.
Thanks for the info,
M.E.Farmer

Scott David Daniels wrote:

M.E.Farmer wrote:

I dont have itertools yet. That module looks like it rocks.
thanks for the pointers,
M.E.Farmer

If you have python 2.3 or 2.4, you have itertools.
--Scott David Danies
Sc***********@Acm.Org

[Gábor Farkas]

Scott David Daniels wrote:

M.E.Farmer wrote:

I dont have itertools yet. That module looks like it rocks.
thanks for the pointers,
M.E.Farmer

If you have python 2.3 or 2.4, you have itertools.

for me it seems that 2.3 does not have itertools.groupby.
it has itertools, but not itertools.groupby.

activepython-2.4:(win)

import itertools
dir(itertools) ['__doc__', '__name__', 'chain', 'count', 'cycle', 'dropwhile',
'groupby', 'ifilter', 'ifilterfalse', 'imap', 'islice', 'izip',
'repeat', 'starmap', 'takewhile', 'tee']
python-2.3:(linux) import itertools
dir(itertools)

['__doc__', '__file__', '__name__', 'chain', 'count', 'cycle',
'dropwhile', 'ifilter', 'ifilterfalse', 'imap', 'islice', 'izip',
'repeat', 'starmap', 'takewhile']
gabor

[Nick Coghlan]

le*******@yahoo.com wrote:

I would like to read this file as a record.
I can do this in perl by defining a record seperator;
is there an equivalent in python?

Depending on your exact use case, you may also get some mileage out of using the
csv module with a custom delimeter.

Py> from csv import reader
Py> parsed = reader(demo, delimiter='|')
Py> for line in parsed: print line
....
['a', 'b', 'c', 'd']
['1', '2', '3', '4']

Cheers,
Nick.

P.S. 'demo' was created via:
Py> from tempfile import TemporaryFile
Py> demo = TemporaryFile()
Py> demo.write(txt)
Py> demo.seek(0)
Py> demo.read()
'a|b|c|d\n1|2|3|4'
Py> demo.seek(0)

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Scott David Daniels]

Gábor Farkas wrote:

Scott David Daniels wrote:

If you have python 2.3 or 2.4, you have itertools.

for me it seems that 2.3 does not have itertools.groupby.
it has itertools, but not itertools.groupby.

True. The 2.4 document says that itertools.groupby() is equivalent to:

class groupby(object):
def __init__(self, iterable, key=None):
if key is None:
key = lambda x: x
self.keyfunc = key
self.it = iter(iterable)
self.tgtkey = self.currkey = self.currvalue = xrange(0)
def __iter__(self):
return self
def next(self):
while self.currkey == self.tgtkey:
self.currvalue = self.it.next() # Exit on StopIteration
self.currkey = self.keyfunc(self.currvalue)
self.tgtkey = self.currkey
return (self.currkey, self._grouper(self.tgtkey))
def _grouper(self, tgtkey):
while self.currkey == tgtkey:
yield self.currvalue
self.currvalue = self.it.next() # Exit on StopIteration
self.currkey = self.keyfunc(self.currvalue)

So you could always just use that code.

--Scott David Daniels
Sc***********@Acm.Org

[Terry Reedy]

'separate' (se-parate == take a-part) and its derivatives are perhaps the
most frequently misspelled English word on clp. Seems to be 'par' for the
course. It has 2 e's bracketing 2 a's. It derives from the Latin
'parare', as does pare, so 'par' is the essential root of the word.

My gripe for the day, just to let non-native writers know what not to
imitate.

tjr

[Fredrik Lundh]

Scott David Daniels wrote:

True. The 2.4 document says that itertools.groupby() is equivalent to:

class groupby(object): So you could always just use that code.

the right way to do that is to use the Python version as a fallback:

try:
from itertools import groupby
except ImportError:
class groupby(object):
...

</F>

[Peter Otten]

Terry Reedy wrote:

'separate' (se-parate == take a-part) and its derivatives are perhaps the
most frequently misspelled English word on clp. Seems to be 'par' for the
course. It has 2 e's bracketing 2 a's. It derives from the Latin
'parare', as does pare, so 'par' is the essential root of the word.

My gripe for the day, just to let non-native writers know what not to
imitate.

I hereby suggest seperate/(separate+seperate) as the hamburger standard (see
http://www.oanda.com/products/bigmac/bigmac.shtml) for technical
communities. Some data points, adjectives only, s.e.e.o.:

the web: 4%
python: 9%
slashdot: 26%
perl: 29% *

Now draw your conclusions...

Peter

(*) Do you really believe I would have posted that if the outcome were in
favour of perl? No, definately not... lest you run out of gripes...

[Reinhold Birkenfeld]

Peter Otten wrote:

Terry Reedy wrote:

'separate' (se-parate == take a-part) and its derivatives are perhaps the
most frequently misspelled English word on clp. Seems to be 'par' for the
course. It has 2 e's bracketing 2 a's. It derives from the Latin
'parare', as does pare, so 'par' is the essential root of the word.

My gripe for the day, just to let non-native writers know what not to
imitate.

I hereby suggest seperate/(separate+seperate) as the hamburger standard (see
http://www.oanda.com/products/bigmac/bigmac.shtml) for technical
communities. Some data points, adjectives only, s.e.e.o.:

the web: 4%
python: 9%
slashdot: 26%
perl: 29% *

How did you get these data points?

Reinhold

--
[Windows ist wie] die Bahn: Man muss sich um nichts kuemmern, zahlt fuer
jede Kleinigkeit einen Aufpreis, der Service ist mies, Fremde koennen
jederzeit einsteigen, es ist unflexibel und zu allen anderen Verkehrs-
mitteln inkompatibel. -- Florian Diesch in dcoulm

[Fredrik Lundh]

Terry Reedy wrote:

My gripe for the day, just to let non-native writers know what not to imitate.

are there any non-native languages where separate are spelled seperate?

</F>

[John Machin]

Nick Coghlan wrote:
[snip]

delimeter.

Hey, Terry, another varmint over here!

[Steven Bethard]

John Machin wrote:

Nick Coghlan wrote:
[snip]

delimeter.

Hey, Terry, another varmint over here!

No, no. He's talking about a deli-meter. It's the metric standard for
measuring subs and sandwiches. ;)

Steve

[Peter Otten]

Reinhold Birkenfeld wrote:

the web: 4%
python: 9%
slashdot: 26%
perl: 29% *

How did you get these data points?

I copied the numbers from these pages:

http://www.google.com/search?q=separate
http://groups-beta.google.com/group/...hon&q=separate
http://www.google.com/search?q=site%...t.org+separate
http://groups-beta.google.com/group/...isc&q=separate

Same thing for the "alternative" spelling.

Peter

[Reinhold Birkenfeld]

Peter Otten wrote:

Reinhold Birkenfeld wrote:

the web: 4%
python: 9%
slashdot: 26%
perl: 29% *

How did you get these data points?

I copied the numbers from these pages:

http://www.google.com/search?q=separate
http://groups-beta.google.com/group/...hon&q=separate
http://www.google.com/search?q=site%...t.org+separate
http://groups-beta.google.com/group/...isc&q=separate

Same thing for the "alternative" spelling.

Thanks. A pity that there is no de.comp.lang.python, as for German posts
the "Standard/Standart" relation could be more accurate...

or-just-count-the-misplaces-apostrophs-ly yours, Reinhold

--
[Windows ist wie] die Bahn: Man muss sich um nichts kuemmern, zahlt fuer
jede Kleinigkeit einen Aufpreis, der Service ist mies, Fremde koennen
jederzeit einsteigen, es ist unflexibel und zu allen anderen Verkehrs-
mitteln inkompatibel. -- Florian Diesch in dcoulm

[Reinhold Birkenfeld]

Reinhold Birkenfeld wrote:

Peter Otten wrote:

Reinhold Birkenfeld wrote:

the web: 4%
python: 9%
slashdot: 26%
perl: 29% *

How did you get these data points?

I copied the numbers from these pages:

http://www.google.com/search?q=separate
http://groups-beta.google.com/group/...hon&q=separate
http://www.google.com/search?q=site%...t.org+separate
http://groups-beta.google.com/group/...isc&q=separate

Same thing for the "alternative" spelling.

Thanks. A pity that there is no de.comp.lang.python, as for German posts
the "Standard/Standart" relation could be more accurate...

or-just-count-the-misplaces-apostrophs-ly yours, Reinhold

s/misplaces/misplaced/

just-don't-try-to-write-a-perfect-posting-ly yours!

--
[Windows ist wie] die Bahn: Man muss sich um nichts kuemmern, zahlt fuer
jede Kleinigkeit einen Aufpreis, der Service ist mies, Fremde koennen
jederzeit einsteigen, es ist unflexibel und zu allen anderen Verkehrs-
mitteln inkompatibel. -- Florian Diesch in dcoulm

[John Machin]

Steven Bethard wrote:

John Machin wrote:

Nick Coghlan wrote:
[snip]

delimeter.
Hey, Terry, another varmint over here!

No, no. He's talking about a deli-meter. It's the metric standard

for measuring subs and sandwiches. ;)

Nobody mention the wurst! I did once, but I think I got away with it.

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

[Jeff Shannon]

John Machin wrote:

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

.... except in the US, where we stubbornly apply the same spelling to
both of those. (It figures that we Americans just ignore subtle
distinctions....)

Jeff Shannon
Technician/Programmer
Credit International

[Brian van den Broek]

John Machin said unto the world upon 2004-12-21 18:20:

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

Not everyone agrees <http://www.bartleby.com/61/18/M0251800.html>. I do,
but then I think it ought be spelled 'colour', too.

Best,

Brian vdB

[Steve Holden]

John Machin wrote:

Steven Bethard wrote:

John Machin wrote:

Nick Coghlan wrote:
[snip]
delimeter.

Hey, Terry, another varmint over here!

No, no. He's talking about a deli-meter. It's the metric standard

for

measuring subs and sandwiches. ;)

Nobody mention the wurst! I did once, but I think I got away with it.

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

So presumably a delimetre is the equivalent of just over three foot-long
subs? And that Monty Python pun was the wurst I've seen this weak.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]

[Nick Coghlan]

John Machin wrote:

Nick Coghlan wrote:
[snip]

delimeter.

Hey, Terry, another varmint over here!

Heh. Just don't get me started on the issues I have with typing apostrophes in
the right spot. My *brain* knows where they go, but for some reason it refuses
to let my fingers in on the secret. . .

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Christos TZOTZIOY Georgiou]

On 21 Dec 2004 15:20:35 -0800, rumours say that "John Machin"
<sj******@lexicon.net> might have written:

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

In this case, English (compared to American English) is closer to the
original "metron" [1]. Now, if only you people wrote the plural of
"parenthesis" as "parentheseis" and not "parentheses", that would ease a
lot my Greek English... :)

[1] through the French "mètre" of course; great job, those
revolutionaries did with the metric system. As Asimov put it, "how many
inches to the mile?"
--
TZOTZIOY, I speak England very best.
"Be strict when sending and tolerant when receiving." (from RFC1958)
I really should keep that in mind when talking with people, actually...

[Mike Meyer]

Christos "TZOTZIOY" Georgiou <tz**@sil-tec.gr> writes:

[1] through the French "mètre" of course; great job, those
revolutionaries did with the metric system. As Asimov put it, "how many
inches to the mile?"

Trivia question: Name the second most powerfull country on earth not
using the metric system for everything.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Andrew Dalke]

Mike Meyer:

Trivia question: Name the second most powerfull country on earth not
using the metric system for everything.

The UK?

Before going there I thought they were a fully metric country.
But I saw weather reports in degrees F, distances in yards
and miles, and of course pints of beer.

Andrew
da***@dalkescientific.com

[Steve Holden]

Andrew Dalke wrote:

Mike Meyer:

Trivia question: Name the second most powerfull country on earth not
using the metric system for everything.

The UK?

Before going there I thought they were a fully metric country.
But I saw weather reports in degrees F, distances in yards
and miles, and of course pints of beer.

Andrew
da***@dalkescientific.com

Plus the builders quite routinely ask suppliers for things like "a meter
of two (inches) by four (inches)". They don;t care that they're getting
the closest metric equivalent, to them a 2 x 4 is still a 2 x 4.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]

[JanC]

Jeff Shannon schreef:

John Machin wrote:

Subtle distinction: A metER is a measuring device. A MetRE is a unit of
distance.

... except in the US, where we stubbornly apply the same spelling to
both of those. (It figures that we Americans just ignore subtle
distinctions....)

Or there is some Dutch influence... ;)
(In Dutch it's "meter" for both meanings.)
--
JanC

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9

[Dennis Lee Bieber]

On Mon, 03 Jan 2005 23:15:23 GMT, Andrew Dalke
<da***@dalkescientific.com> declaimed the following in comp.lang.python:

Before going there I thought they were a fully metric country.
But I saw weather reports in degrees F, distances in yards
and miles, and of course pints of beer.
Yeah, but those pints are bigger than our pints...

Roughly 20 fl.oz. vs 16 fl.oz.

-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

[Dennis Lee Bieber]

On Mon, 03 Jan 2005 19:14:54 -0500, Steve Holden <st***@holdenweb.com>
declaimed the following in comp.lang.python:

Plus the builders quite routinely ask suppliers for things like "a meter
of two (inches) by four (inches)". They don;t care that they're getting
the closest metric equivalent, to them a 2 x 4 is still a 2 x 4.
And what do they get? A normal finished 2x4 runs about 1.5x3.5
inches...

A 1x4 is 0.75x3.5

-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

[Alex Martelli]

Mike Meyer <mw*@mired.org> wrote:
...

Trivia question: Name the second most powerfull country on earth not
using the metric system for everything.

Well, then, just name the 2nd most powerful country on earth, period
(I'm not going to get myself in trouble by guessing whether said 2nd
country is China, Russia, "Europe" [not a country, I know;-)], ...).

TVs and other displays are sold as "20 inches" (or whatever) everywhere,
printers' resolutions are always given in pixels per inch, etc.
Alex

[Steve Holden]

Dennis Lee Bieber wrote:

On Mon, 03 Jan 2005 19:14:54 -0500, Steve Holden <st***@holdenweb.com>
declaimed the following in comp.lang.python:

Plus the builders quite routinely ask suppliers for things like "a meter
of two (inches) by four (inches)". They don;t care that they're getting
the closest metric equivalent, to them a 2 x 4 is still a 2 x 4.

And what do they get? A normal finished 2x4 runs about 1.5x3.5
inches...

A 1x4 is 0.75x3.5

Who knows, but htey don;t complain about it. In the UK also the sizes
refer to the rough cut, before finish planing.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC [phone] [phone]