Text processing and file creation

[malibuster]

I have a text source file of about 20.000 lines.

>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

[kyosohma]

On Sep 5, 11:13 am, "[e-mail]" <[e-mail]>
wrote:

I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file. Reset the counter every 5 lines/
iterations and close the file. To name files with unique names, use
the time module. Something like this:

x = 'filename-%s.txt' % time.time()

Have fun!

Mike

[Arnau Sanchez]

ma********@gmail.com escribió:

I have a text source file of about 20.000 lines.

>>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?

Perhaps you could provide some code to see how you approached it?

[Bjoern Schliessmann]

ky******@gmail.com wrote:

I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file.

file objects are iterables themselves, so there's no need to do that
by using a method.

Reset the counter every 5 lines/ iterations and close the file.

I'd use a generator that fetches five lines of the file per
iteration and iterate over it instead of the file directly.

Have fun!

Definitely -- and also do your homework yourself :)

Regards,
Björn

--
BOFH excuse #339:

manager in the cable duct

[Shawn Milochik]

On 9/5/07, ma********@gmail.com <ma********@gmail.comwrote:

I have a text source file of about 20.000 lines.

From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

I have written a working test of this. Here's the basic setup:


open the input file

function newFileName:
generate a filename (starting with 00001.tmp).
If filename exists, increment and test again (0002.tmp and so on).
return fileName

read a line until input file is empty:

test to see whether I have written five lines. If so, get a new
file name, close file, and open new file

write line to file

close output file final time
Once you get some code running, feel free to post it and we'll help.

[kyosohma]

On Sep 5, 11:57 am, Bjoern Schliessmann <usenet-
[e-mail]:

[e-mail] wrote:

I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file.

file objects are iterables themselves, so there's no need to do that
by using a method.

Very true! Darn it!

>

Reset the counter every 5 lines/ iterations and close the file.

I'd use a generator that fetches five lines of the file per
iteration and iterate over it instead of the file directly.

I still haven't figured out how to use generators, so this didn't even
come to mind. I usually see something like this example for reading a
file:

f = open(somefile)
for line in f:
# do something
http://docs.python.org/tut/node9.html

Okay, so they didn't use readline. I wonder where I saw that.

Have fun!

Definitely -- and also do your homework yourself :)

Regards,

Björn

--
BOFH excuse #339:

manager in the cable duct

Mike

[Paddy]

On Sep 5, 5:13 pm, "[e-mail]" <[e-mail]>
wrote:

I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

If its on unix: use split.
If its your homework: show us what you have so far...

- Paddy.

[malibuster]

On Sep 5, 1:28 pm, Paddy <[e-mail]:

On Sep 5, 5:13 pm, "[e-mail]" <[e-mail]>
wrote:

I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

If its on unix: use split.
If its your homework: show us what you have so far...

- Paddy.

Paddy,

Thanks for making me aware of the (UNIX) split command (split -l 5
inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

-- Martin.
I am still wondering how to do this in Python (being new to Python)

[Arnaud Delobelle]

On Sep 5, 5:13 pm, "[e-mail]" <[e-mail]>
wrote:

I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?

Sure!

In advance, thanks for your help.

from my_useful_functions import new_file, write_first_5_lines,
done_processing_file, grab_next_5_lines, another_new_file, write_these

in_f = open('myfile')
out_f = new_file()
write_first_5_lines(in_f, out_f) # write first 5 lines
close(out_f)
while not done_processing_file(in_f): # until done processing
lines = grab_next_5_lines(in_f) # grab next 5 lines
out_f = another_new_file()
write_these(lines, out_f) # write these
close(out_f)
print "all done!" # All done
print "Now there are 4000 files in this directory..."

Python 3.0 - ready (I've used open() instead of file())

HTH

--
Arnaud

[Steve Holden]

Arnaud Delobelle wrote:
[...]

from my_useful_functions import new_file, write_first_5_lines,
done_processing_file, grab_next_5_lines, another_new_file, write_these

in_f = open('myfile')
out_f = new_file()
write_first_5_lines(in_f, out_f) # write first 5 lines
close(out_f)
while not done_processing_file(in_f): # until done processing
lines = grab_next_5_lines(in_f) # grab next 5 lines
out_f = another_new_file()
write_these(lines, out_f) # write these
close(out_f)
print "all done!" # All done
print "Now there are 4000 files in this directory..."

Python 3.0 - ready (I've used open() instead of file())

bzzzzzzzzzzt!

Python 3.0a1 (py3k:57844, Aug 31 2007, 16:54:27) ...
Type "help", "copyright", "credits" or "license" for more information.

>>print "all done!" # All done

File "<stdin>", line 1
print "all done!" # All done
^
SyntaxError: invalid syntax

>>>

Close, but no cigar ;-)

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

[Arnaud Delobelle]

On Sep 6, 12:46 am, Steve Holden <[e-mail]:

Arnaud Delobelle wrote:

[...]

print "all done!" # All done
print "Now there are 4000 files in this directory..."

Python 3.0 - ready (I've used open() instead of file())

bzzzzzzzzzzt!

Python 3.0a1 (py3k:57844, Aug 31 2007, 16:54:27) ...
Type "help", "copyright", "credits" or "license" for more information.

>>print "all done!" # All done

File "<stdin>", line 1
print "all done!" # All done
^
SyntaxError: invalid syntax

>>>

Damn! That'll teach me to make such bold claims.
At least I'm unlikely to forget again now...

--
Arnaud

[Alberto Griggio]

Thanks for making me aware of the (UNIX) split command (split -l 5

inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

Something like this should do the job:

def nlines(num, fileobj):
done = [False]
def doit():
for i in xrange(num):
l = fileobj.readline()
if not l:
done[0] = True
return
yield l
while not done[0]:
yield doit()

for i, group in enumerate(nlines(5, open('bigfile.txt'))):
out = open('chunk_%d.txt' % i)
for line in group:
out.write(line)

I am still wondering how to do this in Python (being new to Python)

This is just one way of doing it, but not as concise as using split...

Alberto

[Arnau Sanchez]

ma********@gmail.com escribió:

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

You should post some code anyway, it would be easier to give useful advice (it
would also demonstrate that you put some effort on it).

Anyway, here is an option. Text-file objects are line-iterable, so you could use
itertools (perhaps a bit difficult module for a newbie...):

from itertools import islice, takewhile, repeat

def take(it, n):
return list(islice(it, n))

def readnlines(fd, n):
return takewhile(bool, (take(fd, n) for _ in repeat(None)))

def splitfile(path, prefix, nlines, suffix_digits):
sformat = "%%0%dd" % suffix_digits
for index, lines in enumerate(readnlines(file(path), nlines)):
open("%s_%s"%(prefix, sformat % index), "w").writelines(lines)

splitfile("/etc/services", "out", 5, 4)

arnau

[Shawn Milochik]

Here's my solution, for what it's worth:

#!/usr/bin/env python

import os

input = open("test.txt", "r")

counter = 0
fileNum = 0
fileName = ""

def newFileName():

global fileNum, fileName
while os.path.exists(fileName) or fileName == "":
fileNum += 1
x = "%0.5d" % fileNum
fileName = "%s.tmp" % x

return fileName
for line in input:

if (fileName == "") or (counter == 5):
if fileName:
output.close()
fileName = newFileName()
counter = 0
output = open(fileName, "w")

output.write(line)
counter += 1

output.close()

[=?ISO-8859-1?Q?Ricardo_Ar=E1oz?=]

Shawn Milochik wrote:

On 9/5/07, ma********@gmail.com <ma********@gmail.comwrote:

>I have a text source file of about 20.000 lines.

>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

Maybe (untested):

def read5Lines(f):
L = f.readline()
while L :
yield (L,f.readline(),f.readline(),f.readline(),f.readli ne())
L = f.readline()

in = open('C:\YourFile','rb')
for fileNo, fiveLines in enumerate(read5Lines(in)) :
out = open('c:\OutFile'+str(fileNo), 'wb')
out.writelines(fiveLines)
out.close()

or something similar? (notice that in the last output file you may have
a few (4 at most) blank lines)

[George Sakkis]

On Sep 5, 5:17 pm, "[e-mail]" <[e-mail]>
wrote:

On Sep 5, 1:28 pm, Paddy <[e-mail]:

On Sep 5, 5:13 pm, "[e-mail]" <[e-mail]>
wrote:

I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

If its on unix: use split.
If its your homework: show us what you have so far...

- Paddy.

Paddy,

Thanks for making me aware of the (UNIX) split command (split -l 5
inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

-- Martin.

I am still wondering how to do this in Python (being new to Python)

If this was a code golf challenge, a decent entry (146 chars) could
be:

import itertools as it
for i,g in it.groupby(enumerate(open('input.txt')),lambda(i,_ ):i/
5):open("output.%d.txt"%i,'w').writelines(s for _,s in g)

or a bit less cryptically:

import itertools as it
for chunk,enum_lines in it.groupby(enumerate(open('input.txt')),
lambda (i,line): i//5):
open("output.%d.txt" % chunk, 'w').writelines(line for _,line
in enum_lines)
George

[Paddy]

On Sep 7, 3:50 am, George Sakkis <[e-mail]:

On Sep 5, 5:17 pm, "[e-mail]" <[e-mail]>
wrote:
If this was a code golf challenge,

I'd choose the Unix split solution and be both maintainable as well as
concise :-)

- Paddy.