471,321 Members | 2,012 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,321 software developers and data experts.

Text processing and file creation

I have a text source file of about 20.000 lines.
>From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

Sep 5 '07 #1
16 1356
On Sep 5, 11:13 am, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.
I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file. Reset the counter every 5 lines/
iterations and close the file. To name files with unique names, use
the time module. Something like this:

x = 'filename-%s.txt' % time.time()

Have fun!

Mike

Sep 5 '07 #2
ma********@gmail.com escribió:
I have a text source file of about 20.000 lines.
>>From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
Perhaps you could provide some code to see how you approached it?
Sep 5 '07 #3
ky******@gmail.com wrote:
I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file.
file objects are iterables themselves, so there's no need to do that
by using a method.
Reset the counter every 5 lines/ iterations and close the file.
I'd use a generator that fetches five lines of the file per
iteration and iterate over it instead of the file directly.
Have fun!
Definitely -- and also do your homework yourself :)

Regards,
Björn

--
BOFH excuse #339:

manager in the cable duct

Sep 5 '07 #4
On 9/5/07, ma********@gmail.com <ma********@gmail.comwrote:
I have a text source file of about 20.000 lines.
From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

I have written a working test of this. Here's the basic setup:


open the input file

function newFileName:
generate a filename (starting with 00001.tmp).
If filename exists, increment and test again (0002.tmp and so on).
return fileName

read a line until input file is empty:

test to see whether I have written five lines. If so, get a new
file name, close file, and open new file

write line to file

close output file final time
Once you get some code running, feel free to post it and we'll help.
Sep 5 '07 #5
On Sep 5, 11:57 am, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n...@spamgourmet.comwrote:
kyoso...@gmail.com wrote:
I would use a counter in a for loop using the readline method to
iterate over the 20,000 line file.

file objects are iterables themselves, so there's no need to do that
by using a method.
Very true! Darn it!
>
Reset the counter every 5 lines/ iterations and close the file.

I'd use a generator that fetches five lines of the file per
iteration and iterate over it instead of the file directly.
I still haven't figured out how to use generators, so this didn't even
come to mind. I usually see something like this example for reading a
file:

f = open(somefile)
for line in f:
# do something
http://docs.python.org/tut/node9.html

Okay, so they didn't use readline. I wonder where I saw that.
Have fun!

Definitely -- and also do your homework yourself :)

Regards,

Björn

--
BOFH excuse #339:

manager in the cable duct
Mike

Sep 5 '07 #6
On Sep 5, 5:13 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.
If its on unix: use split.
If its your homework: show us what you have so far...

- Paddy.

Sep 5 '07 #7
On Sep 5, 1:28 pm, Paddy <paddy3...@googlemail.comwrote:
On Sep 5, 5:13 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.

If its on unix: use split.
If its your homework: show us what you have so far...

- Paddy.
Paddy,

Thanks for making me aware of the (UNIX) split command (split -l 5
inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

-- Martin.
I am still wondering how to do this in Python (being new to Python)

Sep 5 '07 #8
On Sep 5, 5:13 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close

that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
Sure!
In advance, thanks for your help.
from my_useful_functions import new_file, write_first_5_lines,
done_processing_file, grab_next_5_lines, another_new_file, write_these

in_f = open('myfile')
out_f = new_file()
write_first_5_lines(in_f, out_f) # write first 5 lines
close(out_f)
while not done_processing_file(in_f): # until done processing
lines = grab_next_5_lines(in_f) # grab next 5 lines
out_f = another_new_file()
write_these(lines, out_f) # write these
close(out_f)
print "all done!" # All done
print "Now there are 4000 files in this directory..."

Python 3.0 - ready (I've used open() instead of file())

HTH

--
Arnaud
Sep 5 '07 #9
Arnaud Delobelle wrote:
[...]
from my_useful_functions import new_file, write_first_5_lines,
done_processing_file, grab_next_5_lines, another_new_file, write_these

in_f = open('myfile')
out_f = new_file()
write_first_5_lines(in_f, out_f) # write first 5 lines
close(out_f)
while not done_processing_file(in_f): # until done processing
lines = grab_next_5_lines(in_f) # grab next 5 lines
out_f = another_new_file()
write_these(lines, out_f) # write these
close(out_f)
print "all done!" # All done
print "Now there are 4000 files in this directory..."

Python 3.0 - ready (I've used open() instead of file())
bzzzzzzzzzzt!

Python 3.0a1 (py3k:57844, Aug 31 2007, 16:54:27) ...
Type "help", "copyright", "credits" or "license" for more information.
>>print "all done!" # All done
File "<stdin>", line 1
print "all done!" # All done
^
SyntaxError: invalid syntax
>>>
Close, but no cigar ;-)

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Sep 5 '07 #10
On Sep 6, 12:46 am, Steve Holden <st...@holdenweb.comwrote:
Arnaud Delobelle wrote:
[...]
print "all done!" # All done
print "Now there are 4000 files in this directory..."
Python 3.0 - ready (I've used open() instead of file())

bzzzzzzzzzzt!

Python 3.0a1 (py3k:57844, Aug 31 2007, 16:54:27) ...
Type "help", "copyright", "credits" or "license" for more information.
>>print "all done!" # All done
File "<stdin>", line 1
print "all done!" # All done
^
SyntaxError: invalid syntax
>>>
Damn! That'll teach me to make such bold claims.
At least I'm unlikely to forget again now...

--
Arnaud

Sep 6 '07 #11
Thanks for making me aware of the (UNIX) split command (split -l 5
inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).
Something like this should do the job:

def nlines(num, fileobj):
done = [False]
def doit():
for i in xrange(num):
l = fileobj.readline()
if not l:
done[0] = True
return
yield l
while not done[0]:
yield doit()

for i, group in enumerate(nlines(5, open('bigfile.txt'))):
out = open('chunk_%d.txt' % i)
for line in group:
out.write(line)

I am still wondering how to do this in Python (being new to Python)
This is just one way of doing it, but not as concise as using split...

Alberto
Sep 6 '07 #12
ma********@gmail.com escribió:
I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).
You should post some code anyway, it would be easier to give useful advice (it
would also demonstrate that you put some effort on it).

Anyway, here is an option. Text-file objects are line-iterable, so you could use
itertools (perhaps a bit difficult module for a newbie...):

from itertools import islice, takewhile, repeat

def take(it, n):
return list(islice(it, n))

def readnlines(fd, n):
return takewhile(bool, (take(fd, n) for _ in repeat(None)))

def splitfile(path, prefix, nlines, suffix_digits):
sformat = "%%0%dd" % suffix_digits
for index, lines in enumerate(readnlines(file(path), nlines)):
open("%s_%s"%(prefix, sformat % index), "w").writelines(lines)

splitfile("/etc/services", "out", 5, 4)

arnau
Sep 6 '07 #13
Here's my solution, for what it's worth:

#!/usr/bin/env python

import os

input = open("test.txt", "r")

counter = 0
fileNum = 0
fileName = ""

def newFileName():

global fileNum, fileName
while os.path.exists(fileName) or fileName == "":
fileNum += 1
x = "%0.5d" % fileNum
fileName = "%s.tmp" % x

return fileName
for line in input:

if (fileName == "") or (counter == 5):
if fileName:
output.close()
fileName = newFileName()
counter = 0
output = open(fileName, "w")

output.write(line)
counter += 1

output.close()
Sep 6 '07 #14
Shawn Milochik wrote:
On 9/5/07, ma********@gmail.com <ma********@gmail.comwrote:
>I have a text source file of about 20.000 lines.
>From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.
Maybe (untested):

def read5Lines(f):
L = f.readline()
while L :
yield (L,f.readline(),f.readline(),f.readline(),f.readli ne())
L = f.readline()

in = open('C:\YourFile','rb')
for fileNo, fiveLines in enumerate(read5Lines(in)) :
out = open('c:\OutFile'+str(fileNo), 'wb')
out.writelines(fiveLines)
out.close()

or something similar? (notice that in the last output file you may have
a few (4 at most) blank lines)


Sep 7 '07 #15
On Sep 5, 5:17 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
On Sep 5, 1:28 pm, Paddy <paddy3...@googlemail.comwrote:
On Sep 5, 5:13 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
I have a text source file of about 20.000 lines.>From this file, I like to write the first 5 lines to a new file. Close
that file, grab the next 5 lines write these to a new file... grabbing
5 lines and creating new files until processing of all 20.000 lines is
done.
Is there an efficient way to do this in Python?
In advance, thanks for your help.
If its on unix: use split.
If its your homework: show us what you have so far...
- Paddy.

Paddy,

Thanks for making me aware of the (UNIX) split command (split -l 5
inFile.txt), it's short, it's fast, it's beautiful.

I am still wondering how to do this efficiently in Python (being kind
of new to it... and it's not for homework).

-- Martin.

I am still wondering how to do this in Python (being new to Python)
If this was a code golf challenge, a decent entry (146 chars) could
be:

import itertools as it
for i,g in it.groupby(enumerate(open('input.txt')),lambda(i,_ ):i/
5):open("output.%d.txt"%i,'w').writelines(s for _,s in g)

or a bit less cryptically:

import itertools as it
for chunk,enum_lines in it.groupby(enumerate(open('input.txt')),
lambda (i,line): i//5):
open("output.%d.txt" % chunk, 'w').writelines(line for _,line
in enum_lines)
George

Sep 7 '07 #16
On Sep 7, 3:50 am, George Sakkis <george.sak...@gmail.comwrote:
On Sep 5, 5:17 pm, "malibus...@gmail.com" <malibus...@gmail.com>
wrote:
If this was a code golf challenge,
I'd choose the Unix split solution and be both maintainable as well as
concise :-)

- Paddy.

Sep 7 '07 #17

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Rare Book School | last post: by
8 posts views Thread by Imran | last post: by
2 posts views Thread by Dave Williamson | last post: by
8 posts views Thread by Eddie Suey | last post: by
4 posts views Thread by Alexis Gallagher | last post: by
6 posts views Thread by ivan.perak | last post: by
9 posts views Thread by =?Utf-8?B?QnJpYW4gQ29vaw==?= | last post: by
1 post views Thread by Xah Lee | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.