stdin or optional fileinput

[the.theorist]

I was writing a small script the other day with the following CLI
prog [options] [file]*

I've used getopt to parse out the possible options, so we'll ignore
that part, and assume for the rest of the discussion that args is a
list of file names (if any provided).

I used this bit of code to detect wether i want stdinput or not.

if len(args)==0:
args = [ sys.stdin ]

Now in my main loop I've written:

for file in args:
for line in open( file ):
#do stuff

The probelm occurs when I pass no arguments and python trys to open(
sys.stdin ).
open() customarily accepts a pathname, and returns a file object.
But this code is so elegant I thought that maybe, if open were passed
file object it could re-open that file in a new mode (if a mode was
provided different from the current mode) or simply return the file
object it was passed.

Anyhow, what do you guys think? There might be something about my idea
that makes this proposed behavior of open() unexpected, and I just
haven't thought of it.

[Steven Bethard]

the.theorist wrote:

I was writing a small script the other day with the following CLI
prog [options] [file]*

I've used getopt to parse out the possible options, so we'll ignore
that part, and assume for the rest of the discussion that args is a
list of file names (if any provided).

I used this bit of code to detect wether i want stdinput or not.

if len(args)==0:
args = [ sys.stdin ]

Now in my main loop I've written:

for file in args:
for line in open( file ):
#do stuff

You should probably write:

if not args: # note that len(args) == 0 is repetitively redundant, over
# and over again, in a reiterative manner
files = [sys.stdin]
else:
files = (open(filename) for filename in args)

....

for fileobj in files: # using the name 'file' is a bad idea since it
# shadows the builtin 'file'
for line in fileobj:
# do stuff
STeVe

[the.theorist]

Steven Bethard wrote:

the.theorist wrote:

I was writing a small script the other day with the following CLI
prog [options] [file]*

I've used getopt to parse out the possible options, so we'll ignore
that part, and assume for the rest of the discussion that args is a
list of file names (if any provided).

I used this bit of code to detect wether i want stdinput or not.

if len(args)==0:
args = [ sys.stdin ]

Now in my main loop I've written:

for file in args:
for line in open( file ):
#do stuff

You should probably write:

if not args: # note that len(args) == 0 is repetitively redundant, over
# and over again, in a reiterative manner
files = [sys.stdin]
else:
files = (open(filename) for filename in args)

...

for fileobj in files: # using the name 'file' is a bad idea since it
# shadows the builtin 'file'
for line in fileobj:
# do stuff
STeVe

I'll keep both those in mind for future programs.
my current fix has been

if not args:
args = [ sys.stdin ]
else:
map( open, args )

and then a modification to the main loop, as you proposed.

I thought that one day I might run into a problem opening too many
files though (esp if i used xargs in a pipe). And it just _feels_
better to open and close as I loop over the files list, rather than
open everything at the get go.

OH, i've noticed you used a generator that takes care of that. Except,
the machine I'm working on is currently stuck at Python 2.3. Still,
I'll keep your suggestions in mind for future coding.

[Steve Holden]

the.theorist wrote:

Steven Bethard wrote:

the.theorist wrote:

I was writing a small script the other day with the following CLI
prog [options] [file]*

I've used getopt to parse out the possible options, so we'll ignore
that part, and assume for the rest of the discussion that args is a
list of file names (if any provided).

I used this bit of code to detect wether i want stdinput or not.

if len(args)==0:
args = [ sys.stdin ]

Now in my main loop I've written:

for file in args:
for line in open( file ):
#do stuff
You should probably write:

if not args: # note that len(args) == 0 is repetitively redundant, over
# and over again, in a reiterative manner
files = [sys.stdin]
else:
files = (open(filename) for filename in args)

...

for fileobj in files: # using the name 'file' is a bad idea since it
# shadows the builtin 'file'
for line in fileobj:
# do stuff
STeVe

I'll keep both those in mind for future programs.
my current fix has been

if not args:
args = [ sys.stdin ]
else:
map( open, args )

and then a modification to the main loop, as you proposed.

I thought that one day I might run into a problem opening too many
files though (esp if i used xargs in a pipe). And it just _feels_
better to open and close as I loop over the files list, rather than
open everything at the get go.

Yes, that feels most Pythonic.
OH, i've noticed you used a generator that takes care of that. Except,
the machine I'm working on is currently stuck at Python 2.3. Still,
I'll keep your suggestions in mind for future coding.

Well the nice thing about the generator expression is precisely its
deferral of applying open() to the filename argument until the iterable
element is extracted. But as you observe there's no use trying that in
Python 2.3, so you'll have to adopt a slightly less elegant solution for
now.

The important thing is that you seem to be well in touch with the
various issues, so there's a good chance you won't go wrong.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogspot.com

[Fredrik Lundh]

"the.theorist" wrote:

I used this bit of code to detect wether i want stdinput or not.

if len(args)==0:
args = [ sys.stdin ]

Now in my main loop I've written:

for file in args:
for line in open( file ):
#do stuff

The probelm occurs when I pass no arguments and python trys to open(
sys.stdin ).
open() customarily accepts a pathname, and returns a file object.
But this code is so elegant I thought that maybe, if open were passed
file object it could re-open that file in a new mode (if a mode was
provided different from the current mode) or simply return the file
object it was passed.

so use your own open function:

def myopen(file):
if isinstance(file, basestring):
return open(file)
return file

for file in args:
for line in myopen( file ):
#do stuff
or perhaps

def myopen(file):
if hasattr(file, "read"):
return file
return open(file)

...

or perhaps

if not args:
args = [ "-" ]

def myopen(file):
if file == "-":
return sys.stdin
return open(file)

for file in args:
for line in myopen( file ):
#do stuff

(the "-" = stdin convention is quite common)

</F>

[Peter Otten]

the.theorist wrote:

I'll keep both those in mind for future programs.
my current fix has been

if not args:
****args*=*[*sys.stdin*]
else:
****map(*open,*args*)

and then a modification to the main loop, as you proposed.

I thought that one day I might run into a problem opening too many
files though (esp if i used xargs in a pipe). And it just feels
better to open and close as I loop over the files list, rather than
open everything at the get go.

OH, i've noticed you used a generator that takes care of that. Except,
the machine I'm working on is currently stuck at Python 2.3. Still,
I'll keep your suggestions in mind for future coding.

use

if args:
args = itertools.imap(open, args)
else:
args = [sys.stdin]

to get the same lazy-open behaviour as with the generator expression.
Or have a look at http://docs.python.org/lib/module-fileinput.html.

Peter