BIG successes of Lisp (was ...)

[mike420]

In the context of LATEX, some Pythonista asked what the big
successes of Lisp were. I think there were at least three *big*
successes.

a. orbitz.com web site uses Lisp for algorithms, etc.
b. Yahoo store was originally written in Lisp.
c. Emacs

The issues with these will probably come up, so I might as well
mention them myself (which will also make this a more balanced
post)

a. AFAIK Orbitz frequently has to be shut down for maintenance
(read "full garbage collection" - I'm just guessing: with
generational garbage collection, you still have to do full
garbage collection once in a while, and on a system like that
it can take a while)

b. AFAIK, Yahoo Store was eventually rewritten in a non-Lisp.
Why? I'd tell you, but then I'd have to kill you :)

c. Emacs has a reputation for being slow and bloated. But then
it's not written in Common Lisp.

Are ViaWeb and Orbitz bigger successes than LATEX? Do they
have more users? It depends. Does viewing a PDF file made
with LATEX make you a user of LATEX? Does visiting Yahoo
store make you a user of ViaWeb?

For the sake of being balanced: there were also some *big*
failures, such as Lisp Machines. They failed because
they could not compete with UNIX (SUN, SGI) in a time when
performance, multi-userism and uptime were of prime importance.
(Older LispM's just leaked memory until they were shut down,
newer versions overcame that problem but others remained)

Another big failure that is often _attributed_ to Lisp is AI,
of course. But I don't think one should blame a language
for AI not happening. Marvin Mins ky, for example,
blames Robotics and Neural Networks for that.

[Paul Rubin]

"Andrew Dalke" <ad****@mindspr ing.com> writes:

In other words, writing the domain language as an S-exp
is a short cut to make it easier on the programmer, and not
on the domain specialist. Unless the domain is programming.
And you know, very few of the examples of writing a domain
specific language in Lisp have been for tasks other than
programming.
Actually in my experience that hasn't been a problem. For example, I
wrote a program that crunched EDI documents. There were hundreds of
different document types each with its own rudimentary syntax. We had
a big 3-ring binder containing a printed copy of the ANSI standard
that had a semi-formal English description the syntax of each of these
documents. My program had an embedded Lisp interpreter and worked by
letting you give it the syntax of each document type as a Lisp
S-expression. The stuff in the S-expression followed the document
description in the printed EDI standard pretty closely. I typed in
the first few syntax specs and then was able to hand off the rest to a
non-programmer, who was able to see pretty quickly how the
S-expressions worked and code the rest of them. I think that the
version of the system we actually shipped to customers still had the
S-expression syntax specs buried in its guts somewhere, but my memory
about that is hazy.

The instructions about how to process specific documents were also
entered as Lisp programs at first. That let us very quickly determine
the semantic features we wanted in processing scripts, even though we
knew that our customers wouldn't tolerate Lisp. Once we had the
semantics figured out, we were able to design a language that
superficially looked like an unholy marriage of Basic and Cobol. We
wrote a Yacc script that parsed that language and built up
S-expressions in memory, and then eval'd them with the Lisp
interpreter.

Do people write any significant amount of code in the Python parse
tree syntax?

Peter Norvig talks about this some in his Python/Lisp comparison page:

http://www.norvig.com/python-lisp.html

Basically the Python AST structure is awful, but you arrange your life
so you don't have to deal with it very much.
There's very little evidence that Lisp is significantly better
than Python (or vice versa) for solving most problems.
It's close enough that it's a judgement call to decide which
is more appropriate.

There's one area where Lisp absolutely rules, which is providing a way
to write down complicated data structures without much fuss. These
days, XML is used as a Bizarro cousin of Lisp S-expressions in all
kinds of applications for similar purposes. The EDI program I
mentioned earlier was not originally intended to have an embedded
interpreter. I typed in some EDI syntax specs as S-expressions just
to have something to work with. I then wrote something like a Lisp
reader to read the S-expressions. I found myself then writing a Lisp
printer to debug the Lisp reader. Having a reader and printer it was
then entirely natural to add an eval and gc. The result became a
fairly important product in the EDI world for a time.

[Ville Vainio]

Some people (academics) are paid for being clever. Others (engineers) Ville Vainio http://www.students.tut.fi/~vainio24 ^^^^^^^^

Tee hee! :)

Yes, I have an account at a university. I prefer to use it instead of
that of my ISP (because ISP's come and go) or my work account (to
avoid associating my company with any of my opinions, which I think
should be a standard policy.. also, they don't provide web publishing
space for obvious reasons).

--
Ville Vainio http://www.students.tut.fi/~vainio24

[Andrew Dalke]

Paul Rubin:

Actually in my experience that hasn't been a problem. For example, I
wrote a program that crunched EDI documents.
Ahh, you're right. I tend to omit business-specific languages
when I think about programming.

I conjecture it would have be about the same amount of work to
do it in Python, based solely on your description, but I defer to
you on it.
superficially looked like an unholy marriage of Basic and Cobol.
heh-heh :)

Do people write any significant amount of code in the Python parse
tree syntax?
Peter Norvig talks about this some in his Python/Lisp comparison page:

http://www.norvig.com/python-lisp.html

Basically the Python AST structure is awful, but you arrange your life
so you don't have to deal with it very much.

His description starts

] Python does not have macros. Python does have access to the
] abstract syntax tree of programs, but this is not for the faint of
] heart. On the plus side, the modules are easy to understand,
] and with five minutes and five lines of code I was able to get this:
] >>> parse("2 + 2")
] ['eval_input', ['testlist', ['test', ['and_test', ['not_test',
['comparison',
] ['expr', ['xor_expr', ['and_expr', ['shift_expr', ['arith_expr', ['term',
] ['factor', ['power', ['atom', [2, '2']]]]], [14, '+'], ['term',
['factor',
] ['power', ['atom', [2, '2']]]]]]]]]]]]]]], [4, ''], [0, '']]

I completely agree. Manipulating Python AST and the parse tree
are not for the faint of heart. However, he's not working with
the AST

import compiler
compiler.parse( "2+2") Module(None, Stmt([Discard(Add((Co nst(2), Const(2))))]))

The code I wrote uses the AST and, while clunky, isn't as bad
as his example suggests.
There's one area where Lisp absolutely rules, which is providing a way
to write down complicated data structures without much fuss. These
days, XML is used as a Bizarro cousin of Lisp S-expressions in all
kinds of applications for similar purposes.

That's an old debate. Here's a counter-response
http://www.prescod.net/xml/sexprs.html

Andrew
da***@dalkescie ntific.com

[james anderson]

one of the ironic, telling things about that exposition, is that the
syntax-error contraposition is not quite right.

Andrew Dalke wrote:

There's one area where Lisp absolutely rules, which is providing a way
to write down complicated data structures without much fuss. These
days, XML is used as a Bizarro cousin of Lisp S-expressions in all
kinds of applications for similar purposes.

That's an old debate. Here's a counter-response
http://www.prescod.net/xml/sexprs.html

it also misses the point, that "abstract" is not "concrete", and the surface
syntax in the input stream is not everything. an issue which "xml" will still
be fighting with long after the last so-encoded bits have long faded into the
aether.

analogies apply.

....

[james anderson]

Andrew Dalke wrote:

Kaz Kylheku:

Ah, but in Lisp, this is commonly done at *compile* time.
Compile vs. runtime is an implementation issue. Doesn't
change expressive power, only performance. Type inferencing
suggests that there are other ways to get speed-ups from
dynamic languages.

Moreover,
two or more domain-specific languages can be mixed together, nested in
the same lexical scope, even if they were developed in complete
isolation by different programmers.

We have decidedly different definitions of what a "domain-specific
language" means. To you it means the semantics expressed as
an s-exp. To me it means the syntax is also domain specific. Eg,
Python is a domain specific language where the domain is
"languages where people complain about scope defined by
whitespace." ;)

that is an inaccurate projection of what "domain-specific" means in a
programming environment like lisp. perhaps it says more about what it would
mean in a programming environemtn like python? if the author would take the
example of one of the recent discussions which flew by here, e.weitz's
cl-interpol, it would be interesting to read how
Yes, one can support Python in Lisp as a reader macro -- but
it isn't done because Lispers would just write the Python out
as an S-exp. But then it wouldn't be Python, because the domain
language *includes*domai n*syntax*.
it exemplifies "writing the [ domain-specific language ] out as an s-exp.

In other words, writing the domain language as an S-exp
is a short cut to make it easier on the programmer, and not
on the domain specialist. Unless the domain is programming.
And you know, very few of the examples of writing a domain
specific language in Lisp have been for tasks other than
programming.

the more likely approach to "python-in-lisp" would be a
reader-macro/tokenizer/parser/translater which compiled the
"python-domain-specific-language" into s-expressions.

....

[Duane Rettig]

"Andrew Dalke" <ad****@mindspr ing.com> writes:

Kaz Kylheku:

Moreover,
two or more domain-specific languages can be mixed together, nested in
the same lexical scope, even if they were developed in complete
isolation by different programmers.

We have decidedly different definitions of what a "domain-specific
language" means.

Probably not; more likely it is just a different emphasis.
To you it means the semantics expressed as
an s-exp. To me it means the syntax is also domain specific. Eg,
Python is a domain specific language where the domain is
"languages where people complain about scope defined by
whitespace." ;)

Your whole article leans heavily toward raising the importance of
syntax. Lispers tend to see it differently. For Common Lispers,
and many other lispers who tend to minimize syntax, if the domain-
specific language already has or is able to have similar syntax
as Lisp, then parsing is analready-solved problem, and one can
just use the CL parser (i.e. read) and move on to other more
important problems in the domain language. But if the syntax
doesn't match, then it really still isn't a big deal; a parser
is needed just as it is in any other language, and one must
(and can) solve that problem as well as the rest of the
domain-specifics. The real question is how quickly you can
finally leave the issue of syntax behind you in your new problem
domain and move on to the problem to solve.

In the early '80s, I did an experiment, using Franz Lisp and Naomi
Sager's (NYU) English Grammar parser from her early 1981 book
"Natural Language Information Processing". I wrote a parser
for English out of her BNF and Franz Lisp's character macros and
the lisp reader. When I joined Franz Inc, I was able to port the
parser to Common Lisp with a little cheating (CL doesn't define
infix-macros like Franz Lisp does, so I had to redefine read-list
in order to make the ' macro work well with the lexicon).

[Unfortunately, I can't release this parser, because in my
correspondences with her, Dr. Sager made it clear that although
the sentences and descriptions in the book are not copyrighted,
the BNF and restrictions are. So although you can see the BNF
nodes reresented in a tree below, I won't define the terms;
you'll have to get the book for that...]

The whole point of my experiment was not to write a parser or
to try to parse English, but to show how powerful Lisp's parser
already is. With an input of

"John's going to Mary's house."

for example, with neither John nor Mary being present in the
lexicon, the parser is able to provide the following analysis
(note that the parser was written as a stand-alone program used
in a fashion similar to a scripting language, with no actual
interactive input from the user except via pipe from stdin):

Franz Lisp, Opus 38.89+ plus English Grammar Parser version 0
-> nil
-> nil
-> t
-> t
-> sentence =
(|John's| going to |Mary's| house |.|)
form = <sentence>
revised sentence =
(|John's| going to Mary is house |.|)
form = <sentence>
revised sentence =
(|John's| going to Mary has house |.|)
form = <sentence>
revised sentence =
(John is going to |Mary's| house |.|)
form = <sentence>
found parse 1
1 1 <sentence>
2 2 <introducer>
4 2 <center>
5 3 <assertion>
6 4 <sa>
8 4 <subject>
9 5 <nstg>
10 6 <lnr>
11 7 <ln>
12 8 <tpos>
18 8 <qpos>
20 8 <apos>
22 8 <nspos>
24 8 <npos>
26 7 <nvar>
27 8 <namestg>
28 9 <lnamer>
29 10 <lname>
34 10 n --------------> John
35 10 <rname>
37 7 <rn>
39 4 <sa>
41 4 <tense>
42 5 <null>
43 4 <sa>
45 4 <verb>
47 5 <lv>
49 5 <vvar>
50 6 tv --------------> is
53 4 <sa>
55 4 <object>
56 5 <objectbe>
57 6 <vingo>
58 7 <lvsa>
60 7 <lvingr>
61 8 <lv>
63 8 ving --------------> going
64 8 <rv>
66 9 <rv1>
67 10 <pn>
69 11 <lp>
71 11 p --------------> to
72 11 <nstgo>
73 12 <nstg>
74 13 <lnr>
75 14 <ln>
76 15 <tpos>
77 16 <lnamesr>
78 17 <lname>
83 17 ns --------------> |Mary's|
84 15 <qpos>
86 15 <apos>
88 15 <nspos>
90 15 <npos>
92 14 <nvar>
93 15 n --------------> house
94 14 <rn>
98 7 <sa>
100 7 <object>
101 8 <nullobj>
102 7 <rv>
104 7 <sa>
106 4 <rv>
108 4 <sa>
110 2 <endmark>
111 3 |-.-| --------------> |.|
revised sentence =
(John is going to Mary is house |.|)
form = <sentence>
revised sentence =
(John is going to Mary has house |.|)
form = <sentence>
revised sentence =
(John has going to |Mary's| house |.|)
form = <sentence>
revised sentence =
(John has going to Mary is house |.|)
form = <sentence>
revised sentence =
(John has going to Mary has house |.|)
form = <sentence>
(no more parses count= 1)
->
A more complex structured sentence, which Sager gave in her book,
was "the force with which an isolated heart beats depends on
the concentration of calcium in the medium which surrounds it."
which also was parsed correctly, though I won't show the parse
tree for it here because it is long. I did have trouble with
conjunctions, because they were not covered fullly in her book
and involve splicing copies of parts of the grammar together,
and there are a number of "restrictio ns" (pruning and
well-formedness tests specific to some of the BNF nodes that help
to find the correct parse) which she did not describe in her book.

Again, the point is that syntax and parsing are already-solved
problems., and even problems that don't on the surface look like
problems naturally solved with the lisp reader can come fairly
close to being solved with very little effort. Perhaps we can
thus move a little deeper into the problem space a little faster.

--
Duane Rettig du***@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: [phone]; Fax: [phone]

[Brian Kelley]

Matthew Danish wrote:

On Wed, Oct 22, 2003 at 12:30:01PM -0400, Brian Kelley wrote:

Your two examples do completely different things and the second is
written rather poorly as f might might not exist in the finally block.
It certainly won't if the file doesn't exist.

A better comparison would (*might*, I haven't used lisp in a while) be:

(with-open-file (f filename :direction :output :if-exists :supersede)
(format f "Here are a couple~%of test data lines~%")) => NIL

if os.path.exists( filename):
f = open(filename, 'w')
print >> f, "Here are a couple of test data lines"

How are these in any way equivalent? Pascal posted his example with
try...finally and f.close () for a specific reason. In your Python
example, the file is not closed until presumably the GC collects the
descriptor and runs some finalizer (if that is even the case).

The file is closed when the reference count goes to zero, in this case
when it goes out of scope. This has nothing to do with the garbage
collector, just the reference counter. At least, that's the way I
understand, and I have been wrong before(tm). The upshot is that it has
worked in my experience 100% of the time and my code is structured to
use (abuse?) this. How is this more difficult?

The difference here, as I see it, is that if an exception happens then
the system has to wait for the garbage collector to close the file. In
both examples there was no exception handling after the fact (after the
file is closed). The macro, then, allows execution to continue with the
closed file while the python version stops execution in which case the
file is closed anyway. (unless it is in a another thread of execution
in which the file is closed when it goes out of scope)

In either case one still needs to write handling code to support the
failure as this is most likely application specific. Using macros as a
default handler seems very appropriate in lisp. In python, as I
mentioned in the model-centric view I would create a new file object to
support better handling of file-i/o and failures and hence abstract away
the "error-prone" styles you mentioned.

Now, macros really shine when they also use the local scope. However,
unless a macro is actually doing this I see no real difference between
creating a wrapper for an object and a macro:

f = FileSafeWrapper (open(...))

(with-file-open (f ...)

Except that the macros you are describing are part of the common
distribution. I have to write my own FileSafeWrapper for now...
http://www.lispworks.com/reference/H...with-open-file

So in fact, you pointed out a bug in Pascal's Python example, and one
that is easy to make. All this error-prone code is abstracted away by
WITH-OPEN-FILE in Lisp.

This is a good thing, naturally. But the examples you have given are
completely do-able (in one form or another) in python as it currently
stands, either by creating a wrapper around a file object that can
properly close down on errors or what not. In fact, this might be a
better abstraction in some cases. Consider:

(with-file-that-also-outputs-to-gui ... )
(with-file-that-also-outputs-to-console ...)

to

(with-open-file (f ...

in this case, f is supplied by some constructor that wraps the file to
output to the gui or standard i/o and is passed around to various
functions. Which is the better solution?

I'm not saying that lisp can't do this, it obviously can, but macros
might not be the appropriate solution to this problem ( they certainly
aren't in python ;) )

p.s. I really do enjoy programming in lisp, it was my second programming
language after fortran 77.

[Matthew Danish]

On Thu, Oct 23, 2003 at 10:53:21AM -0400, Brian Kelley wrote:

The file is closed when the reference count goes to zero, in this case
when it goes out of scope. This has nothing to do with the garbage
collector, just the reference counter. At least, that's the way I
understand, and I have been wrong before(tm). The upshot is that it has
worked in my experience 100% of the time and my code is structured to
use (abuse?) this. How is this more difficult?
I would see this as a dependence on an implementation artifact. This
may not be regarded as an issue in the Python world, though. (Are you
sure refcounting is used in all Python implementations ? It is not that
great of a memory management technique except in certain situations).
One of the arguments against using finalizers to deallocate resources is
that it is unpredictable: a stray reference can keep the resource (and
maybe lock) open indefinitely.

This is not to say that Python couldn't achieve a similar solution to
the Lisp one. In fact, it could get quite nearly there with a
functional solution, though I understand that it is not quite the same
since variable bindings caught in closures are immutable. And it would
probably be most awkward considering Python's lambda.
The difference here, as I see it, is that if an exception happens then
the system has to wait for the garbage collector to close the file. In
both examples there was no exception handling after the fact (after the
file is closed). The macro, then, allows execution to continue with the
closed file while the python version stops execution in which case the
file is closed anyway. (unless it is in a another thread of execution
in which the file is closed when it goes out of scope)
I'm not sure I understand this paragraph. The macro only executes the
body code if the file is successfully opened. The failure mode can be
specified by a keyword argument. The usual HANDLER-CASE and
HANDLER-BIND can be used to handle conditions with or without unwinding
the stack (you could fix and continue from a disk full error, for
example). If the stack is unwound out of the macro, by a condition
(abnormally), then there is an attempt to restore the state of the
filesystem (method probably dependent on the :if-exists parameter). If
control exits normally, the file is closed normally.
In either case one still needs to write handling code to support the
failure as this is most likely application specific. Using macros as a
default handler seems very appropriate in lisp.
Not sure what `using macros as a default handler' means.
This is a good thing, naturally. But the examples you have given are
completely do-able (in one form or another) in python as it currently
stands, either by creating a wrapper around a file object that can
properly close down on errors or what not. In fact, this might be a
better abstraction in some cases. Consider:

(with-file-that-also-outputs-to-gui ... )
(with-file-that-also-outputs-to-console ...)

to

(with-open-file (f ...

in this case, f is supplied by some constructor that wraps the file to
output to the gui or standard i/o and is passed around to various
functions. Which is the better solution?

I'm not saying that lisp can't do this, it obviously can, but macros
might not be the appropriate solution to this problem ( they certainly
aren't in python ;) )

The WITH-OPEN-FILE macro is not really an example of a macro that
performs something unique. It is, I find, simply a handy syntactic
abstraction around something that is more complicated than it appears at
first. And, in fact, I find myself creating similar macros all the time
which guide the use of lower-level functions. However, that doesn't
mean, for example, that I would try to defeat polymorphism with macros
(WITH-OPEN-FILE happens to be a very often used special case). I would
write the macro to take advantage of that situation.

--
; Matthew Danish <md*****@andrew .cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian. org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."

[Jock Cooper]

Matthew Danish <md*****@andrew .cmu.edu> writes:

On Wed, Oct 22, 2003 at 09:52:42AM -0700, Jock Cooper wrote:

One of the important differences is that MY-FUNC is lexically isolated
from the environment where WITH-OPEN-FILE appears. The macro version
does not suffer this; and it is often convenient for the code block
in the WITH-OPEN-FILE to access that environment.

(call-with-open-file
#'(lambda (stream)
...)
"somefile"
:direction :input)

WITH-OPEN-FILE happens to be one of those macros which doesn't require
compile-time computation, but rather provides a convenient interface to
the same functionality as above.

--

Right, if the function is specified as a lambda inside the call then it
*would* have access to the surrounding lexical space.

The recent thread on macros has got me thinking more carefully about
when I use macros, and if I could use functions instead.

[Tayss]

Brian Kelley <bk*****@wi.mit .edu> wrote in message news:<3f******* **************@ senator-bedfellow.mit.e du>...

The file is closed when the reference count goes to zero, in this case
when it goes out of scope. This has nothing to do with the garbage
collector, just the reference counter. At least, that's the way I
understand, and I have been wrong before(tm). The upshot is that it has
worked in my experience 100% of the time and my code is structured to
use (abuse?) this. How is this more difficult?

As I understand, you're arguing that it's ok to let Python's
refcounter automagically close the file for you.

Please read this:
http://groups.google.com/groups?hl=e...%40alcyone.com
Erik Max Francis explains that expecting the system to close files
leads to brittle code. It's not safe or guaranteed.

After learning Python, people write this bug for months, until they
see some article or usenet post with the try/finally idiom. This
idiom isn't obvious from the docs; the tutorial doesn't say how
important closing a file is. (I was lucky enough to already know how
exceptions could bust out of code.)

Now, I love Python, but this really is a case where lots of people
write lots of potentially hard-to-reproduce bugs because the language
suddenly stopped holding their hands. This is where it really hurts
Python to make the tradeoff against having macros. The tradeoff may
be worth it, but ouch!