Python Scripts to logon to websites

[BartlebyScrivener]

New to Python and Programming. Trying to make scripts that will open
sites and automatically log me on.

The following example is from the urllib2 module.

What are "realm" and "host" in this example.

import urllib2
# Create an OpenerDirector with support for Basic HTTP
Authentication. ..
auth_handler = urllib2.HTTPBas icAuthHandler()
auth_handler.ad d_password('rea lm', 'host', 'username', 'password')
opener = urllib2.build_o pener(auth_hand ler)
# ...and install it globally so it can be used with urlopen.
urllib2.install _opener(opener)
urllib2.urlopen ('http://www.example.com/login.html')

Does anyone have a simple example of a script that opens, say, gmail or
some other commonly accessed site that requires a username and password
so that I can see how one is made?

Thanks very much for any help.

rpd

[Peter Hansen]

BartlebyScriven er wrote:

New to Python and Programming. Trying to make scripts that will open
sites and automatically log me on.

The following example is from the urllib2 module.

What are "realm" and "host" in this example.
http://www.ietf.org/rfc/rfc2617.txt probably provides more background
than you want on that topic, but googling for "basic authentication" and
maybe "realm" and/or "host" will find you other sites with less
technically detailed material. The first hit has a little summary
amidst some Apache-specific detail.
Does anyone have a simple example of a script that opens, say, gmail or
some other commonly accessed site that requires a username and password
so that I can see how one is made?

"realm" and "host" are associated with "basic authentication" and not
all sites use that. If the browser pops up a little dialog box of its
own (i.e not some Javascript-triggered thing) and you have to enter your
username and password there, that's probably a "basic auth" (or "digest
auth") site. If you fill that info into a form (as on gmail.com) you
don't want any of that "realm/host" stuff.

I'll leave it to others more expert in this to provide a more directly
useful answer.

-Peter

[Mike Meyer]

"BartlebyScrive ner" <rp*******@gmai l.com> writes:

New to Python and Programming. Trying to make scripts that will open
sites and automatically log me on.
A common enough things to want to do.
The following example is from the urllib2 module.

What are "realm" and "host" in this example.
Host is a domain name that can be mapped to a ip address. Realm is
from HTTP authentication schemes. When the server asks for
authentication, it gives out a "realm" name as well, so that different
parts of the host can use different authentication systems.
Does anyone have a simple example of a script that opens, say, gmail or
some other commonly accessed site that requires a username and password
so that I can see how one is made?

Yes, but its not clear how much good it'll do you. As Peter indicated,
not everyone uses HTTP based authentication. In fact, pretty much
anyone who wants to control how the authentication boxes look (which
seems to be 99% of the people writing web apps, never mind that they
can't really do that) use something other than HTTP-based
authentication. How you go about dealing with such sites depends on
where they put the user name/login information,and how they encode the
fact that you've authenticated as user "xxxx".

So I could show you my script for accessing yahoo. However, it
probably won't work on another site without changes to accomodate the
other site.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Tomi Kyöstilä]

BartlebyScriven er wrote:

New to Python and Programming. Trying to make scripts that will open
sites and automatically log me on. [snip] Does anyone have a simple example of a script that opens, say, gmail or
some other commonly accessed site that requires a username and password
so that I can see how one is made?

I see your example uses HTTP authentication, but I still recommend
checking out mechanoid [1] if you want to access a site with a
form-based login system. The source contains an example that retreives
and sends email through Yahoo.

[1] http://cheeseshop.python.org/pypi/mechanoid/

--
dOb

[BartlebyScrivener]

> but googling for "basic authentication" and

maybe "realm" and/or "host" will find you other sites with less
technically detailed material.
This looks promising, but it'll take me a week to understand it :)

http://www.voidspace.org.uk/python/a...ntication.shtm

Thanks for your help with the search terms.

rpd
l
Peter Hansen wrote: BartlebyScriven er wrote:

New to Python and Programming. Trying to make scripts that will open
sites and automatically log me on.

The following example is from the urllib2 module.

What are "realm" and "host" in this example.

http://www.ietf.org/rfc/rfc2617.txt probably provides more background
than you want on that topic, but googling for "basic authentication" and
maybe "realm" and/or "host" will find you other sites with less
technically detailed material. The first hit has a little summary
amidst some Apache-specific detail.

Does anyone have a simple example of a script that opens, say, gmail or
some other commonly accessed site that requires a username and password
so that I can see how one is made?

"realm" and "host" are associated with "basic authentication" and not
all sites use that. If the browser pops up a little dialog box of its
own (i.e not some Javascript-triggered thing) and you have to enter your
username and password there, that's probably a "basic auth" (or "digest
auth") site. If you fill that info into a form (as on gmail.com) you
don't want any of that "realm/host" stuff.

I'll leave it to others more expert in this to provide a more directly
useful answer.

-Peter

[Peter Hansen]

BartlebyScriven er wrote:

but googling for "basic authentication" and
maybe "realm" and/or "host" will find you other sites with less
technically detailed material.

This looks promising, but it'll take me a week to understand it :)

http://www.voidspace.org.uk/python/a...ntication.shtm

(Minor typo... needs an extra "l" on the end:

http://www.voidspace.org.uk/python/a...tication.shtml
)

By the way, note that neither basic auth nor digest auth provide any
real security, and in fact with basic auth the userid and password are
sent *in cleartext*. For any serious production site these techniques
should probably not be used without additional security measures in
place, such as HTTPS encryption.

-Peter

[BartlebyScrivener]

Thanks, Peter.

Peter Hansen wrote:

BartlebyScriven er wrote:

but googling for "basic authentication" and
maybe "realm" and/or "host" will find you other sites with less
technically detailed material.

This looks promising, but it'll take me a week to understand it :)

http://www.voidspace.org.uk/python/a...ntication.shtm

(Minor typo... needs an extra "l" on the end:

http://www.voidspace.org.uk/python/a...tication.shtml
)

By the way, note that neither basic auth nor digest auth provide any
real security, and in fact with basic auth the userid and password are
sent *in cleartext*. For any serious production site these techniques
should probably not be used without additional security measures in
place, such as HTTPS encryption.

-Peter

[Mike Meyer]

Peter Hansen <pe***@engcorp. com> writes:

By the way, note that neither basic auth nor digest auth provide any
real security, and in fact with basic auth the userid and password are
sent *in cleartext*. For any serious production site these techniques
should probably not be used without additional security measures in
place, such as HTTPS encryption.

To be clear, the HTTP authentication schemes don't provide any
security for the *content* that gets passed back and forth, and they
don't claim to. If someone can intercept that content, they can read
it. For some applications, this is really important. For others, it
doesn't matter at all.

Basic auth doesn't (quite) pass the user name and password in
cleartext. It uses rot-13. For all the protection it provides, it
might as well be cleartext.

Digest passes around md5 sums of varous bits and pieces. While md5 has
been compromised, I don't believe that's happened in a way that
compromises the security of digest auth. The password and username
that pass over the wire are about as secure as they're going to get
without noticably heavier mechanisms than digest auth requires. On the
downside, the server has to have the clear text password available.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/ Independent
WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Peter Hansen]

Mike Meyer wrote:

Peter Hansen <pe***@engcorp. com> writes:

By the way, note that neither basic auth nor digest auth provide any
real security, and in fact with basic auth the userid and password are
sent *in cleartext*. For any serious production site these techniques
should probably not be used without additional security measures in
place, such as HTTPS encryption.
To be clear, the HTTP authentication schemes don't provide any
security for the *content* that gets passed back and forth, and they
don't claim to. If someone can intercept that content, they can read
it. For some applications, this is really important. For others, it
doesn't matter at all.

If someone can see the content, they can also see the userid and
password. If they can see the password, they will (with how most people
operate) now have a userid and password that will work on many other
sites, including possibly someone's banking site, no matter how secure
even the content might be for that site.

Most people on the web are simply too ignorant of security issues for
those of us building systems that require passwords to ignore this
issue. To do so is to endanger the security and privacy of the very
people you are hoping to have as users and customers, which is lazy and
careless (and perhaps in some countries even criminal these days).
Basic auth doesn't (quite) pass the user name and password in
cleartext. It uses rot-13. For all the protection it provides, it
might as well be cleartext.
It's actually base64 encoding, but it amounts to the same thing, as you
say, as cleartext, since it's trivially reversible. The protection is
useless against all but honest people who might otherwise accidentally
see it while looking at packet monitoring dumps or such.
Digest passes around md5 sums of varous bits and pieces. While md5 has
been compromised, I don't believe that's happened in a way that
compromises the security of digest auth. The password and username
that pass over the wire are about as secure as they're going to get
without noticably heavier mechanisms than digest auth requires. On the
downside, the server has to have the clear text password available.

My information about digest was either obsolete or simply wrong, as I
didn't realize it had all the nonce and anti-replay support it appears
to have. (I may have been remembering articles about how much of that
wasn't supported widely at some time in the past, meaning replays were
still quite possible in most cases. No longer sure.) Thanks for the
correction.

In my own opinion, however, requiring that passwords be stored in clear
text on the server is still quite a bad thing to do. I don't think even
system administrators should ever have access to user passwords. But
many people don't seem to agree (or at least, are more than happy to be
lazy rather than diligent in protecting their users' privacy).

-Peter

[Paul Rubin]

Peter Hansen <pe***@engcorp. com> writes:

My information about digest was either obsolete or simply wrong, as I
didn't realize it had all the nonce and anti-replay support it appears
to have. (I may have been remembering articles about how much of that
wasn't supported widely at some time in the past, meaning replays were
still quite possible in most cases. No longer sure.) Thanks for the
correction.
Digest is actually rarely used, since sites with enough security
requirements to make it worthwhile generally use SSL/TLS with either
basic auth, or with some login mechanism implemented by the
application. Actually, HTTP authentication (basic or digest) is not
used all that much in general these days, since nontrivial web apps
generally prefer to do their own authentication. It was more common
in the early days of the web when most pages were static.
In my own opinion, however, requiring that passwords be stored in
clear text on the server is still quite a bad thing to do.

Digest auth, like basic auth, doesn't require storing the cleartext
password; only a hash of the password needs to be stored. See RFC
2617 for details.