using HTTP Digest auth with arbitrary HTTP methods?

[John Reese]

Hello there. I've run into some missing functionality with HTTP Digest
authentication in the 2.3 library and I was wondering if I'm just
missing something.

Missing functionality the first: urllib2

1a. You can add "handlers" to your opener indicating that you want to
use HTTP Digest auth. This is nice way to handle it, but I don't
see any way to use a custom verb in your URLOpener -- it always
uses either GET or POST depending on whether you provided data.
Is there any way to specify an arbitrary method? This would allow
urllib2 to be used to write WebDAV clients.

1b. HTTPDigestAuthHandler is initialized with an HTTPPasswordMgr
object, which unfortunately deals in cleartext passwords. Digest
authentication can be computed using only a hash of username,
password, and realm; it would be nice if there was an alternate
version of HTTPPasswordMgr that let you deal in hashes instead of
or in addition to plaintext passwords.

Missing functionality the second: httplib.

2a. httplib.HTTPConnection lets you execute arbitrary HTTP methods
with arbitrary headers and data; this is the missing functionality
in 1a above. However, you have to deal with following redirects
and authentication and so forth yourself. Is there any way to use
the flexibility of
HTTPConnection.request(method, url[, body[, headers]])
with the convenience of the chains of urllib2 handlers?
The upshot is what I'm trying to do is write a WebDAV-using DAV client
library, and I almost have everything I need to do it; the problem is I
can't find an easy way to do digest authentication for arbitrary HTTP
methods. WebDAV (RFC 2518), for those not familiar, is an extension to
HTTP that defines some new methods and settles the semantics of some
existing but rarely implemented HTTP methods (PUT and DELETE, for
example) to define something similar to a file system. It's intended
for things like letting a group of people author a web site.

[John Reese]

In comp.lang.python, [i] wrote:

Hello there. I've run into some missing functionality with HTTP Digest
authentication in the 2.3 library and I was wondering if I'm just
missing something.

Missing functionality the first: urllib2

1a. You can add "handlers" to your opener indicating that you want to
use HTTP Digest auth. This is nice way to handle it, but I don't
see any way to use a custom verb in your URLOpener -- it always
uses either GET or POST depending on whether you provided data.
Is there any way to specify an arbitrary method? This would allow
urllib2 to be used to write WebDAV clients.

1b. HTTPDigestAuthHandler is initialized with an HTTPPasswordMgr
object, which unfortunately deals in cleartext passwords. Digest
authentication can be computed using only a hash of username,
password, and realm; it would be nice if there was an alternate
version of HTTPPasswordMgr that let you deal in hashes instead of
or in addition to plaintext passwords.

Well, I figured out a workaround, but it requires changing urllib2.py
since the bugs are in base classes.

I instead copied it (to urllib3.py) and made the following changes:
a. in AbstractDigestAuthHandler.get_authorization, call
req.get_method() instead of req.has_data() and 'POST' or 'GET'
(python has a ternary operator, who knew)
b. in AbstractHTTPHandler.do_open, call req.get_method instead of the
hard-coded if-logic which is the same as that in req.get_method

Both of these seem like bugs in urllib2.

Then I overrode urllib2.Request and made it possibly to set the method,
and then passed an instance of my custom Request class (the one that
shouldn't have to exist, since Request should allow method to be set
explicitly) to OpenerDirector.open().

I'd like to see these changes make it into the standard library -- after
being vetted by whoever's in charge of urllib2. Anybody know who I
should talk to?

[John J. Lee]

John Reese <jt*@ofb.net> writes:

In comp.lang.python, [i] wrote: [...] I instead copied it (to urllib3.py) and made the following changes:
a. in AbstractDigestAuthHandler.get_authorization, call
req.get_method() instead of req.has_data() and 'POST' or 'GET'
(python has a ternary operator, who knew)
(Re ternary operator: Everybody who read this list at certain times in
the past is painfully aware of that fact, and of precisely why it's
not quite true, and of all the syntax alternatives for real ternary
conditionals that will never be part of Python ;-)

b. in AbstractHTTPHandler.do_open, call req.get_method instead of the
hard-coded if-logic which is the same as that in req.get_method

Both of these seem like bugs in urllib2.
Yup, bugs both.

Then I overrode urllib2.Request and made it possibly to set the method,
and then passed an instance of my custom Request class (the one that
shouldn't have to exist, since Request should allow method to be set
explicitly) to OpenerDirector.open().

I'd like to see these changes make it into the standard library -- after
being vetted by whoever's in charge of urllib2. Anybody know who I
should talk to?

Nobody is really in charge: just go ahead and submit a patch. Drop me
an email when you do, and I'll try to review it. The only reason
urllib2 doesn't already do arbitrary HTTP methods is that nobody has
spent the time to think carefully if a .set_method() really is the
right way to do it, then followed through with the work needed to get
a patch applied.

As always, a precondition for change is that somebody thinks something
through carefully, writes tests, documentation, patch and submits all
three to the SF patch tracker with a brief explanation like the one
you give above.

BTW, Greg Stein started work on adding the stuff you need at the
httplib level (as module httpx). He seems too busy to finish it, but
see modules httpx and davlib (one or both are in the Python CVS
sandbox). He thinks httplib is a better place for DAV than urllib2,
and he should know. But go ahead and fix urllib2 anyway... :-)
John

[John Reese]

On 03 Jan 2005 18:11:06 +0000, John J. Lee <jj*@pobox.com> wrote:

(Re ternary operator: Everybody who read this list at certain times in
the past is painfully aware of that fact, and of precisely why it's
not quite true, and of all the syntax alternatives for real ternary
conditionals that will never be part of Python ;-)
Yeah, I ran across the PEP after I posted this. Sorry to bring up a
sore subject....
Nobody is really in charge: just go ahead and submit a patch. Drop me
an email when you do, and I'll try to review it. The only reason
urllib2 doesn't already do arbitrary HTTP methods is that nobody has
spent the time to think carefully if a .set_method() really is the
right way to do it, then followed through with the work needed to get
a patch applied.
Patch id #1095362 on Sourceforge. It turns out one of the bugs had
already been fixed in CVS -- the digest bug was the only one left, so
this is a one-line fix.

As always, a precondition for change is that somebody thinks something
through carefully, writes tests, documentation, patch and submits all
three to the SF patch tracker with a brief explanation like the one
you give above.

BTW, Greg Stein started work on adding the stuff you need at the
httplib level (as module httpx). He seems too busy to finish it, but
see modules httpx and davlib (one or both are in the Python CVS
sandbox). He thinks httplib is a better place for DAV than urllib2,
and he should know. But go ahead and fix urllib2 anyway... :-)

These files are 2 and 3 years old. He's probably right that an
interface more like httplib would be more appropriate for a DAV client;
urllib2 is more about generalizing open() to use URLs than fully
exposing the protocol. But urllib2 does have all those convenient
handler-chains, and a good DAV client needs to handle a lot of those
same situations, so it might be a good idea to at least find a way to
share the handler classes.