469,609 Members | 2,189 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,609 developers. It's quick & easy.

Submitting forms over HTTPS with mechanize

Rex
Hello,

I am working on an academic research project where I need to log in to
a website (www.lexis.com) over HTTPS and execute a bunch of queries to
gather a data set. I just discovered the mechanize module, which seems
great because it's a high-level tool. However, I can't find any decent
documentation for mechanize apart from the docstrings, which are
pretty thin. So I just followed some other examples I found online, to
produce the following:

baseurl = 'http://www.lexis.com/'
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-Agent', 'Firefox')]
br.open(baseurl)
br.select_form(name="formauth")
br["USER_ID"]="my_user_id"
br["PASSWORD"]="my_password"
result = br.submit()

This code hangs at br.submit(), and I can't tell what I'm doing wrong.
Typically I would inspect the HTTP data with an HTTP debugging proxy
(Fiddler), but I guess since this is HTTPS I can't do that. Any
glaring errors in my code?

By the way, does anyone have suggestions for Python modules that I
should use instead of mechanize (and that are sufficiently easy)? If
mechanize fails, I might try modifying some similar Perl code a friend
sent me that logs into lexis.com.

Thanks so much,

Rex
Sep 4 '08 #1
2 4717
Rex wrote:
Hello,

I am working on an academic research project where I need to log in to
a website (www.lexis.com) over HTTPS and execute a bunch of queries to
gather a data set. I just discovered the mechanize module, which seems
great because it's a high-level tool. However, I can't find any decent
documentation for mechanize apart from the docstrings, which are
pretty thin. So I just followed some other examples I found online, to
produce the following:

baseurl = 'http://www.lexis.com/'
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-Agent', 'Firefox')]
br.open(baseurl)
br.select_form(name="formauth")
br["USER_ID"]="my_user_id"
br["PASSWORD"]="my_password"
result = br.submit()

This code hangs at br.submit(), and I can't tell what I'm doing wrong.
Typically I would inspect the HTTP data with an HTTP debugging proxy
(Fiddler), but I guess since this is HTTPS I can't do that. Any
glaring errors in my code?

By the way, does anyone have suggestions for Python modules that I
should use instead of mechanize (and that are sufficiently easy)? If
mechanize fails, I might try modifying some similar Perl code a friend
sent me that logs into lexis.com.

Thanks so much,

Rex
I've used mechanize quite successfully but others have suggested Twill
http://twill.idyll.org/. It seems to be at least documented.

-Larry
Sep 4 '08 #2
Rex
On Sep 3, 10:17*pm, Larry Bates <larry.ba...@vitalEsafe.comwrote:
Rex wrote:
Hello,
I am working on an academic research project where I need to log in to
a website (www.lexis.com) over HTTPS and execute a bunch of queries to
gather a data set. I just discovered the mechanize module, which seems
great because it's a high-level tool. However, I can't find any decent
documentation for mechanize apart from the docstrings, which are
pretty thin. So I just followed some other examples I found online, to
produce the following:
baseurl = 'http://www.lexis.com/'
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-Agent', 'Firefox')]
br.open(baseurl)
br.select_form(name="formauth")
br["USER_ID"]="my_user_id"
br["PASSWORD"]="my_password"
result = br.submit()
This code hangs at br.submit(), and I can't tell what I'm doing wrong.
Typically I would inspect the HTTP data with an HTTP debugging proxy
(Fiddler), but I guess since this is HTTPS I can't do that. Any
glaring errors in my code?
By the way, does anyone have suggestions for Python modules that I
should use instead of mechanize (and that are sufficiently easy)? If
mechanize fails, I might try modifying some similar Perl code a friend
sent me that logs into lexis.com.
Thanks so much,
Rex

I've used mechanize quite successfully but others have suggested Twillhttp://twill.idyll.org/. *It seems to be at least documented.

-Larry
Thanks for the reply, Larry. I ran my code again and it worked; there
was probably some temporary issue with either my computer or the
server that caused it to hang.
Sep 4 '08 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Display Name | last post: by
2 posts views Thread by Kowalski | last post: by
3 posts views Thread by Hari Sekhon | last post: by
6 posts views Thread by sureshbup | last post: by
reply views Thread by Jett | last post: by
reply views Thread by Solution2021 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.