472,802 Members | 1,405 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,802 software developers and data experts.

how to browse using urllib2 and cookeilib the correct way

Hi ,

I am using python2.4 "urllib2" and "cookelib".
In line "5" below i provide my credentials to
login into a web site.During the first attempt i "fail",
judging from the output of line "6".
I try again and the second time i succeed,judging
from the output of line "8".

Now using the "twill" module (http://www.idyll.org/~t/www-tools/twill/)
it suceeds in the first attempt(lines "10" to "19").

can anybody explain, in the first case why i need to do two attempts.

1) >>>import urllib2,urllib,cookielib
2) >>>cj = cookielib.CookieJar()
3) >>>opener = urllib2.build_opener( urllib2.HTTPCookieProcessor(cj))
4) >>>data = urllib.urlencode ( { "username" : "user" ,
"password" :"****" } )
5) >>>fp = opener.open(
"https://rhn.redhat.com/rhn/LoginSubmit.do",data )
6) >>>fp.url

'https://rhn.redhat.com/rhn/ReLogin.do?url_bounce=/network/index.pxt'
7) >>>fp = opener.open(
"https://rhn.redhat.com/rhn/LoginSubmit.do",data )
8) >>>fp.url
'https://rhn.redhat.com/network/index.pxt'

#twill module

10) >>>from twill.commands import *
11) >>>go("https://rhn.redhat.com")
12) ==> at https://rhn.redhat.com
13) 'https://rhn.redhat.com'
14) >>>fv("2","username","user")
15) >>>fv("2","password","****")
16) >>>submit()
17) Note: submit is using submit button: name="None", value=" ¬ Sign
In¬ "
18) >>>url('https://rhn.redhat.com/network/index.pxt')
19) 'https://rhn.redhat.com/network/index.pxt'

Thanks
jn
jn

Apr 26 '06 #1
1 1548
jn***@ensim.com writes:
[...]
1) >>>import urllib2,urllib,cookielib
2) >>>cj = cookielib.CookieJar()
3) >>>opener = urllib2.build_opener( urllib2.HTTPCookieProcessor(cj))
4) >>>data = urllib.urlencode ( { "username" : "user" ,
"password" :"****" } )
5) >>>fp = opener.open(
"https://rhn.redhat.com/rhn/LoginSubmit.do",data )
6) >>>fp.url
..url is not a public interface. Use .geturl() .

'https://rhn.redhat.com/rhn/ReLogin.do?url_bounce=/network/index.pxt'
7) >>>fp = opener.open(
"https://rhn.redhat.com/rhn/LoginSubmit.do",data )
8) >>>fp.url
'https://rhn.redhat.com/network/index.pxt'


Dunno. Perhaps because a Refresh is used to handle the login? twill
is based on mechanize, which handles Refresh headers.
John

Apr 26 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: O. Koch | last post by:
Until now, i know that ftplib doesn't support proxies and that i have to use urllib2. But i don't know how to use the urllib2 correct. I found some examples, but i don't understand them. Is...
2
by: Andre Bocchini | last post by:
I'm having some trouble using proxy authentication. I can't figure out how to authenticate with a Squid proxy. I know for a fact the proxy is using Basic instead of Digest for the authentication....
4
by: bmiras | last post by:
I've got a problem using urllib2 to get a web page. I'm going through a proxy using user/password authentification and i'm trying to get a page asking for a HTTP authentification. And I'm using...
0
by: Lee, Dustin | last post by:
I'm using python2.2.2 on redhat linux 7.3 Here's a snippet of what I'm trying to run: # set up proxy import urllib2 proxy_support =...
0
by: jacob c. | last post by:
When I request a URL using urllib2, it appears that urllib2 always makes the request using HTTP 1.0, and not HTTP 1.1. I'm trying to use the "If-None-Match"/"ETag" HTTP headers to conserve...
4
by: Fuzzyman | last post by:
urllib2 (under windows) will auto-detect your proxy settings and use those. Normally that's a good thing (I guess), except when it's not ! How do I switch off this behaviour ? I'm behind a...
0
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's...
3
by: jnair | last post by:
Hi , I am using python2.4 "urllib2" and "cookelib". In line "5" below i provide my credentials to login into a web site.During the first attempt i "fail", judging from the output of line "6"....
0
by: Phoe6 | last post by:
Hi All, I am able to use urlib2 through proxy. I give proxy credentials and use # Set the Proxy Address proxy_ip = "10.0.1.1:80" proxy_user = 'senthil_or' proxy_password_orig='password'
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 2 August 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
linyimin
by: linyimin | last post by:
Spring Startup Analyzer generates an interactive Spring application startup report that lets you understand what contributes to the application startup time and helps to optimize it. Support for...
0
by: erikbower65 | last post by:
Here's a concise step-by-step guide for manually installing IntelliJ IDEA: 1. Download: Visit the official JetBrains website and download the IntelliJ IDEA Community or Ultimate edition based on...
0
by: kcodez | last post by:
As a H5 game development enthusiast, I recently wrote a very interesting little game - Toy Claw ((http://claw.kjeek.com/))。Here I will summarize and share the development experience here, and hope it...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Sept 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
14
DJRhino1175
by: DJRhino1175 | last post by:
When I run this code I get an error, its Run-time error# 424 Object required...This is my first attempt at doing something like this. I test the entire code and it worked until I added this - If...
5
by: DJRhino | last post by:
Private Sub CboDrawingID_BeforeUpdate(Cancel As Integer) If = 310029923 Or 310030138 Or 310030152 Or 310030346 Or 310030348 Or _ 310030356 Or 310030359 Or 310030362 Or...
0
by: lllomh | last post by:
How does React native implement an English player?
0
by: Mushico | last post by:
How to calculate date of retirement from date of birth

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.