Session Timeout Webpages

[Tofubia]

Hi,

This is kinda new to me but I'm desparate for some help on python programming!!

I'm currently writing a scrape for a site, but unfortunately, everytime I copy and paste the link into python and use python to access that page, it returns a "session timeout" error...I'm not sure what the language is behind the site, perhaps Javascript because "jsession" was associated with the session ID naming convention.

I've tried using urllib2.urlopen('website') to get the session id and then incorporate this into the link then pass it through again, but same problem exists.

I've also used win32com.dispatch(internetExplorer.application) to simulate a user browsing through the webpage, but to no success.

Can anyone out there help or suggest methods to overcome this problem?

Many thanks in advance!

Cheers

[dazzler]

do you get the same timeout message with all www-sites?
maybe firewall is blocking your python application?

[Tofubia]

do you get the same timeout message with all www-sites?
maybe firewall is blocking your python application?

I am pretty sure that Python has access to all ports.

Take the following link:

http://www.yellowpages.com.au/browse/listingsSearch.do?sbs=&headingCode=29025&sortByAlp habetical=false&sortByClosestMatch=false&sortByDis tance=false&region=australia&sortByDetail=true&saf eLocationClue=All+States&stateId=9&currentLetter=

By the time you open this link, the session has already expired....you can try yourself as well.....go into any category from the yellow pages site, then use Python to open the same link you are viewing on IE...you can instanteously get a response saying the link has expired....now...copy and paste the exact same link into IE again and go....I think you get what I mean by now....

This is rather annoying...as I need to iterate through the pages....

[dazzler]

it might be quite hard to browse that page with python

you must use cookies with python (so the session id will be saved into cookie), I have no idea how to do it, maybe there's a simple solution available, google it ;D

try this one: if you disable your browser's cookies and go to that yellowpages sites, surf there a little bit and you will see that now that the session cookie cannot be saved into cookie because it's disable, the session id is now in your url address, something like this "jsessionid=70D2F96D52EA769F6A6DDF9FC4593277.yp305 _22122"

so you have two options:
use cookies with python modules
somehow include that session id into your url address