By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
432,086 Members | 1,875 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 432,086 IT Pros & Developers. It's quick & easy.

writing a web client

P: n/a
I want to write a program which will automatically login to my ISPs
website, retrieve data and do some processing. Can this be done? Can
you point me to any example python programs which do similar things?

Regards,
Ajar

Jul 29 '05 #1
Share this Question
Share on Google+
7 Replies


P: n/a

Ajar wrote:
I want to write a program which will automatically login to my ISPs
website, retrieve data and do some processing. Can this be done? Can
you point me to any example python programs which do similar things?

Regards,
Ajar


Very easily. Have a look at my article on the ``urllib2`` module.

http://www.voidspace.org.uk/python/articles.shtml#http

You may need to use ClientCookie/cookielib to handle cookies and may
have to cope with BASIC authentication. There are also articles about
both of these as well.

If you want to handle filling in forms programattically then the module
ClientForm is useful (allegedly).

Best Regards,

Fuzzyman
http://www.voidspace.org.uk/python

Jul 29 '05 #2

P: n/a
fuzzy's urllib2 info is excellent. The other way peopel snarf stuff
over HTTP and FTP is using 'wget' or 'libcurl'. THere's

http://pycurl.sourceforge.net/

Jul 29 '05 #3

P: n/a
Jay
thats pretty cool, could someone post a example program of a python
web-based program?

Jul 29 '05 #4

P: n/a
Jay
thats pretty cool, could someone post a example program of a python
web-based program?

Jul 29 '05 #5

P: n/a
"Fuzzyman" <fu******@gmail.com> writes:
Ajar wrote:
I want to write a program which will automatically login to my ISPs
website, retrieve data and do some processing. Can this be done? Can
you point me to any example python programs which do similar things?

Regards,
Ajar


Very easily. Have a look at my article on the ``urllib2`` module.

http://www.voidspace.org.uk/python/articles.shtml#http

You may need to use ClientCookie/cookielib to handle cookies and may
have to cope with BASIC authentication. There are also articles about
both of these as well.

If you want to handle filling in forms programattically then the module
ClientForm is useful (allegedly).


The last piece of the puzzle is BeautifulSoup. That's what you use to
extract data from the web page.

For instance a lot of web pages listing data have something like this
on it:

<table>
....
<tr><th>Item:</th><td>Value</td></tr>
....
</table>

You can extract value from such with BeautifulSoup by doing something like:

soup.fetchText('Item:')[0].findParent(['td', 'th']).nextSibling.string

Where this checks works for the item being in either a td or th tag.

Of course, I recommend doing things a little bit more verbosely. In my
case, I'm writing code that's expected to work on a large number of
web pages with different formats, so I put in a lot of error checking,
along with informative errors.

links = table.fetchText(name)
if not links:
raise BadTableMatch, "%s not found in table" % name
td = links[0].findParent(['td', 'th'])
if not td:
raise BadmatchTable, "td/th not a parent of %s" % name
next = td.nextSibling
if not next:
raise BadTableMatch, "td for %s has no sibling" % name
out = get_contents(next)
if not out:
raise BadTableMatch, "no value string found for %s" % name
return out

BeautifulSoup would raise exceptions if the conditions I check for are
true and I didn't check them - but the error messages wouldn't be as
informative.

Oh yeah - get_contents isn't from BeautifulSoup. I ran into cases
where the <td> tag held other tags, and wanted the flat text
extracted. Couldn't find a BeautifulSoup method to do that, so I wrote:

def get_contents(ele):
"""Utility function to return all the text in a tag."""

if ele.string:
return ele.string # We only have one string. Done
return ''.join(get_contents(x) for x in ele)

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 29 '05 #6

P: n/a
Here:
http://diveintopython.org/http_web_s...html#oa.divein

Here:
Cookbook rel 2, and:
http://aspn.activestate.com/ASPN/Coo...Python?kwd=Web

Here:other really good py intros which cover 2.2 / urllib (maybe 2.3, I
don't have them with me), but examples should all not give deprecation
warnings: Practical Py (Hetland), Text Processing (Mertz).

Jul 30 '05 #7

P: n/a
EP
Prabahar wrote:
I want to know difference between
Python-cgi and Perl-cgi and also I want
to which one is efficient from the performance.

The difference between a cgi program written in Perl and a cgi program written in Python is the choice of programming language. Both work quite well.You probably want to use whichever programming language you like best, and that can only be discovered from writing programs in both.

In terms of "efficient" (efficiency) , are you talking about how much memory the program uses, how fast the program executes, or how much time it takes to write and debug the program? There are differences between the two in these regards, but I do not have any reliable information that quantifiesthose differences, and the differences are likely to vary greatly depending on the nature and length of the program, and how efficiently written the code is.

I think, generally, if you really like programming in Perl, there is probably no reason not to use it (at least for typical/short cgi programs) - it has been the default choice for years; conversely, if you really like programming in Python, you will probably be happy to tackle any challenges associated with using it for cgi.
Notes:
Perl cgi execution is generally offered by all hosts that offer cgi; Pythoncgi execution is offered by fewer hosts (but some of those hosts are really great).

With either language, if execution "performance" is a concern, you may wantto use something along the lines of mod_python or mod_perl which embed a persistent interpreter in your web server. This lets you avoid the overhead of starting an external interpreter and avoids the penalty of Perl/Python start-up time, giving faster time to execution.

Was this at all the information you were looking for? Why do I think you may have intended a very technical question? If so, you may have to be a little more specific about the sort of differences you are interested in understanding.
Eric
Jul 30 '05 #8

This discussion thread is closed

Replies have been disabled for this discussion.