By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,400 Members | 903 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,400 IT Pros & Developers. It's quick & easy.

web scrapping - POST and auto-login

P: n/a
please excuse my ignorance of Python. i know enough to be dangerous
(was able however to make Freevo support my FM tuner with a little hacking).

Ultimate goal:
read N bytes from PATTERN on page A into VARIABLE.
POST VARIABLE into TABLE on site B.
email VARIABLE to myself.

what i have to work with:
---------------------------
#!/usr/local/bin/python

from urllib import urlopen
from re import search

URL = "http://wifirouter/WL_WPATable.asp"
pattern = "wl_key1 value='*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
print result.group(1)
---------------------------

I'm trying to update the WEP key on a wireless router via script and
email the results to myself. this will be run once a week. Site A
already generates random keys, so i figure that's a good place to get
VARIABLE from. The example above gives me the current key of the router.
What i'm having trouble with is the key on site A (one i wrote myself
http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi ) provides 128bit
keys by default, but i only want to use 64bit (10 characters). so..
rather than make it select 64bit and all i figure it would be easier to
use the above example for getting the /current/ key, modified it to read
the entire 26 character string from site A, but [1] how do i crop it to
10 characters.
[2] how can i then auto-login to site B (the router's web config via
pop-up dialog box. *not* a form) and then [3] post VARIABLE and save the
changes.

I know this sounds like a lot, but if anyone can provide me with any
example code i can maybe modify to fit, or point me to some good
reference material. all the pages i found on logging into a site
involve ssl and cookies, which, in this case at least, isn't involved here.
If there is any other info i can provide (ie. sections of relevent
HTML) then let me know. any assistance you have to offer is greatly
appreciated.

--
- - james <at> hal-pc.org - -

"Friends don't let friends use Mandrake"
- - - - - - - - - - - - - - -
Sep 20 '05 #1
Share this Question
Share on Google+
9 Replies


P: n/a
"james <at> hal-pc.org" wrote:
the entire 26 character string from site A, but [1] how do i crop it to
10 characters.
[1] still at a loss on this one, but i can get to it later, unless you've got
any ideas.
[2] how can i then auto-login to site B (the router's web config via
pop-up dialog box. *not* a form) and then [3] post VARIABLE and save the
changes.
ok.. this one was stupid easy. should have remembered that.
[2] http://user:password@wifirouter/WL_WPATable.asp
If there is any other info i can provide (ie. sections of relevent
HTML) then let me know. any assistance you have to offer is greatly
appreciated.


argh!
[3]
now i'm trying to get it to find the BUTTON "Save Settings". from what i see in
the code it's labled "save_button", but i only get:
------------------------
....
....
<IgnoreControl(save_button"=<None>)>
<IgnoreControl(cancel"=<None>)>>
Traceback (most recent call last):
File "./wifi.py", line 19, in ?
response = urlopen(form.click("save_button"))
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2628, in click
self._request_class)
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2788, in _click
control = self._find_control(name, type, "clickable", id, None, nr)
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2783, in
_find_control
raise ControlNotFoundError("no control matching "+description)
ClientForm.ControlNotFoundError: no control matching name 'save_button', kind
'clickable'
---------------------------

relevent HTML:
---------------------------
....
....
<TD
height=25>&nbsp;<script>Capture(wlansec.key1)</script>&nbsp;:&nbsp;</TD>
<TD height=25>&nbsp;<INPUT size=36 name=wl_key1 value='5465a6576b'></TD>
....
....
<script>document.write("<input type=button name=save_button" + " value=\"" +
sbutton.save + "\" onClick=to_submit(this.form)>");</script>&nbsp;
....
....
---------------------------

any ideas on making it "see" the button for what it is in amongst the garbage?
I think /it's/ getting confused here.
Sep 20 '05 #2

P: n/a
"james <at> hal-pc.org" wrote in
news:43***********************@news.hal-pc.org:
"james <at> hal-pc.org" wrote:
the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.


[1] still at a loss on this one, but i can get to it later,
unless you've got any ideas.


strings are slicable:
s='a string of characters'
s[:10] 'a string o' s[-10:] 'characters'


As far as your other problem, it might make sense(I don't know...) to
just generate the post request for saving the new password yourself
and never bother with parsing/filling the form.

max
Sep 20 '05 #3

P: n/a
"james <at> hal-pc.org" wrote:
I'm trying to update the WEP key on a wireless router via script and
email the results to myself. this will be run once a week.


Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the
more low-level ClientForm by the same author. This will be _much_
easier than doing it by hand.

-Jonathan

Sep 20 '05 #4

P: n/a
Jonathan Ellis wrote:
"james <at> hal-pc.org" wrote:
I'm trying to update the WEP key on a wireless router via script and
email the results to myself. this will be run once a week.

Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the
more low-level ClientForm by the same author. This will be _much_
easier than doing it by hand.

-Jonathan


what's the fun in that though? :)
actually, i was just looking at that site, which is where i came across
ClientForm. i think it's getting confused.. picks up "save_button", but lists
it as "<IgnoreControl(save_button"=<None>)>"

the stray " is from:
<script>document.write("<input type=button name=save_button" + " value=\".....

if i can just get it to see that correctly, then i think i can make it work from
there. any ideas on that?
Sep 20 '05 #5

P: n/a
Max Erickson wrote:
the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.

strings are slicable:


The only reason i've gotten this far is a basic understanding of syntax and
programming in general :)
care to enlighten me a little on how to do that?
As far as your other problem, it might make sense(I don't know...) to
just generate the post request for saving the new password yourself
and never bother with parsing/filling the form.


i'm not quite sure how to generate it from scratch.
thanks for the input/advice!
Sep 20 '05 #6

P: n/a
"james <at> hal-pc.org" wrote in
news:43***********************@news.hal-pc.org:
Max Erickson wrote:
the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.


strings are slicable:


The only reason i've gotten this far is a basic understanding of
syntax and programming in general :)
care to enlighten me a little on how to do that?


I did. If you have some text in VARIABLE, and do something like:
substring=VARIABLE[:10], substring will be the first 10 characters of
VARIABLE.
Sep 20 '05 #7

P: n/a
Max Erickson wrote:
strings are slicable:


wep64 = wep128[0:10]

sorry for the stupid question. thanks.
Sep 20 '05 #8

P: n/a

What i have so far:
--------------------
#!/usr/local/bin/python

from urllib import urlopen
from ClientForm import ParseResponse
from re import search

URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
pattern = "<!-- marker -->*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
wep64 = result.group(1)[:10]
#print wep64

forms = ParseResponse(urlopen("http://un:pw@wifirouter/WL_WPATable.asp"))
form = forms[0]
form["wl_key1"] = wep64
#print form
response = urlopen(form.click("save_button"))

URL = "http://un:pw@wifirouter/WL_WPATable.asp"
pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
doc2 = urlopen(URL).read()
result2 = search(pattern2, doc2)
#print result2.group(1)
---------------------------

the only problems now are getting it to see the "save_button" correctly or
generate the POST from scratch as Max suggested.. that will take some more
hunting to figure that out... any better phrases to search for maybe?
and email it out. can probably "borrow" some things from Freevo on calling
another program, just passing the variable to it as a PIPE is going to be the
hard part. thinking of using sendmail (not a problem), but how to get the data?
i'll keep looking. thanks again both of you.
Sep 20 '05 #9

P: n/a

What i have so far:
--------------------
#!/usr/local/bin/python

from urllib import urlopen
from ClientForm import ParseResponse
from re import search

URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
pattern = "<!-- marker -->*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
wep64 = result.group(1)[:10]
#print wep64

forms = ParseResponse(urlopen("http://un:pw@wifirouter/WL_WPATable.asp"))
form = forms[0]
form["wl_key1"] = wep64
#print form
response = urlopen(form.click("save_button"))

URL = "http://un:pw@wifirouter/WL_WPATable.asp"
pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
doc2 = urlopen(URL).read()
result2 = search(pattern2, doc2)
#print result2.group(1)
---------------------------

the only problems now are getting it to see the "save_button" correctly or
generate the POST from scratch as Max suggested.. that will take some more
hunting to figure that out... any better phrases to search for maybe?
and email it out. can probably "borrow" some things from Freevo on calling
another program, just passing the variable to it as a PIPE is going to be the
hard part. thinking of using sendmail (not a problem), but how to get the data?
i'll keep looking. thanks again both of you.
Sep 20 '05 #10

This discussion thread is closed

Replies have been disabled for this discussion.