Web automation with twill

[qwweeeit]

Hi all,
this post is a kind of continuation of my
"Expanding Python as a macro language"

Among the replies [e-mail] directed me to:
http://www.idyll.org/~t/www-tools/twill.html
(a python tool with a language to script "web commands")

I applied twill to a problem I had solved "by hand"
clicking 220 times on a button and saving the data sent
by the server (it is an "asp" file).

In this case web automation is needed!
I need the help of some expert or of the author (Titus Brown).

The asp file is (I am Italian!):
http://www.scienzemfn.unito.it/job_p...o_completo.asp
If you open it you see a button with "Page Down" that must be clicked
to get all the data in chunks.

I used twill (vers. 0.7.3) in interactive mode:
twill-sh -u
http://www.scienzemfn.unito.it/job_p...o_completo.asp
or twill-sh ... and then >>go webaddress

With "show" you see the the list of the html file representing
the 1st chunk (page 1 of 223)
With "showforms" you obtain:
Form #1
## __Name______ __Type___ __ID________ __Value___
PageNo hidden (None) 1
1 Mv submit (None) Page Down
Form #2
## __Name______ __Type___ __ID________ __Value___
PageNo hidden (None) 1
None button (None) None
1 Mv submit (None) Page Down

Using twill you can access the following page and save it by:

fv 1 "Mv" "Page down"
submit
save_html file_name
with >> showforms you obtain:
Form #1
## __Name______ __Type___ __ID________ __Value___
PageNo hidden (None) 2
1 Mv submit (None) Page Down
2 Mv submit (None) Page Up
Form #2
## __Name______ __Type___ __ID________ __Value___
PageNo hidden (None) 2
None button (None) None
1 Mv submit (None) Page Down
2 Mv submit (None) Page Up
Beeing the second page a further button shows up ("Page Up")

But now I get lost... I am not able to load the 3rd page...
In fact repeating the aforementioned sequence: fv ... + submit

the button activated by the "submit" command is "Page Up"!

Any help?

IMHO, twill is great ...expecially if we can make it work
outside from its first use (automatic web testing).
I must also say that it can be called as a module...

Bye.

[Grig Gheorghiu]

You might want to post your question to the twill mailing list. Info
about the list is available at <http://lists.idyll.org/listinfo/twill>

Grig

[qwweeeit]

I solved the problem by myself with the classical
method of newbyes (trial and error).
Bye.

[Michele Simionato]

BTW, O'Reilly just published an article of mines on twill:

http://www.onlamp.com/pub/a/python/2.../03/twill.html

Michele Simionato

[qwweeeit]

Hi Michele,
I taught to be the smartest in town!
But when the experts take the field it is better that us newbies
retire in good order...
Thank you for your article and, with respect to Grig Gheorghiu
(another expert), I must apologize for having be a little rude.

I hadn't discovered in Internet your contributions because I
hardly read the first 10 hits of a search.
I need web automation also for that (in order to elaborate more search
results reducing then my help requests and mainly my useless posts).

By the way, are you aware that C. Titus Brown (twill's author)
tells "peste e corna" of Zope?
Bye.

[Michele Simionato]

qwwwee:

By the way, are you aware that C. Titus Brown (twill's author)
tells "peste e corna" of Zope?

No, but I am not surprised. I also say "peste e corna" of Zope ;)
In general I am an anti-frameworks guy (in good company with
many Pythonistas including Guido).

Michele Simionato

[qwweeeit]

Hi Michele,
I only made the observation about Zope, because I hoped to
hear a different point of view as you are/were involved in
web development using Zope/Plone (as referred in your article...).
Besides that, at a Linux Day, I followed a presentation of
Zope/Plone framework, which stroke me a lot.
Bye.