473,289 Members | 1,959 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Mechanoid Web Browser - Recording Capability

I am trying to find a way to sign onto my Wall Street Journal account
(http://online.wsj.com/public/us) and automatically download various
financial pages on stocks and mutual funds that I am interested in
tracking. I have a subscription to this site and am trying to figure
out how to use python, which I have been trying to learn for the past
year, to automatically login and capture a few different pages.
I have mastered capturing web pages on non-password sites, but am
struggling otherwise and have been trying to learn how to program the
Mechanoid module (http://cheeseshop.python.org/pypi/mechanoid) to get
past the password protected site hurdle.

My questions are:
1. Is there an easier way to grab these pages from a password protected
site, or is the use of Mechanoid a reasonable approach?
2. Is there an easy way of recording a web surfing session in Firefox
to see what the browser sends to the site? I am thinking that this
might help me better understand the Mechanoid commands, and more easily
program it. I do a fair amount of VBA Programming in Microsoft Excel
and have always found the Macro Recording feature a very useful
starting point which has greatly helped me get up to speed.

Thanks for your help/insights.
Seymour

Sep 16 '06 #1
7 2926
"Seymour" <se************@gmail.comwrites:
I am trying to find a way to sign onto my Wall Street Journal account
(http://online.wsj.com/public/us) and automatically download various
financial pages on stocks and mutual funds that I am interested in
tracking. I have a subscription to this site and am trying to figure
[...]
My questions are:
1. Is there an easier way to grab these pages from a password protected
site, or is the use of Mechanoid a reasonable approach?
This is the first time I heard of anybody using mechanoid. As the
author of mechanize, of which mechnoid is a fork, I was always in the
dark about why the author decided to fork it (he hasn't emailed
me...).

I don't know if there's any activity on the mechanoid project, but I'm
certainly still working on mechanize, and there's an active mailing list:

http://wwwsearch.sourceforge.net/

https://lists.sourceforge.net/lists/...search-general

2. Is there an easy way of recording a web surfing session in Firefox
to see what the browser sends to the site? I am thinking that this
might help me better understand the Mechanoid commands, and more easily
program it. I do a fair amount of VBA Programming in Microsoft Excel
and have always found the Macro Recording feature a very useful
starting point which has greatly helped me get up to speed.
With Firefox, you can use the Livehttpheaders extension:

http://livehttpheaders.mozdev.org/
The mechanize docs explain how to turn on display of HTTP headers that
it sends.
Going further, certainly there's at least one HTTP-based recorder for
twill, which actually watches your browser traffic and generates twill
code for you (twill is a simple language for functional testing and
scraping built on top of mechanize):

http://twill.idyll.org/

http://darcs.idyll.org/%7Et/projects/scotch/doc/
That's not an entirely reliable process, but some people might find it
helpful.

I think there may be one for zope.testbrowser too (or ZopeTestBrowser
(sp?), the standalone version that works without Zope) -- I'm not
sure. (zope.testbrowser is also built on mechanize.) Despite the
name, I'm told this can be used for scraping as well as testing.

I would imagine that it would be fairly easy to modify or extend
Selenium IDE to emit mechanize or twill or zope.testbrowser (etc.)
code (perhaps without any coding, I used too many Firefox Selenium
plugins and now forget which had which features). Personally I would
avoid using Selenium itself to actually automate tasks, though, since
unlike mechanize &c., Selenium drags in an entire browser, which
brings with it some inflexibility (though not as bad as in the past).
It does have advantages though: most obviously, it knows JavaScript.
John
Sep 17 '06 #2
"Seymour" <se************@gmail.comwrites:
[...]
struggling otherwise and have been trying to learn how to program the
Mechanoid module (http://cheeseshop.python.org/pypi/mechanoid) to get
past the password protected site hurdle.

My questions are:
1. Is there an easier way to grab these pages from a password protected
site, or is the use of Mechanoid a reasonable approach?
[...]

Again, can't speak for mechanoid, but it should be straightforward
with mechanize (simplifiying one of the examples from the URL below):
http://wwwsearch.sourceforge.net/mechanize/

br = Browser()
br.add_password("http://example.com/protected/", "joe", "password")
br.set_debug_http(True) # Print HTTP headers.
br.open("http://www.example.com/protected/blah.html")
print br.response().read()
John
Sep 17 '06 #3
"Seymour" <se************@gmail.comwrites:
I am trying to find a way to sign onto my Wall Street Journal account
(http://online.wsj.com/public/us) and automatically download various
financial pages on stocks and mutual funds that I am interested in
tracking. I have a subscription to this site and am trying to figure
out how to use python, which I have been trying to learn for the past
year, to automatically login and capture a few different pages.
[...]

Just to add: It's quite possible that site has an "no scraping"
condition in their terms of use. It seems standard legal boilerplate
on commercial sites these days. Not a good thing on the whole, I tend
to think, but you should be aware of it.
John
Sep 17 '06 #4
Thanks John!
Lots of great leads in your post that I am busy looking at. I did try
one program, MaxQ, that records web surfing. It seems to work great.
I have looked at all of your leads and plan to give them all a try.
BTW, I am not sure how I came accross Mechanoid before Mechanize, but I
did and started to study that. Somehow I had the notion that
Mechanize was a Pearl script.
Thanks again,
Seymour

John J. Lee wrote:
"Seymour" <se************@gmail.comwrites:
I am trying to find a way to sign onto my Wall Street Journal account
(http://online.wsj.com/public/us) and automatically download various
financial pages on stocks and mutual funds that I am interested in
tracking. I have a subscription to this site and am trying to figure
out how to use python, which I have been trying to learn for the past
year, to automatically login and capture a few different pages.
[...]

Just to add: It's quite possible that site has an "no scraping"
condition in their terms of use. It seems standard legal boilerplate
on commercial sites these days. Not a good thing on the whole, I tend
to think, but you should be aware of it.
John
Sep 18 '06 #5
"Seymour" <se************@gmail.comwrites:
Somehow I had the notion that Mechanize was a Pearl script.
mechanize the Python module started as a port of Andy Lester's Perl
module WWW::Mechanize (in turn based on Gisle Aas' libwww-perl), and
on some very high level has "the same" conceptual interface, but most
of the details (internal structure, features and bugs ;-) are
different to LWP and WWW::Mechanize due to the integration with
urllib2, httplib and friends, and with my own code. Most parts of the
code are no longer recognisable as having originated in LWP (and of
course, lots of it *didn't* originate there).
John

Sep 19 '06 #6
"Seymour" <se************@gmail.comwrites:
[...]
one program, MaxQ, that records web surfing. It seems to work great.
[...]

There are lots of such programs about (ISTR twill used to use MaxQ for
its recording feature, but I think Titus got rid of it in favour of
his own code, for some reason). How useful they are depends on the
details of what you're doing: the information that goes across HTTP is
on a fairly low level, so e.g., most obviously, you may need to be
sending a session ID that varies per-request. Those programs usually
have some way of dealing with that specific problem, but you may run
into other problems that have the same origin. Don't let me put you
off it gets your job done, but it's good to be a bit wary: All current
web-scraping approaches using free software suck in one way or
another.
John

Sep 19 '06 #7
You can try SWExplorerAutomation (SWEA) (http:\\webunittesting.com).
It works very well with the password protected sites. SWEA is .Net API,
but you can use IronPython to access it.

Seymour wrote:
I am trying to find a way to sign onto my Wall Street Journal account
(http://online.wsj.com/public/us) and automatically download various
financial pages on stocks and mutual funds that I am interested in
tracking. I have a subscription to this site and am trying to figure
out how to use python, which I have been trying to learn for the past
year, to automatically login and capture a few different pages.
I have mastered capturing web pages on non-password sites, but am
struggling otherwise and have been trying to learn how to program the
Mechanoid module (http://cheeseshop.python.org/pypi/mechanoid) to get
past the password protected site hurdle.

My questions are:
1. Is there an easier way to grab these pages from a password protected
site, or is the use of Mechanoid a reasonable approach?
2. Is there an easy way of recording a web surfing session in Firefox
to see what the browser sends to the site? I am thinking that this
might help me better understand the Mechanoid commands, and more easily
program it. I do a fair amount of VBA Programming in Microsoft Excel
and have always found the Macro Recording feature a very useful
starting point which has greatly helped me get up to speed.

Thanks for your help/insights.
Seymour
Sep 20 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

25
by: Ryan Stewart | last post by:
I'm working on a project to collect web application usage statistics. What are the recommended ways of detecting whether a browser is JavaScript enabled and/or capable? Obviously I can write a...
3
by: Prabhat | last post by:
Hi All, I have IE6.0 SP1 and I executed one asp file with the below script to check the browser capability but see the result below the script...Copied from MSDN :) <% Set bc =...
6
by: Dovelet | last post by:
Hi all, I would like to write DOS program to change the MS Windows Sound Recording source. When I run it with the parameter, it will change the recording source as follow: C:\> abc.exe...
3
by: ms | last post by:
Hi Everyone, You all would be aware of the fact that we boast about .net supporting multiple web browsers. I hope we have all experienced that our screen layouts look different in every other...
4
by: Paul W | last post by:
Hi - can someone point me to info on the issues/resolutions of supporting the safari browser? To help me understand, if I was developing pages in say FrontPage, what attributes would I set for...
1
by: Sakharam Phapale | last post by:
Hi All, I am developing an application like sound recorder. While recording if there is a silence for more than given time (say 5 seconds), Recording should be paused.
1
by: Tom Yee | last post by:
I would like to write a Windows Service that can communicate with any open browser windows that an interactive user may be running. The service itself does not need to have a UI. The company...
5
by: Pipp | last post by:
Hi, this simple code works well to add a bookmark on IE, but it doesn't work on Firefox <a href="javascript:window.external.AddFavorite('http:// www.mysite.com','My site is cool');"> Can...
0
by: suchiate | last post by:
Hi All, Would like to have a discussion regarding the current and near future mobile technologies on whether which technologies have reached a certain capability for doing video streaming and live...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.