Folks:
I am screen scraping a large volume of data from Yahoo Finance each
evening, and parsing with Beautiful Soup.
I was wondering if anyone could give me some pointers on how to make
it less obvious to Yahoo that this is what I am doing, as I fear that
they probably monitor for this type of activity, and will soon ban my
IP.
-DE
4  3320 
On 11 May 2007 12:32:55 -0700, di************* *@gmail.com
<di************ **@gmail.comwro te:
Folks:
I am screen scraping a large volume of data from Yahoo Finance each
evening, and parsing with Beautiful Soup.
I was wondering if anyone could give me some pointers on how to make
it less obvious to Yahoo that this is what I am doing, as I fear that
they probably monitor for this type of activity, and will soon ban my
IP.
-DE
So long as you are sending a regular http request, as from a browser,
then they will have no way of knowing. Just keep your queries down to
no more than once every 3-5 seconds and you should be fine. Rotate
your IP, too, if you can.
Dotan Cohen http://lyricslist.com/lyrics/artist_...rmen_eric.html http://what-is-what.com/what_is/eula.html
On May 11, 2:32 pm, different.eng.. .@gmail.com wrote:
Folks:
I am screen scraping a large volume of data from Yahoo Finance each
evening, and parsing with Beautiful Soup.
I was wondering if anyone could give me some pointers on how to make
it less obvious to Yahoo that this is what I am doing, as I fear that
they probably monitor for this type of activity, and will soon ban my
IP.
-DE
Depends on what you're doing exactly. I've done something like this
and it only hits the page once:
URL = 'http://quote.yahoo.com/d/quotes.csv?s=%s &f=sl1c1p2'
TICKS = ('AMZN', 'AMD', 'EBAY', 'GOOG', 'MSFT', 'YHOO')
u = urlopen(URL % ','.join(TICKS) )
for data in u:
tick, price, chg, per = data.split(',')
# do something with data
If you're grabbing all the data in one fell swoop (which is what you
should aim for), then it's harder for Yahoo! to know what you're doing
exactly. And I can't see why they'd care as that is all a browser does
anyway. It's when you hit the site a bunch of times in a short period
of time that sets off the alarms.
Mike
On Fri, 11 May 2007 12:32:55 -0700, different.engin e wrote:
Folks:
I am screen scraping a large volume of data from Yahoo Finance each
evening, and parsing with Beautiful Soup.
I was wondering if anyone could give me some pointers on how to make
it less obvious to Yahoo that this is what I am doing, as I fear that
they probably monitor for this type of activity, and will soon ban my
IP.
Write a virus to hijack tens of thousands of Windows PCs around the world,
and use your army of zombie-PCs to do the screen scraping for you. Each
one only needs to scrape a small amount of data, and Yahoo will have no
way of telling that it is you.
*wink*
--
Steven. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Jonathan Epstein |
last post by:
I would like to perform a more classical type of "screen scraping"
than what most people now associate with this term. I only want to
find all the text on the current screen, and obtain associated screen
coordinates. This probably must be done using OCR.
This need only run on Windows. A fairly-pure Python solution would be
ideal because most of the software which would use this functionality
is also written in Python.
The ideal...
|
by: David Jones |
last post by:
Hi, I'm interested in learning about web scraping/site scraping using
Python. Does anybody know of some online resources or have any modules that
are available to help out. O'Reilly published an interesting book
"Spidering Hacks" which covered some great scraping hacks but it is all
written in Perl. I don't know Perl and don't want to. I'm new to
programing and have been advised to start with Python. So far so good ...
but need some...
|
by: Roland Hall |
last post by:
Am I correct in assuming screen scraping is just the response text sent to
the browser? If so, would that mean that this could not be screen scraped?
function moi() {
var tag = '<a href=';
var tagType1 = '"mail'+'to:', tagType2 = '">', tagType3 = '<\/a>';
var user1 = 'web', user2 = 'master', user3 = '@';
var dom1 = 'danger', dom2 = 'ous', dom3 = 'ly';
var tld = '.us';...
|
by: Jim Giblin |
last post by:
I need to scrape specific information from another website, specifically the
prices of precious metals from several different vendors. While I will
credit the vendors as the data source, I do not want to use the format of
their pages, and want the inforamtion consolidated to a single page of my
design.
I did something like this for a client a couple of years ago in ASP, but it
was complex, and I do not have access to the code. A...
|
by: rachel |
last post by:
Hello,
I am currently contracted out by a real estate agent. He
has a page that he has created himself that has a list of
homes.. their images and data in html format.
He wants me to take this page and reformat it so that it
looks different.
Do I use screen scraping to do this?
Could someone please point me to a good screen scraping
| | |
by: bruce |
last post by:
Hi..
I'm not that familiar with Pythin, but I wasn wondering if there are any
XPath/Python Gurus that I might be able to talk to regarding screen scraping
applications...
Thanks
-Bruce
bedouglas@earthlink.net
|
by: jeffbg123 |
last post by:
Hey,
I am trying to make a bot for a flash game using python. However I am
having some trouble with a screen scraping strategy. Is there an
accepted way to compare a full screenshot with the image that I want
to locate? It is a math based game, so I just have to check what
number, 1-9, appears in the center of the flash game. Is there an
easier method to do this?
Thanks
|
by: bruce |
last post by:
Hi...
got a short test app that i'm playing with. the goal is to get data off the
page in question.
basically, i should be able to get a list of "tr" nodes, and then to
iterate/parse them. i'm missing something, as i think i can get a single
node, but i can't figure out how to display the contents of the node.. nor
how to get the list of the "tr" nodes....
|
by: bruce |
last post by:
Hi Paul...
Thanks for the reply. Came to the same conclusion a few minutes before I saw
your email.
Another question:
tr=d.xpath(foo)
gets me an array of nodes.
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
