Hi,
I am trying to write a BOT to read the search results from Google.
When I read from the www.google.com, the code works fine, loads in
the web page. When I try to load in a url with the search results,
http://www.google.com/search?hl=en&q= ted', I get a web page that says
I do not have permissions. Is theree a way around this, or is Google
just to smart????
the program looks like this
import urllib
print "-------- Google Web Page --------"
print urllib.urlopen( 'http://www.google.com//').read()
print "-------- Google Search Web Page --------"
print urllib.urlopen( 'http://www.google.com/search?
hl=en&q=ted').r ead() 1  988 
On Wed, 20 Aug 2008 05:42:34 -0700 (PDT), te*******@gmail .com wrote:
the web page. When I try to load in a url with the search results,
http://www.google.com/search?hl=en&q= ted', I get a web page that says
I do not have permissions. Is theree a way around this, or is Google
just to smart????
Try to imitate the web browser. Add 'User-Agent' (with add_header
method) to your http request. If it won't help, try to add more
browser-specific variables to your headers. Also, take a look at
mechanize and its Browser class: http://wwwsearch.sourceforge.net/mechanize/
FYI and AFAIK, google doesn't allow to use their search engine
in this way. They even block certain IP addresses it it's constantly
abusing the search engine with too many requests.
--
Regards,
Wojtek Walczak, http://tosh.pl/gminick/ This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Bill Sneddon |
last post by:
Has anyone used googles API who would be will to share
a simple example. I have been wanting to play around with
SOAP for a while and this looks like a place to start.
I am going to mess with it in ASP when our IT guys set it up
for me.
I like Python but have not done anything with SOAP yet.
|
by: iaesun |
last post by:
i'm trying to write a batch that will perform a complex task using
set-based selects instead of a row-based cursor. let me know if you can
help me figure out how.
description of what i'm trying to do:
there is TABLE1, TABLE2, and TABLE3
i want to select each row from TABLE1, do some analysis on the data of
that row, and then perform an insert of some data into TABLE2, and some
|
by: Dica |
last post by:
i've used the sample code from msdn to create an encyption/decryption
assembly as found here:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnnetsec/html/SecNetHT10.asp
i'm able to encrypt and then decrypt data okay as in the following code:
// encrypt the data //
Encryptor enc = new Encryptor(EncryptionAlgorithm.TripleDes);
byte key = Encoding.ASCII.GetBytes("0123456789012345");
|
by: jonnylbluejeans |
last post by:
I've read everywhere to use the following:
objDS.Tables("Results").Columns(0).ColumnMapping = MappingType.Hidden
objDS.AcceptChanges()
dgNotes.DataSource = objDS.Tables("Results")
dgNotes.DataBind()
|
by: Guadala Harry |
last post by:
This has to be easy...
I have a DataSet stored in the Session object. I simply need to read it
later and get the following error:
Cannot implicitly convert type 'object' to 'System.Data.DataSet'
From a code-behind module of an ASPX page, how do I read the contents of a
DataSet that is stored in the Session?
Thanks in advance.
| | |
by: eBob.com |
last post by:
I want to read a web page. That is, I have a URL and I want to read
the HTML and parse it and put the info into a data base. I've never
done any TCP/IP programming.
Thanks to a recent post here I am aware of the System.Web.*
namespaces, but I've looked through some of them and they all seem to
be server-side oriented. I need the kind of code used by the
client/browser side.
Thanks, Bob
|
by: Jim Anderson |
last post by:
This is my first attempt at XML documentation.
I'm trying to get started with docbook so I can put a set
of documentation into docbook tags. I'm using 'XML In A
Nutshell" and "DocBook The Definitive Guide", both of which
are a bit outdated already.
I have a simple file that parses, but when I read it into
Netscape or Konqueror, I do not get the results that I would
hope for.
|
by: tedpottel |
last post by:
Hi,
My program reads as follows
import urllib
print "-------- Google Web Page --------"
print urllib.urlopen('http://www.google.com//').read()
print "-------- Google Search Web Page --------"
|
by: tedpottel |
last post by:
Hi,
I am trying to install the mechanize lib so I can use python to do
webbrowseing.
First I set up easy_install
When I ran the script, it download the files ok, then I got these
error messages
sun is not reganized as a internal command
I did a sercah on sun.* and the sercah came up empty, am I missing
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
| | |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
