User vs. Crawler - ASP / Active Server Pages

Gomez

Hi,
Is there a way to know if a session on my web server is from an actual user or an automated crawler.

please advise.
G

Jul 19 '05 #1

Subscribe Reply

2227

Ray at

Perhaps by looking at the user-agent that is making the request, although I
can't say for certain that crawling software doesn't send browser-looking
user-agent info, or that this isn't possible if the crawler software writer
wants it to. That would be pretty snakey though, IMO.

Load a page with this code. Note the http_user_agent variable,
specifically.

<%
For each x in Request.serverV ariables
Response.Write x & " = " & Request.Serverv ariables(x)
Response.Write "<hr>"
Next
%>

Ray at work

"Gomez" <an*******@disc ussions.microso ft.com> wrote in message
news:68******** *************** ***********@mic rosoft.com...

Hi,
Is there a way to know if a session on my web server is from an actual user or an automated crawler.
please advise.
G

Jul 19 '05 #2

Evertjan.

"Gomez" <an*******@disc ussions.microso ft.com> wrote in message
news:68******** *************** ***********@mic rosoft.com...

Hi, Is there a way to know if a session on my web server
is from an actual user or an automated crawler.
Ray at <%=sLocation% > [MVP] wrote on 18 feb 2004 in
microsoft.publi c.inetserver.as p.general: Perhaps by looking at the user-agent that is making the request,
although I can't say for certain that crawling software doesn't send
browser-looking user-agent info, or that this isn't possible if the
crawler software writer wants it to.

Yes

Google puts "googlebot in Request.ServerV ariables("HTTP_ USER_AGENT")

More:

1
using
r = Request.ServerV ariables("HTTP_ REFERER")

you can check for:

"64.68." crawler googlebot.com
"216.239." crawler googlebot.com
"209.237.23 7." Alexa
"209.237.23 8." Alexa
"62.69.162. " ingrid.ilse.nl
"216.39." AltaVista
etc., etc.

2
In hindsight, if a Request.ServerV ariables("HTTP_ REFERER") single IP
looks at many pages in a timeframe surpassing the maximum
vingerleftmousc lick and if every page is a seperate session, it is
probably eiter a crawlerbot or [if there is also a strange supect
querystring] a hacker.
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)

Jul 19 '05 #3

Similar topics

2660

custom user agent

by: studio | last post by:

I have a simple crawler as part of a website. It uses file() to copy HTML content from a web page on another website and reformat it into a new HTML page using a template. Is there a simple way of having the crawler supply a custom HTTP user agent header to identify itself to the website being crawled? I don't want to alter the php.ini file, I need to set the user agent value from PHP if possible. Thanks

PHP

1097

C# Crawler and performance (speed of crawling)

by: Benjamin Lefevre | last post by:

I am currently developping a web crawler, mainly crawling mobile page (wml, mobile xhtml) but not only (also html/xml/...), and I ask myself which speed I can reach. This crawler is developped in C# using multithreading and HttpWebRequest. Actually my crawler is able to download and crawl pages at the speed of around 5 pages per second. It's running on a development machine with 512Mb Ram and a shared ADSL-connection (2Mbits). Is it...

.NET Framework

2510

crawler pool

by: Steve Ocsic | last post by:

Hi, I've coded a basic crawler where by you enter the URL and it will then crawl the said URL. What I would like to do now is to take it one step further and do the following: 1. pick up the url's I would like to crawl from a database and pass them to the crawler. Once the crawler has crawled the website I would then like to put a flag against it so that the url is not processed for a certain period of time.

C# / C Sharp

1525

Why crawler don't get to my aspx page unless the Work Process is started?

by: Nicolas | last post by:

I need HELP!!!!! The crawler (Google or other) don't index my web site unless the web site is currently visited If there is nobody visiting those .aspx page therefor activating the aspnet no crawler is going throught the site I play with the robots file the meta tag etc. Also played with the crawler class but no success Sub Application_BeginRequest(ByVal sender As Object, ByVal e As EventArgs)

ASP.NET

3730

Determine if User Agent is a crawler?

by: Bill | last post by:

Is there a fast way, with Classic ASP, to determine if a user agent is a search engine spider? I know that ASP.NET has Request.Browser.Crawler, I'm looking to see if classic ASP has something built in, or do I have to look for strings in the user agent. Thanks!

ASP / Active Server Pages

5730

Using Request.Browser.Crawler - is it reliable?

by: Bill | last post by:

Has anyone used/tested Request.Browser.Crawler ? Is it reliable, or are there false positives/negatives? Thanks!

ASP.NET

5910

web crawler in python or C?

by: abhinav | last post by:

Hi guys.I have to implement a topical crawler as a part of my project.What language should i implement C or Python?Python though has fast development cycle but my concern is speed also.I want to strke a balance between development speed and crawler speed.Since Python is an interpreted language it is rather slow.The crawler which will be working on huge set of pages should be as fast as possible.One possible implementation would be...

Python

3975

Is my web crawler being blocked?

by: mh121 | last post by:

I am trying to write a web crawler (for academic research purposes) that grabs the number of links different websites/domain names have from other websites, as listed on Google (for example, to get the number of websites linking to YouTube, you could type into Google 'Link:YouTube.com' and get 11,100). I have a list of websites in a spreadsheet and would like to be able to output the number of links for each website in the sheet. When I run...

Python

4093

Web crawler on python

by: sonich | last post by:

I need simple web crawler, I found Ruya, but it's seems not currently maintained. Does anybody know good web crawler on python or with python interface?

Python

9643

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

10147

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...

Online Marketing

10087

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

8971

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

7496

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

6737

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

5380

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...

Networking - Hardware / Configuration

4046

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

3645

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP