473,321 Members | 1,622 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,321 software developers and data experts.

Can I automate browsing of this site?

I'm trying to automate this for a mass-processing of GDC number
information (scrapping). I've tried everything I can think of and
every example I've come across to no avail. The root site is:
http://www.dcor.state.ga.us/GDC/Offe...OffQryForm.jsp

posting the following:
http://www.dcor.state.ga.us/GDC/Offe...isclaimer=True

Yields the form I need to fill out to access individual GDC records.
On this page, I need to use option UNO_NO (gdc number) and pass along
an actual GDC number (of which I have thousands to process).

Problem is that the form doesn't post to itself:
<form method="post" action="OffQryRedirector.jsp"

If I try opening a webrequest with:
http://www.dcor.state.ga.us/GDC/Offe...enderId=365172

It just returns me to the main page. :(

Any ideas whatsoever would be extremely appreciated. I've tried
cookies (as best as I understand them), multiple webrequests, setting
autoredirect to true, etc. This doesn't seem like an absurd request,
so I'm hoping I'm missing some piece of understanding that clears up
the automation of this process. Thanks!

Jun 27 '06 #1
1 1442
Hi Branden,

If you get this working, what happens if they change the page? Rename a
field? Rename the page? How will your program handle a change in
javascript validation logic that prevents the page from posting some time
after you finally get it working?

A web service might be much better for you. Try contact the site admin and
see if there is a better way. Will they let you connect directly to their
database?

Are you displaying the HTML that is retrieved from the WebRequest in your
application? Could you just use a WebBrowser control instead?
Problem is that the form doesn't post to itself:
<form method="post" action="OffQryRedirector.jsp"
That shouldn't matter if you pass all the data that it expects.
OffQryRedirector.jsp might do something simple like check the HTTP_REFERER
on POST. I don't think that HTTP_REFERER will be accessible to their server
when using a WebRequest object unless it's added by code but I'm not sure of
that.
If I try opening a webrequest with:
http://www.dcor.state.ga.us/GDC/Offe...enderId=365172

It just returns me to the main page. :(
Does OffQryRedirector.jsp expect just a query string? What about the
required post data?
This doesn't seem like an absurd request,
so I'm hoping I'm missing some piece of understanding that clears up
the automation of this process. Thanks!


Maybe not absurd but maybe not possible as well. Don't rule out the
possibility that it won't work. You may be forced to find another way. Try
my suggestion of contacting the site admin for a web service or data access
ability of some sort. That would probably be your best and only shot.

HTH
Jun 27 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

26
by: Dan Nash | last post by:
Hi guys I have a page that is *supposed* to list the directories on the server. Here's the code... folderspec = server.MapPath("./images/") set fso =...
10
by: Geoff Cox | last post by:
Hello Is it possibel to automate the access time allowed to a folder? I am using .htpasswd etc and would like to give say 0.5 hour access to users so that they can sample documents available on...
5
by: James Johnson | last post by:
Dear C#dex, You can automate a POST and interrogate the resulting HTTP stream using code such as: HttpWebRequest hwreq = (HttpWebRequest)WebRequest.Create(url); hwreq.Method = "POST";...
0
by: Tim Smith | last post by:
Dear All, Uncovered a problem with a home grown site replicator. We have directory browsing enabled to allow our remote clients to "dynamically discover" new files/directories (sites) we have...
1
by: WolfsonNYC | last post by:
Anyone know how to enable Directory Browsing using the Cassini web server on .Net 2.0 ? Right now it says HTTP Error 403 - Forbidden when I go to a folder on my web site. Thanks, JW
2
by: andrea | last post by:
I should make a page that modify the web.config file, appsettings area. For what I know, every modify to config file mean a reboot of the context application, so if a user is browsing on a web...
3
by: samadams_2006 | last post by:
On Yahoo Finance I can automate specific calls to the Web Site by changing the URL to: http://finance.yahoo.com/q?s=msft http://finance.yahoo.com/q?s=ibm http://finance.yahoo.com/q?s=aol I'd...
8
by: =?Utf-8?B?cGF0cmlja2RyZA==?= | last post by:
Hi all! Does anyone know of a way to keep a permanent url throughout a user's browsing? For example, if my web site's address is http://www.mysite.com, I have noticed that some sites keep...
2
by: Joe Kovac | last post by:
Hi! I have a Web Site, where I turned directory browsing off. I want only one sub folder to be browsable: e.g. MyApp/Files/... How do I do that? Any setting in the web.config? Thanks Joe
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.