473,414 Members | 1,618 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,414 software developers and data experts.

Screen Scraping a web page

I am working on an application to screen scrape information from a web page.
I have the base code working but the problem is I have to login before I can
get the info I need. The page is hosted on my Router. When I go to the IP of
the router I get the following page.

<HTML>
<head>
<meta http-equiv="content-type" content="text/html;charset=iso-8859-1">
<title>Login</title>
</head>

<BODY bgcolor="#f79900">
<form action="LOGIN.HTM" method="post" name="tF">
<input type="hidden" name="page" value="login">
<table border="0" width="100%" height="184" cellspacing="0">
<tr>
<td width="100%" height="103" colspan="2" align="center">
<a href="http://support.speedstream.com"><img border="0"
src="IMAGE/SIELOGOBLACK.JPG" width="270" height="40"></a>
</td>
</tr>
<tr>
<td width="100%" height="19" colspan="2" align="center">
<H2><font face="Arial, Helvetica, sans-serif" color="#FFFFFF">Login
&nbsp;Screen</font></H2>
</td>
</tr>

<tr>
<td width="50%" height="19" align="right">
<font face="Arial, Helvetica, sans-serif" size="2"
color="#FFFFFF">Password&nbsp;&nbsp;&nbsp;:</font></td>
<td width="50%" height="19" align="left">
<INPUT type="password" maxLength=12 size=9 name=pws></td><p>
</tr>
<tr>
<td width="50%" height="19">&nbsp;</td>
<td width="50%" height="19">&nbsp;</td>
</tr>
<tr>
<td width="50%" height="19" align="right">
<INPUT type="submit" value=" Login ">
</td>
<td width="50%" height="19" align="left">
<INPUT class=button onclick=window.close(); type=button value= Cancel >
</td>
</tr>
</table>
</form></BODY>
</HTML>

Using the informaation on this page I have developed the following code for
my application, but ever time I run it I get:

An unhandled exception of type 'System.Net.WebException' occurred in
system.dll

Additional information: The underlying connection was closed: The server
committed an HTTP protocol violation.

My code is as follows:

Imports System.Net

Imports System.IO

Public Class Form1

Inherits System.Windows.Forms.Form

#Region " Windows Form Designer generated code "

Public Sub New()

MyBase.New()

'This call is required by the Windows Form Designer.

InitializeComponent()

'Add any initialization after the InitializeComponent() call

End Sub

'Form overrides dispose to clean up the component list.

Protected Overloads Overrides Sub Dispose(ByVal disposing As Boolean)

If disposing Then

If Not (components Is Nothing) Then

components.Dispose()

End If

End If

MyBase.Dispose(disposing)

End Sub

'Required by the Windows Form Designer

Private components As System.ComponentModel.IContainer

'NOTE: The following procedure is required by the Windows Form Designer

'It can be modified using the Windows Form Designer.

'Do not modify it using the code editor.

Friend WithEvents lblMyIp As System.Windows.Forms.Label

Friend WithEvents txtMyIp As System.Windows.Forms.TextBox

<System.Diagnostics.DebuggerStepThrough()> Private Sub InitializeComponent()

Dim resources As System.Resources.ResourceManager = New
System.Resources.ResourceManager(GetType(Form1))

Me.lblMyIp = New System.Windows.Forms.Label

Me.txtMyIp = New System.Windows.Forms.TextBox

Me.SuspendLayout()

'

'lblMyIp

'

Me.lblMyIp.Location = New System.Drawing.Point(8, 8)

Me.lblMyIp.Name = "lblMyIp"

Me.lblMyIp.Size = New System.Drawing.Size(56, 23)

Me.lblMyIp.TabIndex = 0

Me.lblMyIp.Text = "My Ip:"

'

'txtMyIp

'

Me.txtMyIp.Location = New System.Drawing.Point(64, 8)

Me.txtMyIp.Multiline = True

Me.txtMyIp.Name = "txtMyIp"

Me.txtMyIp.Size = New System.Drawing.Size(376, 248)

Me.txtMyIp.TabIndex = 1

Me.txtMyIp.Text = ""

'

'Form1

'

Me.AutoScaleBaseSize = New System.Drawing.Size(7, 19)

Me.ClientSize = New System.Drawing.Size(448, 262)

Me.Controls.Add(Me.txtMyIp)

Me.Controls.Add(Me.lblMyIp)

Me.Font = New System.Drawing.Font("Times New Roman", 12.0!,
System.Drawing.FontStyle.Regular, System.Drawing.GraphicsUnit.Point,
CType(0, Byte))

Me.Icon = CType(resources.GetObject("$this.Icon"), System.Drawing.Icon)

Me.MaximizeBox = False

Me.MinimizeBox = False

Me.Name = "Form1"

Me.StartPosition = System.Windows.Forms.FormStartPosition.CenterScree n

Me.Text = "MyIpReader"

Me.ResumeLayout(False)

End Sub

#End Region

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles MyBase.Load

txtMyIp.Text = ReadHTMLPage(http://192.168.1.1:88/login.htm)

End Sub

Public Function ReadHTMLPage(ByVal url As String) As String

Dim result As String = ""

Dim strPost As String = "page=login&pws=password"

Dim myWriter As StreamWriter

Dim objRequest As HttpWebRequest = WebRequest.Create(url)

objRequest.Method = "POST"

objRequest.ContentLength = strPost.Length

objRequest.ContentType = "application/x-www-form-urlencoded"

Try

myWriter = New StreamWriter(objRequest.GetRequestStream())

myWriter.Write(strPost)

Catch e As Exception

Return e.Message

Finally

myWriter.Close()

End Try

Dim objResponse As HttpWebResponse = objRequest.GetResponse()

Dim sr As StreamReader

sr = New StreamReader(objResponse.GetResponseStream())

result = sr.ReadToEnd()

sr.Close()

Return result

End Function

End Class

I can't figure out what I am doing wrong here? Any guidance would be
appreciated.
Nov 21 '05 #1
0 3617

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Roland Hall | last post by:
Am I correct in assuming screen scraping is just the response text sent to the browser? If so, would that mean that this could not be screen scraped? function moi() { var tag = '<a href='; var...
3
by: Jim Giblin | last post by:
I need to scrape specific information from another website, specifically the prices of precious metals from several different vendors. While I will credit the vendors as the data source, I do not...
1
by: niv | last post by:
Hello, I would like to screen scrape certain parts of a webpage...how can I do this in asp.net For instance.... a stockticker thats embeded on a webpage.. I dont want the entire page.. I...
4
by: rachel | last post by:
Hello, I am currently contracted out by a real estate agent. He has a page that he has created himself that has a list of homes.. their images and data in html format. He wants me to take...
4
by: Ronald S. Cook | last post by:
I've been asked to extract data from web pages. Given that they are rendered in HTML and not any sort of XML I'm wondering how to go about "scraping" such a web page of data. Can anyone give me...
2
by: Alan Silver | last post by:
Hello, I would like to pull some information off a site that requires a log in. I have a subscription to a premium content site, and I would like to be able to do a few automatic requests...
4
by: onetitfemme | last post by:
Say, people would like to log into their hotmail, yahoo and gmail accounts and "keep an eye" on some text/part of a site .. I think something like that should be out there, since not all sites...
9
by: Knoxy | last post by:
Hi guys, I've got this working but I have issues when there is any kind of c# coding on the page that I'm trying to scrape (pages within my site - its for a print page view basically), I get this...
3
by: WFDGW2 | last post by:
I want to write or obtain C++ code that will scrape text from a dialog box within a poker client, and then record that text somewhere else. What do I do? Thanks.
3
by: bruce | last post by:
Hi... got a short test app that i'm playing with. the goal is to get data off the page in question. basically, i should be able to get a list of "tr" nodes, and then to iterate/parse them....
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.