By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,663 Members | 2,157 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,663 IT Pros & Developers. It's quick & easy.

Using Httpwebrequest to Submit multipart/form-data

P: n/a
I'm trying to write a program in vb.net to automate filling out a
series of forms on a website. There are three forms I need to
fill out in sequence. The first one is urlencoded. My program is
able to fill that one out just fine.

The second form is multipart/form-data. Unfortunately, I haven't
been able to fill that out in a way that makes the server happy.

I set up a copy of this form at my web site so that I could see
exactly what a browser sends to the server and compare that to
what my program sends. I saw a few mistakes in my program and I
fixed them, but the real web site still rejects my program's
submissions.

As best I can tell, the contents that my program submits are
identical to the contents submitted by a browser. Is there
anything else that might be causing my problem?
Thanks for any help.
--
Greg
----
greg -at- spencersoft -dot- com
Jul 21 '05 #1
Share this Question
Share on Google+
10 Replies


P: n/a
Gregory A Greenman wrote:
I'm trying to write a program in vb.net to automate filling out a
series of forms on a website. There are three forms I need to
fill out in sequence. The first one is urlencoded. My program is
able to fill that one out just fine.

The second form is multipart/form-data. Unfortunately, I haven't
been able to fill that out in a way that makes the server happy.

I set up a copy of this form at my web site so that I could see
exactly what a browser sends to the server and compare that to
what my program sends. I saw a few mistakes in my program and I
fixed them, but the real web site still rejects my program's
submissions.

As best I can tell, the contents that my program submits are
identical to the contents submitted by a browser. Is there
anything else that might be causing my problem?


Well, analyzing your problem from 10000' isn't that easy... can you provide
more details (code, error messages, ...)?

Cheers,

--
Joerg Jooss
www.joergjooss.de
ne**@joergjooss.de
Jul 21 '05 #2

P: n/a
What error message is it giving you? Is it looking for a specified set of
credentials, etc.?

Sean McCormack
Open Source for .NET
http://www.adapdev.org

"Gregory A Greenman" wrote:
I'm trying to write a program in vb.net to automate filling out a
series of forms on a website. There are three forms I need to
fill out in sequence. The first one is urlencoded. My program is
able to fill that one out just fine.

The second form is multipart/form-data. Unfortunately, I haven't
been able to fill that out in a way that makes the server happy.

I set up a copy of this form at my web site so that I could see
exactly what a browser sends to the server and compare that to
what my program sends. I saw a few mistakes in my program and I
fixed them, but the real web site still rejects my program's
submissions.

As best I can tell, the contents that my program submits are
identical to the contents submitted by a browser. Is there
anything else that might be causing my problem?
Thanks for any help.
--
Greg
----
greg -at- spencersoft -dot- com

Jul 21 '05 #3

P: n/a
In article <uD**************@tk2msftngp13.phx.gbl>,
jo*********@gmx.net says...
Gregory A Greenman wrote:
I'm trying to write a program in vb.net to automate filling out a
series of forms on a website. There are three forms I need to
fill out in sequence. The first one is urlencoded. My program is
able to fill that one out just fine.

The second form is multipart/form-data. Unfortunately, I haven't
been able to fill that out in a way that makes the server happy.

I set up a copy of this form at my web site so that I could see
exactly what a browser sends to the server and compare that to
what my program sends. I saw a few mistakes in my program and I
fixed them, but the real web site still rejects my program's
submissions.

As best I can tell, the contents that my program submits are
identical to the contents submitted by a browser. Is there
anything else that might be causing my problem?


Well, analyzing your problem from 10000' isn't that easy... can you provide
more details (code, error messages, ...)?


Okay. I'm getting "The remote server returned an error: (500)
internal server error".

I've got a form with two text boxes in the upper left corner,
txtHalfID and txtHalfPassword. There are three text boxes in the
upper right corner, txtISBN, txtDescription and txtPrice. There's
also a drop down combo in the upper right, cmbCondition.

The bottom portion of the screen has a read only multi line text
box called txtResponse, where the raw HTML from the server is
displayed.

Below the left text boxes is a button, btnSignOn. If you click
that button, it will sign you onto Half.com using the ID and
password in the textboxes above it. This part works fine for me.
I can see the "Welcome to eBay" page in txtResponse.

Below the right boxes is a button called btnListBook. When I
click on it it should call four web pages on half.com in order.

First it calls http://half.ebay.com/help/sell_books.cfm. It fills
in the field in the form there with txtISBN.

Next it calls http://half.ebay.com/cat/sell/pmsearch.cgi. It
fills out the form there with txtDescription and cmbCondition.
This form allows file uploads, so it's a multipart form. Although
my program's output looks good to me, I get that server error
when I submit this form.

Here's the code:

- Start Code ---------------------------------------------------
Imports System.Net

Public Class SBM
Inherits System.Windows.Forms.Form

Dim cc As New CookieCollection
Const encURL As Integer = 0
Const encMulti As Integer = 1

Private Sub SBM_Load(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles MyBase.Load
txtHalfID.Text = ""
txtHalfPassword.Text = ""

txtISBN.Text = "0764560255"
txtDescription.Text = "good good"
txtPrice.Text = "99.99"

With cmbCondition
.Items.Add(New Conditions("Brand New", "830"))
.Items.Add(New Conditions("Like New", "840"))
.Items.Add(New Conditions("Very Good", "849"))
.Items.Add(New Conditions("Good", "859"))
.Items.Add(New Conditions("Acceptable", "864"))
.SelectedIndex = 0
End With

cmbCondition.SelectedIndex = 0
End Sub

Private Sub btnSignOn_Click(ByVal sender As System.Object,
ByVal e As System.EventArgs) Handles btnSignOn.Click
Dim HWRequest As HttpWebRequest
Dim strURL As String
Dim HWParameters As Parameters
Dim intFound As Integer

HWRequest = GetRequest
("https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&UsingSSL=1
&co_partnerid=2&siteid=20")
strURL = "https://signin.half.ebay.com/ws/eBayISAPI.dll"

HWParameters = ReadResponse(HWRequest, strURL)

intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "userid"
p.Value = txtHalfID.Text
intFound += 1
Case "pass"
p.Value = txtHalfPassword.Text
intFound += 1
End Select

If intFound = 2 Then
Exit For
End If
Next

HWRequest = PostRequest(strURL, HWParameters, encURL)
GetResponse(HWRequest)
End Sub

Private Sub btnListBook_Click(ByVal sender As System.Object,
ByVal e As System.EventArgs) Handles btnListBook.Click
Dim strPostData As String
Dim HWRequest As HttpWebRequest
Dim strURL As String
Dim HWParameters As Parameters
Dim intFound As Integer
Dim pr As Parameter

Try
HWRequest = GetRequest
("http://half.ebay.com/help/sell_books.cfm")
strURL = "http://half.ebay.com/cat/sell/pmsearch.cgi"

HWParameters = ReadResponse(HWRequest, strURL)

intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "p_code"
p.Value = txtISBN.Text
intFound += 1
End Select

If intFound = 1 Then
Exit For
End If
Next

HWRequest = PostRequest(strURL, HWParameters, encURL)

strURL = "/cat/sell/save_new_listing.cgi"

HWParameters = ReadResponse(HWRequest, strURL)
pr = New Parameter
pr.Name = "x"
pr.Value = 20
pr.Type = Parameter.Input
HWParameters.Add(pr)
pr = New Parameter
pr.Name = "y"
pr.Value = 20
pr.Type = Parameter.Input
HWParameters.Add(pr)
intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "notes"
p.Value = txtDescription.Text
intFound += 1
Case "condition"
p.Value = cmbCondition.Items
(cmbCondition.SelectedIndex).ItemData
intFound += 1
End Select

If intFound = 2 Then
Exit For
End If
Next

strURL =
"http://half.ebay.com/cat/sell/save_new_listing.cgi"

HWRequest = PostRequest(strURL, HWParameters,
encMulti)

strURL = "sell.jsp"
HWRequest.Referer =
"http://half.ebay.com/cat/sell/pmsearch.cgi"
'the internal server error is generated in this call
'to ReadResponse
HWParameters = ReadResponse(HWRequest, strURL)
intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "itemPrice"
p.Value = txtPrice.Text
intFound += 1
End Select

If intFound = 1 Then
Exit For
End If
Next

strURL = "http://half.ebay.com/cat/sell/sell.jsp"

HWRequest = PostRequest(strURL, HWParameters, encURL)

GetResponse(HWRequest)

MessageBox.Show("Success!", MsgBoxStyle.OKOnly, "Book
Posted")

Catch ex As Exception
MessageBox.Show("Error: " & ex.Message, "Error!!!",
MessageBoxButtons.OK, MessageBoxIcon.Exclamation)

MsgBox("Half.com appears to have changed its posting
procedures. As a result, this version of SBM cannot list books on
it.", MsgBoxStyle.Exclamation, "Half.com Listing Problem")
End Try
End Sub

Private Function GetRequest(ByVal strURL As String) As
HttpWebRequest
GetRequest = CreateRequest(strURL)
GetRequest.Method = "GET"
End Function

Private Function PostRequest(ByVal strURL As String, ByVal
PostParameters As Parameters, ByVal intType As Integer) As
HttpWebRequest
Dim encoding As New System.Text.ASCIIEncoding
Dim byte1 As Byte()
Dim newStream As System.IO.Stream
Dim strData As String
Const strBoundary As String =
"---------------------------7d4285126106b0"

PostRequest = CreateRequest(strURL)
PostRequest.Method = "POST"

If intType = encURL Then
PostRequest.ContentType = "application/x-www-form-
urlencoded"
Else
PostRequest.ContentType = "multipart/form-data,
boundary=" & strBoundary
End If

strData = BuildRequestString(PostParameters, intType,
strBoundary)

PostRequest.ContentLength = strData.Length

byte1 = encoding.GetBytes(strData)

newStream = PostRequest.GetRequestStream
newStream.Write(byte1, 0, byte1.Length)
newStream.Close()
End Function

Private Function BuildRequestString(ByVal RequestParameters
As Parameters, ByVal intType As Integer, ByVal strBoundary As
String)
BuildRequestString = ""

Select Case intType
Case encURL
For Each p As Parameter In RequestParameters
BuildRequestString &= p.Name.Trim & "=" &
p.Value.Trim & "&"
Next

BuildRequestString = Mid(BuildRequestString, 1,
Len(BuildRequestString) - 1)
Case encMulti
'the requeststring generated for the multipart form is
'shown below
For Each p As Parameter In RequestParameters
BuildRequestString &= "--" & strBoundary &
vbCrLf & "Content-Disposition: form-data; name=""" & p.Name.Trim
& """"
Select Case p.Type
Case Parameter.File
BuildRequestString &= ";
filename=""""" & vbCrLf & "Content-Type: application/octet-
stream" & vbCrLf & vbCrLf & vbCrLf
Case Parameter.Input
BuildRequestString &= vbCrLf & vbCrLf
& p.Value & vbCrLf
End Select
Next
BuildRequestString &= "--" & strBoundary & "--"
End Select
End Function

Private Function CreateRequest(ByVal strURL As String) As
HttpWebRequest
Dim Uri As Uri

Uri = New Uri(strURL)
CreateRequest = HttpWebRequest.Create(Uri)
CreateRequest.AllowAutoRedirect = True
CreateRequest.CookieContainer = New CookieContainer

If cc.Count > 0 Then
CreateRequest.CookieContainer.Add(cc)
End If
End Function

Private Sub GetResponse(ByVal ReadRequest As HttpWebRequest)
Dim HWResponse As HttpWebResponse
Dim cookie As Cookie

HWResponse = ReadRequest.GetResponse()

If HWResponse.Cookies.Count > 0 Then
For Each cookie In HWResponse.Cookies
cc.Add(cookie)
Next
End If

'debugging
Dim sr As System.IO.StreamReader
Dim strResult As String

sr = New System.IO.StreamReader
(HWResponse.GetResponseStream())

txtResponse.Text = sr.ReadToEnd

sr.Close()
'debugging
End Sub

Private Function ReadResponse(ByVal ReadRequest As
HttpWebRequest, ByVal ReadURL As String) As Parameters
Dim HWResponse As HttpWebResponse
Dim cookie As Cookie
Dim sr As System.IO.StreamReader
Dim strResult As String
Dim intTagPos As Integer
Dim intLength As Integer
Dim intURLPos As Integer
Dim strTag As String
Dim blnFormFound As Boolean
Dim blnMoreForms As Boolean
Dim intPos As Integer
Dim strName As String
Dim strValue As String
Dim ReadParameter As Parameter
Dim intType As Integer
Dim intInputPos As Integer
Dim intTextPos As Integer
Dim intSelectPos As Integer
Dim blnTagFound As Boolean

ReadResponse = New Parameters
'the next line generates the internal server error
HWResponse = ReadRequest.GetResponse()

If HWResponse.Cookies.Count > 0 Then
For Each cookie In HWResponse.Cookies
cc.Add(cookie)
Next
End If

sr = New System.IO.StreamReader
(HWResponse.GetResponseStream())

strResult = sr.ReadToEnd

'debugging
txtResponse.Text = strResult
'debugging

blnFormFound = False
blnMoreForms = True
intTagPos = 1

While Not blnFormFound And blnMoreForms
intTagPos = InStr(intTagPos, strResult.ToUpper,
"<FORM", CompareMethod.Text)

If intTagPos <> 0 Then
intLength = InStr(intTagPos, strResult, ">",
CompareMethod.Text) - intTagPos
strTag = strResult.Substring(intTagPos,
intLength)

intURLPos = InStr(1, strTag.ToUpper,
ReadURL.ToUpper, CompareMethod.Text)

If intURLPos <> 0 Then
intLength = InStr(intTagPos,
strResult.ToUpper, "</FORM>", CompareMethod.Text) - intTagPos
strResult = strResult.Substring(intTagPos,
intLength)
blnFormFound = True
Else
intTagPos += 1
End If
Else
blnMoreForms = False
End If
End While

If blnFormFound Then
intTagPos = 1

While intTagPos <> 0
intInputPos = InStr(intTagPos, strResult.ToUpper,
"<INPUT", CompareMethod.Text)
intTextPos = InStr(intTagPos, strResult.ToUpper,
"<TEXTAREA", CompareMethod.Text)
intSelectPos = InStr(intTagPos,
strResult.ToUpper, "<SELECT", CompareMethod.Text)

intTagPos = IIf(intTextPos <> 0 And intTextPos <
intInputPos, intTextPos, intInputPos)
intTagPos = IIf(intSelectPos <> 0 And
intSelectPos < intTagPos, intSelectPos, intTagPos)

intType = Parameter.Input

If intTagPos <> 0 Then
intLength = InStr(intTagPos, strResult, ">",
CompareMethod.Text) - intTagPos
strTag = strResult.Substring(intTagPos,
intLength)

intPos = InStr(1, strTag.ToUpper,
"TYPE=SUBMIT", CompareMethod.Text)
If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"TYPE=""SUBMIT""", CompareMethod.Text)
End If

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"NAME=""", CompareMethod.Text)

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"NAME=", CompareMethod.Text)
intLength = InStr(intPos + 5, strTag,
" ", CompareMethod.Text) - intPos - 5
strName = strTag.Substring(intPos +
4, intLength)
Else
intLength = InStr(intPos + 6, strTag,
"""", CompareMethod.Text) - intPos - 6
strName = strTag.Substring(intPos +
5, intLength)
End If

If intPos <> 0 Then
intPos = InStr(1, strTag.ToUpper,
"VALUE=""", CompareMethod.Text)

If intPos = 0 Then
strValue = ""
Else
intLength = InStr(intPos + 7,
strTag, """", CompareMethod.Text) - intPos - 7
strValue = strTag.Substring
(intPos + 6, intLength)
End If

intPos = InStr(1, strTag.ToUpper,
"TYPE=""FILE""", CompareMethod.Text)

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"TYPE=FILE", CompareMethod.Text)
End If

intType = IIf(intPos = 0, intType,
Parameter.File)

ReadParameter = New Parameter
ReadParameter.Name = strName
ReadParameter.Value = strValue
ReadParameter.Type = intType
ReadResponse.Add(ReadParameter)
End If
End If

intTagPos += 1
End If
End While
Else
Throw New System.Exception("Form not found.")
End If
End Function
End Class

Public Class Parameters
Inherits System.Collections.CollectionBase

Public Sub Add(ByVal sFld As Parameter)
List.Add(sFld)
End Sub

Public ReadOnly Property Item(ByVal index As Integer) As
Parameter
Get
Return CType(List.Item(index), Parameter)
End Get
End Property
End Class


Public Class Parameter
Public Const Input As Integer = 0
Public Const File As Integer = 1

Dim strName As String
Dim strValue As String
Dim intType As Integer

Public Property Name() As String
Get
Return strName
End Get
Set(ByVal vName As String)
strName = vName
End Set
End Property

Public Property Value() As String
Get
Return strValue
End Get
Set(ByVal vName As String)
strValue = vName
End Set
End Property

Public Property Type() As Integer
Get
Return intType
End Get
Set(ByVal vType As Integer)
intType = vType
End Set
End Property
End Class
Public Class Conditions
Private strName As String
Private strID As String

Public Sub New()
strName = ""
strID = ""
End Sub

Public Sub New(ByVal Name As String, ByVal ID As String)
strName = Name
strID = ID
End Sub

Public Property Name() As String
Get
Return strName
End Get

Set(ByVal sValue As String)
strName = sValue
End Set
End Property

Public Property ItemData() As String
Get
Return strID
End Get

Set(ByVal iValue As String)
strID = iValue
End Set
End Property

Public Overrides Function ToString() As String
Return strName
End Function
End Class
- End Code -----------------------------------------------------
Here's the requeststring generated for the multipart form:

- Start requeststring ------------------------------------------
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="condition"

830
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="notes"

good good
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="image_file"; filename=""
Content-Type: application/octet-stream
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="image_url"
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="version"

729
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="domain_id"

1856
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="meta_id"

1
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="context_name"

w13.1097912464.0000318444
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="x"

20
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="y"

20
-----------------------------7d4285126106b0--
- End requeststring --------------------------------------------

The requeststring looks right to me. I'm wondering if there's a
header I should set that I'm missing.

Thanks for the help.


--
Greg
----
greg -at- spencersoft -dot- com
Jul 21 '05 #4

P: n/a
That got right down to the details, didn't it?

I take it that, even though this thread is not new, you have not solved the
problem yet.

You look like you are doing things fairly well. You are collecting the
cookies and passing them back, and the formatting looks pretty good. I
would suggest, however, that the page that fails is calling a JSP, which
means that the server side may have a good deal of logic to PREVENT folks
like you from pretending to be a browser.

I don't see that you have set the User Agent header. It is probably normal
for the JSP page to be attempting to identify the browser, perhaps to send
back a message that will render correctly every time. I've done similar
code (a little differently, but same concept) and I've had problems when I
didn't set the User Agent header. That's where I would start.

If that doesn't work, there may be some software that you can install that
will sniff your own port 80, showing you what packets are travelling. You
can then access the pages using IE and then using your app, to see if there
is another header, perhaps one added by some Javascript code that downloaded
on the previous page, that you are not sending. I know that there is a tool
for doing this that is downloadable open source using the Unix compatibility
libraries (I used it about 18 months ago... don't remember the name but I'm
sure I can find it, so if you can't, let me know and I will dig a little).

Good luck
--- Nick Malik
http://weblogs.asp.net/nickmalik

"Gregory A Greenman" <se*@sig.below> wrote in message
news:MP************************@netnews.comcast.ne t...
In article <uD**************@tk2msftngp13.phx.gbl>,
jo*********@gmx.net says...
Gregory A Greenman wrote:
I'm trying to write a program in vb.net to automate filling out a
series of forms on a website. There are three forms I need to
fill out in sequence. The first one is urlencoded. My program is
able to fill that one out just fine.

The second form is multipart/form-data. Unfortunately, I haven't
been able to fill that out in a way that makes the server happy.

I set up a copy of this form at my web site so that I could see
exactly what a browser sends to the server and compare that to
what my program sends. I saw a few mistakes in my program and I
fixed them, but the real web site still rejects my program's
submissions.

As best I can tell, the contents that my program submits are
identical to the contents submitted by a browser. Is there
anything else that might be causing my problem?


Well, analyzing your problem from 10000' isn't that easy... can you provide more details (code, error messages, ...)?


Okay. I'm getting "The remote server returned an error: (500)
internal server error".

I've got a form with two text boxes in the upper left corner,
txtHalfID and txtHalfPassword. There are three text boxes in the
upper right corner, txtISBN, txtDescription and txtPrice. There's
also a drop down combo in the upper right, cmbCondition.

The bottom portion of the screen has a read only multi line text
box called txtResponse, where the raw HTML from the server is
displayed.

Below the left text boxes is a button, btnSignOn. If you click
that button, it will sign you onto Half.com using the ID and
password in the textboxes above it. This part works fine for me.
I can see the "Welcome to eBay" page in txtResponse.

Below the right boxes is a button called btnListBook. When I
click on it it should call four web pages on half.com in order.

First it calls http://half.ebay.com/help/sell_books.cfm. It fills
in the field in the form there with txtISBN.

Next it calls http://half.ebay.com/cat/sell/pmsearch.cgi. It
fills out the form there with txtDescription and cmbCondition.
This form allows file uploads, so it's a multipart form. Although
my program's output looks good to me, I get that server error
when I submit this form.

Here's the code:

- Start Code ---------------------------------------------------
Imports System.Net

Public Class SBM
Inherits System.Windows.Forms.Form

Dim cc As New CookieCollection
Const encURL As Integer = 0
Const encMulti As Integer = 1

Private Sub SBM_Load(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles MyBase.Load
txtHalfID.Text = ""
txtHalfPassword.Text = ""

txtISBN.Text = "0764560255"
txtDescription.Text = "good good"
txtPrice.Text = "99.99"

With cmbCondition
.Items.Add(New Conditions("Brand New", "830"))
.Items.Add(New Conditions("Like New", "840"))
.Items.Add(New Conditions("Very Good", "849"))
.Items.Add(New Conditions("Good", "859"))
.Items.Add(New Conditions("Acceptable", "864"))
.SelectedIndex = 0
End With

cmbCondition.SelectedIndex = 0
End Sub

Private Sub btnSignOn_Click(ByVal sender As System.Object,
ByVal e As System.EventArgs) Handles btnSignOn.Click
Dim HWRequest As HttpWebRequest
Dim strURL As String
Dim HWParameters As Parameters
Dim intFound As Integer

HWRequest = GetRequest
("https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&UsingSSL=1
&co_partnerid=2&siteid=20")
strURL = "https://signin.half.ebay.com/ws/eBayISAPI.dll"

HWParameters = ReadResponse(HWRequest, strURL)

intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "userid"
p.Value = txtHalfID.Text
intFound += 1
Case "pass"
p.Value = txtHalfPassword.Text
intFound += 1
End Select

If intFound = 2 Then
Exit For
End If
Next

HWRequest = PostRequest(strURL, HWParameters, encURL)
GetResponse(HWRequest)
End Sub

Private Sub btnListBook_Click(ByVal sender As System.Object,
ByVal e As System.EventArgs) Handles btnListBook.Click
Dim strPostData As String
Dim HWRequest As HttpWebRequest
Dim strURL As String
Dim HWParameters As Parameters
Dim intFound As Integer
Dim pr As Parameter

Try
HWRequest = GetRequest
("http://half.ebay.com/help/sell_books.cfm")
strURL = "http://half.ebay.com/cat/sell/pmsearch.cgi"

HWParameters = ReadResponse(HWRequest, strURL)

intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "p_code"
p.Value = txtISBN.Text
intFound += 1
End Select

If intFound = 1 Then
Exit For
End If
Next

HWRequest = PostRequest(strURL, HWParameters, encURL)

strURL = "/cat/sell/save_new_listing.cgi"

HWParameters = ReadResponse(HWRequest, strURL)
pr = New Parameter
pr.Name = "x"
pr.Value = 20
pr.Type = Parameter.Input
HWParameters.Add(pr)
pr = New Parameter
pr.Name = "y"
pr.Value = 20
pr.Type = Parameter.Input
HWParameters.Add(pr)
intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "notes"
p.Value = txtDescription.Text
intFound += 1
Case "condition"
p.Value = cmbCondition.Items
(cmbCondition.SelectedIndex).ItemData
intFound += 1
End Select

If intFound = 2 Then
Exit For
End If
Next

strURL =
"http://half.ebay.com/cat/sell/save_new_listing.cgi"

HWRequest = PostRequest(strURL, HWParameters,
encMulti)

strURL = "sell.jsp"
HWRequest.Referer =
"http://half.ebay.com/cat/sell/pmsearch.cgi"
'the internal server error is generated in this call
'to ReadResponse
HWParameters = ReadResponse(HWRequest, strURL)
intFound = 0

For Each p As Parameter In HWParameters
Select Case p.Name
Case "itemPrice"
p.Value = txtPrice.Text
intFound += 1
End Select

If intFound = 1 Then
Exit For
End If
Next

strURL = "http://half.ebay.com/cat/sell/sell.jsp"

HWRequest = PostRequest(strURL, HWParameters, encURL)

GetResponse(HWRequest)

MessageBox.Show("Success!", MsgBoxStyle.OKOnly, "Book
Posted")

Catch ex As Exception
MessageBox.Show("Error: " & ex.Message, "Error!!!",
MessageBoxButtons.OK, MessageBoxIcon.Exclamation)

MsgBox("Half.com appears to have changed its posting
procedures. As a result, this version of SBM cannot list books on
it.", MsgBoxStyle.Exclamation, "Half.com Listing Problem")
End Try
End Sub

Private Function GetRequest(ByVal strURL As String) As
HttpWebRequest
GetRequest = CreateRequest(strURL)
GetRequest.Method = "GET"
End Function

Private Function PostRequest(ByVal strURL As String, ByVal
PostParameters As Parameters, ByVal intType As Integer) As
HttpWebRequest
Dim encoding As New System.Text.ASCIIEncoding
Dim byte1 As Byte()
Dim newStream As System.IO.Stream
Dim strData As String
Const strBoundary As String =
"---------------------------7d4285126106b0"

PostRequest = CreateRequest(strURL)
PostRequest.Method = "POST"

If intType = encURL Then
PostRequest.ContentType = "application/x-www-form-
urlencoded"
Else
PostRequest.ContentType = "multipart/form-data,
boundary=" & strBoundary
End If

strData = BuildRequestString(PostParameters, intType,
strBoundary)

PostRequest.ContentLength = strData.Length

byte1 = encoding.GetBytes(strData)

newStream = PostRequest.GetRequestStream
newStream.Write(byte1, 0, byte1.Length)
newStream.Close()
End Function

Private Function BuildRequestString(ByVal RequestParameters
As Parameters, ByVal intType As Integer, ByVal strBoundary As
String)
BuildRequestString = ""

Select Case intType
Case encURL
For Each p As Parameter In RequestParameters
BuildRequestString &= p.Name.Trim & "=" &
p.Value.Trim & "&"
Next

BuildRequestString = Mid(BuildRequestString, 1,
Len(BuildRequestString) - 1)
Case encMulti
'the requeststring generated for the multipart form is
'shown below
For Each p As Parameter In RequestParameters
BuildRequestString &= "--" & strBoundary &
vbCrLf & "Content-Disposition: form-data; name=""" & p.Name.Trim
& """"
Select Case p.Type
Case Parameter.File
BuildRequestString &= ";
filename=""""" & vbCrLf & "Content-Type: application/octet-
stream" & vbCrLf & vbCrLf & vbCrLf
Case Parameter.Input
BuildRequestString &= vbCrLf & vbCrLf
& p.Value & vbCrLf
End Select
Next
BuildRequestString &= "--" & strBoundary & "--"
End Select
End Function

Private Function CreateRequest(ByVal strURL As String) As
HttpWebRequest
Dim Uri As Uri

Uri = New Uri(strURL)
CreateRequest = HttpWebRequest.Create(Uri)
CreateRequest.AllowAutoRedirect = True
CreateRequest.CookieContainer = New CookieContainer

If cc.Count > 0 Then
CreateRequest.CookieContainer.Add(cc)
End If
End Function

Private Sub GetResponse(ByVal ReadRequest As HttpWebRequest)
Dim HWResponse As HttpWebResponse
Dim cookie As Cookie

HWResponse = ReadRequest.GetResponse()

If HWResponse.Cookies.Count > 0 Then
For Each cookie In HWResponse.Cookies
cc.Add(cookie)
Next
End If

'debugging
Dim sr As System.IO.StreamReader
Dim strResult As String

sr = New System.IO.StreamReader
(HWResponse.GetResponseStream())

txtResponse.Text = sr.ReadToEnd

sr.Close()
'debugging
End Sub

Private Function ReadResponse(ByVal ReadRequest As
HttpWebRequest, ByVal ReadURL As String) As Parameters
Dim HWResponse As HttpWebResponse
Dim cookie As Cookie
Dim sr As System.IO.StreamReader
Dim strResult As String
Dim intTagPos As Integer
Dim intLength As Integer
Dim intURLPos As Integer
Dim strTag As String
Dim blnFormFound As Boolean
Dim blnMoreForms As Boolean
Dim intPos As Integer
Dim strName As String
Dim strValue As String
Dim ReadParameter As Parameter
Dim intType As Integer
Dim intInputPos As Integer
Dim intTextPos As Integer
Dim intSelectPos As Integer
Dim blnTagFound As Boolean

ReadResponse = New Parameters
'the next line generates the internal server error
HWResponse = ReadRequest.GetResponse()

If HWResponse.Cookies.Count > 0 Then
For Each cookie In HWResponse.Cookies
cc.Add(cookie)
Next
End If

sr = New System.IO.StreamReader
(HWResponse.GetResponseStream())

strResult = sr.ReadToEnd

'debugging
txtResponse.Text = strResult
'debugging

blnFormFound = False
blnMoreForms = True
intTagPos = 1

While Not blnFormFound And blnMoreForms
intTagPos = InStr(intTagPos, strResult.ToUpper,
"<FORM", CompareMethod.Text)

If intTagPos <> 0 Then
intLength = InStr(intTagPos, strResult, ">",
CompareMethod.Text) - intTagPos
strTag = strResult.Substring(intTagPos,
intLength)

intURLPos = InStr(1, strTag.ToUpper,
ReadURL.ToUpper, CompareMethod.Text)

If intURLPos <> 0 Then
intLength = InStr(intTagPos,
strResult.ToUpper, "</FORM>", CompareMethod.Text) - intTagPos
strResult = strResult.Substring(intTagPos,
intLength)
blnFormFound = True
Else
intTagPos += 1
End If
Else
blnMoreForms = False
End If
End While

If blnFormFound Then
intTagPos = 1

While intTagPos <> 0
intInputPos = InStr(intTagPos, strResult.ToUpper,
"<INPUT", CompareMethod.Text)
intTextPos = InStr(intTagPos, strResult.ToUpper,
"<TEXTAREA", CompareMethod.Text)
intSelectPos = InStr(intTagPos,
strResult.ToUpper, "<SELECT", CompareMethod.Text)

intTagPos = IIf(intTextPos <> 0 And intTextPos <
intInputPos, intTextPos, intInputPos)
intTagPos = IIf(intSelectPos <> 0 And
intSelectPos < intTagPos, intSelectPos, intTagPos)

intType = Parameter.Input

If intTagPos <> 0 Then
intLength = InStr(intTagPos, strResult, ">",
CompareMethod.Text) - intTagPos
strTag = strResult.Substring(intTagPos,
intLength)

intPos = InStr(1, strTag.ToUpper,
"TYPE=SUBMIT", CompareMethod.Text)
If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"TYPE=""SUBMIT""", CompareMethod.Text)
End If

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"NAME=""", CompareMethod.Text)

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"NAME=", CompareMethod.Text)
intLength = InStr(intPos + 5, strTag,
" ", CompareMethod.Text) - intPos - 5
strName = strTag.Substring(intPos +
4, intLength)
Else
intLength = InStr(intPos + 6, strTag,
"""", CompareMethod.Text) - intPos - 6
strName = strTag.Substring(intPos +
5, intLength)
End If

If intPos <> 0 Then
intPos = InStr(1, strTag.ToUpper,
"VALUE=""", CompareMethod.Text)

If intPos = 0 Then
strValue = ""
Else
intLength = InStr(intPos + 7,
strTag, """", CompareMethod.Text) - intPos - 7
strValue = strTag.Substring
(intPos + 6, intLength)
End If

intPos = InStr(1, strTag.ToUpper,
"TYPE=""FILE""", CompareMethod.Text)

If intPos = 0 Then
intPos = InStr(1, strTag.ToUpper,
"TYPE=FILE", CompareMethod.Text)
End If

intType = IIf(intPos = 0, intType,
Parameter.File)

ReadParameter = New Parameter
ReadParameter.Name = strName
ReadParameter.Value = strValue
ReadParameter.Type = intType
ReadResponse.Add(ReadParameter)
End If
End If

intTagPos += 1
End If
End While
Else
Throw New System.Exception("Form not found.")
End If
End Function
End Class

Public Class Parameters
Inherits System.Collections.CollectionBase

Public Sub Add(ByVal sFld As Parameter)
List.Add(sFld)
End Sub

Public ReadOnly Property Item(ByVal index As Integer) As
Parameter
Get
Return CType(List.Item(index), Parameter)
End Get
End Property
End Class


Public Class Parameter
Public Const Input As Integer = 0
Public Const File As Integer = 1

Dim strName As String
Dim strValue As String
Dim intType As Integer

Public Property Name() As String
Get
Return strName
End Get
Set(ByVal vName As String)
strName = vName
End Set
End Property

Public Property Value() As String
Get
Return strValue
End Get
Set(ByVal vName As String)
strValue = vName
End Set
End Property

Public Property Type() As Integer
Get
Return intType
End Get
Set(ByVal vType As Integer)
intType = vType
End Set
End Property
End Class
Public Class Conditions
Private strName As String
Private strID As String

Public Sub New()
strName = ""
strID = ""
End Sub

Public Sub New(ByVal Name As String, ByVal ID As String)
strName = Name
strID = ID
End Sub

Public Property Name() As String
Get
Return strName
End Get

Set(ByVal sValue As String)
strName = sValue
End Set
End Property

Public Property ItemData() As String
Get
Return strID
End Get

Set(ByVal iValue As String)
strID = iValue
End Set
End Property

Public Overrides Function ToString() As String
Return strName
End Function
End Class
- End Code -----------------------------------------------------
Here's the requeststring generated for the multipart form:

- Start requeststring ------------------------------------------
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="condition"

830
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="notes"

good good
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="image_file"; filename=""
Content-Type: application/octet-stream
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="image_url"
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="version"

729
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="domain_id"

1856
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="meta_id"

1
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="context_name"

w13.1097912464.0000318444
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="x"

20
-----------------------------7d4285126106b0
Content-Disposition: form-data; name="y"

20
-----------------------------7d4285126106b0--
- End requeststring --------------------------------------------

The requeststring looks right to me. I'm wondering if there's a
header I should set that I'm missing.

Thanks for the help.


--
Greg
----
greg -at- spencersoft -dot- com

Jul 21 '05 #5

P: n/a
In article <HAdcd.482227$8_6.62731@attbi_s04>,
ni*******@hotmail.nospam.com says...
That got right down to the details, didn't it?

I take it that, even though this thread is not new, you have not solved the
problem yet.

Nope, I had an important meeting out of town I had to prepare
for, plus I wanted to make my code more presentable before I
posted it.

You look like you are doing things fairly well. You are collecting the
cookies and passing them back, and the formatting looks pretty good. I
would suggest, however, that the page that fails is calling a JSP, which
means that the server side may have a good deal of logic to PREVENT folks
like you from pretending to be a browser.

Here's the form tag for the three pages I'm trying to access. The
problem I was having was with the second form. It looks like the
third one uses Java, so that may be a problem there.

<FORM action="http://half.ebay.com/cat/sell/pmsearch.cgi"
method="post">

<form action="/cat/sell/save_new_listing.cgi"
enctype="multipart/form-data" method=post>

<form name="pricing" method="post" action="sell.jsp" onSubmit="if
(this.submitted) return false; else { this.submitted = true;
disableSubmits(this); return true; }">

I don't see that you have set the User Agent header. It is probably normal
for the JSP page to be attempting to identify the browser, perhaps to send
back a message that will render correctly every time. I've done similar
code (a little differently, but same concept) and I've had problems when I
didn't set the User Agent header. That's where I would start.

That seems to have solved the problem, at least for this step. I
added this line:

HWRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows
NT 5.1; SV1; .NET CLR 1.1.4322)"

and I was able to get past the problem. I did get a 404 error, so
I'm not all the way home yet, but I'm one step closer. Also, the
404 error came later than the 505 I was getting previously.

I'm about to head out the door, so I'll see if I can make any
more progress later tonight.

If that doesn't work, there may be some software that you can install that
will sniff your own port 80, showing you what packets are travelling. You
can then access the pages using IE and then using your app, to see if there
is another header, perhaps one added by some Javascript code that downloaded
on the previous page, that you are not sending. I know that there is a tool
for doing this that is downloadable open source using the Unix compatibility
libraries (I used it about 18 months ago... don't remember the name but I'm
sure I can find it, so if you can't, let me know and I will dig a little).

I found this: tcptrace http://www.tcptrace.com/. Hopefully I'll
be able to cross this next hurdle without it.

Thanks for the help,
Greg
Jul 21 '05 #6

P: n/a
Gregory A Greenman wrote:
In article <HAdcd.482227$8_6.62731@attbi_s04>,
ni*******@hotmail.nospam.com says...


I think the way you handle upload files may be the culprit. You're uploading
files (or rather pretend to) exclusively as application/octet-stream. From a
web application perspective, application/octet-stream is rather
unspecific -- I cannot imagine what a web application could do with some
unspecific binary content (other than archive it in some way).

As far useful tools are concerned, I strongly suggest using Fiddler
(http://www.fiddlertool.com/). It's a debugging proxy that runs locally and
can track all HTTP traffic between your application and the remote host.

Cheers,

--
Joerg Jooss
www.joergjooss.de
ne**@joergjooss.de
Jul 21 '05 #7

P: n/a
In article <u$**************@TK2MSFTNGP09.phx.gbl>,
jo*********@gmx.net says...
Gregory A Greenman wrote:
In article <HAdcd.482227$8_6.62731@attbi_s04>,
ni*******@hotmail.nospam.com says...
I think the way you handle upload files may be the culprit. You're uploading
files (or rather pretend to) exclusively as application/octet-stream. From a
web application perspective, application/octet-stream is rather
unspecific -- I cannot imagine what a web application could do with some
unspecific binary content (other than archive it in some way).


I had set up a mock version of that form on my web site. When the
button was clicked on that form, the cgi would just record what
the browser sent. I copied that into my program. Both IE and
Mozilla Firefox used application/octet-stream when there was no
actual upload.

I was finally able to get past that page by adding the following
line to my code:

HWRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows
NT 5.1; SV1; .NET CLR 1.1.4322)"

Stripping it of all extraneous HTML, the form that I was trying
to fill out is:

<form action="/cat/sell/save_new_listing.cgi"
enctype="multipart/form-data" method=post>
<select name=condition SELECTED>
<option value=''>Select One
<option value=830 >Brand New
<option value=840 >Like New
<option value=849 >Very Good
<option value=859 >Good
<option value=864 >Acceptable
</select>
<textarea name="notes" rows=5 cols=40></textarea>
<INPUT TYPE=FILE NAME="image_file">
<INPUT TYPE=TEXT NAME="image_url" VALUE=>
<input type=submit name="Continue" value="Continue &gt;" border=
0>
<input type=hidden name="version" value="729">
<input type=hidden name="domain_id" value="1856">
<input type=hidden name="meta_id" value="1">
<input type=hidden name="context_name"
value="w48.1097876311.000022364">
</form>

I noticed that when IE filled out the form, it put the "x" and
"y" parameters at the end, after all the hidden parameters. OTOH,
Mozilla Firefox placed "x" and "y" where the button is, before
the hidden parameters. Maybe that's why that page needs to know
what the browser is when the others didn't.

As far useful tools are concerned, I strongly suggest using Fiddler
(http://www.fiddlertool.com/). It's a debugging proxy that runs locally and
can track all HTTP traffic between your application and the remote host.

Thanks, I'm now having a problem filling out the next (and last)
form in the sequence. I may take a look at that, if I can't
figure out the solution otherwise.

Thanks for the help.
--
Greg
----
greg -at- spencersoft -dot- com
Jul 21 '05 #8

P: n/a
Gregory A Greenman wrote:
I had set up a mock version of that form on my web site. When the
button was clicked on that form, the cgi would just record what
the browser sent. I copied that into my program. Both IE and
Mozilla Firefox used application/octet-stream when there was no
actual upload.
That's because you probably tried to upload some arbitrary binary. Both
browsers use proper the MIME type for file uploads if possible.
I noticed that when IE filled out the form, it put the "x" and
"y" parameters at the end, after all the hidden parameters. OTOH,
Mozilla Firefox placed "x" and "y" where the button is, before
the hidden parameters. Maybe that's why that page needs to know
what the browser is when the others didn't.


There's no defined order in which form parameters are to be sent, so that
shouldn't make a difference. If a web application does assume a specific
order, it's broken.

Cheers,

--
Joerg Jooss
www.joergjooss.de
ne**@joergjooss.de
Jul 21 '05 #9

P: n/a
In article <#X**************@tk2msftngp13.phx.gbl>,
jo*********@gmx.net says...
Gregory A Greenman wrote:
I had set up a mock version of that form on my web site. When the
button was clicked on that form, the cgi would just record what
the browser sent. I copied that into my program. Both IE and
Mozilla Firefox used application/octet-stream when there was no
actual upload.


That's because you probably tried to upload some arbitrary binary. Both
browsers use proper the MIME type for file uploads if possible.

No, I've never tried to do an upload with this form. Apparently,
if there's no upload, IE and FF both use application/octet-stream
with no data.

I noticed that when IE filled out the form, it put the "x" and
"y" parameters at the end, after all the hidden parameters. OTOH,
Mozilla Firefox placed "x" and "y" where the button is, before
the hidden parameters. Maybe that's why that page needs to know
what the browser is when the others didn't.


There's no defined order in which form parameters are to be sent, so that
shouldn't make a difference. If a web application does assume a specific
order, it's broken.

I can't seem to find any equivalent for multipart forms, but
according to RFC 1866 browsers should list the parameters in the
same order they appear in in the form.

8.2.1.2:
"The fields are listed in the order they appear in the document
with the name separated from the value by `=' and the pairs
separated from each other by `&'. Fields with null values may be
omitted."

http://www.faqs.org/rfcs/rfc1866.html

In an early pass, I wasn't preserving the field order. For the
multipart form, half was complaining that I hadn't sent it the
condition parameter. So, I think the order matters for this.

Thanks for the help.
--
Greg
----
greg -at- spencersoft -dot- com
Jul 21 '05 #10

P: n/a
Gregory A Greenman wrote:
In article <#X**************@tk2msftngp13.phx.gbl>,
jo*********@gmx.net says...

[...]
There's no defined order in which form parameters are to be sent, so
that shouldn't make a difference. If a web application does assume a
specific order, it's broken.

I can't seem to find any equivalent for multipart forms, but
according to RFC 1866 browsers should list the parameters in the
same order they appear in in the form.

8.2.1.2:
"The fields are listed in the order they appear in the document
with the name separated from the value by `=' and the pairs
separated from each other by `&'. Fields with null values may be
omitted."

http://www.faqs.org/rfcs/rfc1866.html


You're right! I was stuck in my own little word of
application/x-www-form-urlencoded ;-)

That's what happnes once you got used not to rely on parameter ordering due
to the way the Servlet API or ASP.NET work...

Cheers,

--
Joerg Jooss
www.joergjooss.de
ne**@joergjooss.de
Jul 21 '05 #11

This discussion thread is closed

Replies have been disabled for this discussion.