By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
429,077 Members | 2,160 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 429,077 IT Pros & Developers. It's quick & easy.

Retrieving web page data with the microsoft winsock control.

AaronL
P: 99
Hello,

I am currently working on a project that has me in sort of a bind. What I want to do is retrieve web pages from the internet, and strip them down to just text. I'll get using Regular Expressions to strip out the HTML code itself, the problem is actually getting the web pages from the internet.

I tried using the Microsoft Internet Transfer Control but my client was experiencing problems with some web pages not downloading, this particular page would just get to the <body> tag and then stop. My client reported to me that there are issues reported with the ITC so we determined to find an alternative.

Before I tried the ITC, I used the Microsoft Winsock Control, but with that control, I had problems with web pages being truncated causing my HTML strip out routine to malfunction. We don't want to use Microsoft Internet Controls either at the request of my client.

I feel the Microsoft Winsock Control is the best way to go as to my knowledge is the low level way of communicating with servers on the internet. I feel that it's my lack of understanding on how the Winsock Control works is what is causing my problems. So can someone look at this code and tell me what I'm doing wrong or tell me how I can use the Winsock Control for this project? My job is kind of on the line at this point.

Here is code that I put in a module:
Expand|Select|Wrap|Line Numbers
  1. 'Data variables
  2. Public DataIn As String
  3. Public DataOut As String
  4. Public ErrMsg As String
  5.  
  6. 'Network variables
  7. Public URL As String
  8. Public Port As Integer
  9.  
  10. 'Functional Variables
  11. Public NetExecuting As Boolean
  12. Public NetTimer As Long
  13. Public Function GetHTML(URL As String, Port As Integer)
  14.  
  15. NetTimer = 0
  16.  
  17. 'If winsock is still executing, wait
  18. If NetExecuting = True Then
  19.     Do
  20.         DoEvents
  21.     Loop Until NetExecuting = False
  22. End If
  23.  
  24. DataIn = ""
  25.  
  26. If frmWinSock.Winsock.State <> sckClosed Then
  27.     frmWinSock.Winsock.Close
  28. End If
  29.  
  30. 'If port is 0 then set it to 80
  31. If Port <> 0 Then
  32.     frmWinSock.Winsock.RemotePort = Port
  33. Else
  34.     frmWinSock.Winsock.RemotePort = 80
  35. End If
  36.  
  37. 'Send connection string compatible with Internet Explorer
  38. DataOut = "GET / HTTP/1.0" _
  39. & vbCrLf & "Accept: text/html" _
  40. & vbCrLf & "Host: " & URL _
  41. & vbCrLf & "Connection: open " _
  42. & vbCrLf & "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1)" _
  43. & vbCrLf & "Referer: " _
  44. & vbCrLf & "Cookie: " _
  45. & vbCrLf & vbCrLf
  46.  
  47. frmWinSock.Winsock.RemoteHost = URL
  48.  
  49. frmWinSock.Winsock.Connect
  50. Do
  51.     DoEvents
  52. Loop Until frmWinSock.Winsock.State = sckClosed And DataIn <> ""
  53.  
  54. GetHTML = DataIn
  55.  
  56. End Function
  57.  
Here is my code that runs the winsock functions:
Expand|Select|Wrap|Line Numbers
  1. Private Sub timNetTimeOut_Timer()
  2. Winsock.Close
  3. End Sub
  4.  
  5. Private Sub Winsock_Close()
  6. NetExecuting = False
  7. End Sub
  8.  
  9. Private Sub Winsock_Connect()
  10. timNetTimeOut.Enabled = True
  11. NetExecuting = True
  12. ErrMsg = ""
  13. Winsock.SendData DataOut
  14. End Sub
  15.  
  16. Private Sub Winsock_DataArrival(ByVal bytesTotal As Long)
  17. Dim DataArrived As String
  18. On Error Resume Next
  19. Winsock.GetData DataArrived
  20. DataIn = DataIn & DataArrived
  21. End Sub
  22.  
  23. Private Sub Winsock_Error(ByVal Number As Integer, Description As String, ByVal Scode As Long, ByVal Source As String, ByVal HelpFile As String, ByVal HelpContext As Long, CancelDisplay As Boolean)
  24. NetExecuting = False
  25. ErrMsg = "A network error has occurred: " & Number & " " & Description
  26. End Sub
  27.  
Now I set the timeout timer I made to 10 seconds which should be plenty of time to retrieve a web page, but for example http://news.yahoo.com gets truncated.
Jan 6 '10 #1
Share this Question
Share on Google+
3 Replies


AaronL
P: 99
I found that winsock has a problem with receiving large data streams. If anyone knows a way around this, I could use the help. Thanks!
Jan 6 '10 #2

P: n/a
I found your code clean and I liked it, thanks.

It works out for me to use a buffer and save it away periodically during the reception of data.
Oct 16 '10 #3

P: n/a
Expand|Select|Wrap|Line Numbers
  1. Private Sub Wsck_DataArrival(ByVal bytesTotal As Long)
  2.  
  3.  Wsck.GetData Response
  4.  Buffer = Buffer & Response
  5.  
  6.  If InStr(1, Buffer, "</HTML>", vbTextCompare) > 0 Then
  7.    Open App.Path & "/folder/" & i & ".txt" For Output As #1
  8.        Print #1, Buffer
  9.        Close #1       
  10.  End If
  11.  
  12. End Sub
Oct 16 '10 #4

Post your reply

Sign in to post your reply or Sign up for a free account.