473,670 Members | 2,550 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Screen scraping, then caching.... not working on web parts page?

We have a web parts intranet at our office here I have been working
on.

I have been trying to create a way of getting our OCE TDS 600 plotter
queue to be a web part on the page, seeing as their is no way of
interfacing apart from their 'print exec workgroup' web based program.

So I decided to screen scrape, I've done a scrape of the intranet page
of print exec workgroup (http://tds600 on our network) seemed to go
fine, then i used regex, and some replace's to delete the stuff I
didn't want and change some formatting to suit my page then wrote the
html code to a label, seemed to work fine.

I then decided maybe I should do some caching, as this plotter doesnt
want to be hit up when ever user in the network (with autorefresh
every 5 mins) as it is receiving drawings and stuff.

So I tried caching it just for a minute or two, just so that way only
one person hits it every 2 minutes - the rest get it from cache.

But it doesn't seem to be working, I made a test page, seemed to cache
fine, but then when I made it an ascx control and put it on my web
parts page, it never caches. And perhaps as a result, actually
retreiving the scrape is a hit and miss afair, and even worse, people
can't access print exec (which is used to send drawings to plotter) as
it says theres too many users connected (theres no license limit - but
it's obviously a hard limit set so that bandwidth for plots doesnt
suffer - so the web connections must be making lots and overloading
it)

I'm making two parts, 1 to get the status on rolls (empty icons etc)
and one for queue, they are essentiall they same.

Expand|Select|Wrap|Line Numbers
  1. Sub Page_Load(sender as Object, e as EventArgs)
  2. Dim RollStatus
  3. Const strURL As String = "http://tds600/servlet/PEBServlet?
  4. pag=info"
  5.  
  6.  
  7. If Cache.Get("RollState") IsNot Nothing Then
  8. RollStatus = Cache.Get("RollState")
  9. Response.Write("retrieved from cache")
  10. End If
  11.  
  12. Else
  13. response.write ("Regenerating Data")
  14. Dim PrintExecPage As String
  15. Dim req As HttpWebRequest = WebRequest.Create(strURL)
  16. req.Timeout = 1000
  17.  
  18. Dim resp As HttpWebResponse = req.GetResponse()
  19.  
  20. Dim sr As New StreamReader(resp.GetResponseStream())
  21. PrintExecPage = sr.ReadToEnd()
  22.  
  23. Dim RegEx As New Regex("<table width='100%' border='0'
  24. cellpadding='2' cellspacing='0'><tr>(.*?)Feeder</td>(.*?)</table>",
  25. RegexOptions.None)
  26.  
  27. Dim objmatch As Match
  28.  
  29. For Each objmatch In RegEx.Matches(PrintExecPage)
  30.  
  31. 'filter out the shit, store it in RollStatus string
  32.  
  33.  
  34. Next
  35.  
  36. If Len(RollStatus) 20 Then
  37. Cache.Insert("RollState", RollStatus, Nothing,
  38. DateTime.Now.AddMinutes(3), TimeSpan.Zero)
  39. response.write ("Cache Written")
  40.  
  41. End If
  42.  
  43. sr.Close()
  44. resp.Close()
  45.  
  46.  
  47. End If
  48.  
  49.  
  50. If RollStatus Is Nothing Then
  51. RollStatus = "Did not load."
  52. End If
  53.  
  54.  
  55. lblHTMLOutput.Text = RollStatus
  56.  
  57. End Sub
  58.  
whenever the stream actually loads, it tells me its regenerating data,
and that its written to cache, then I hit refresh, and it says the
same thing - no recall from cache.
Any ideas? what have I done wrong? Is it possible our web server is
deleting cache, is there any way to check?

Jun 25 '07 #1
0 1332

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
5732
by: Roland Hall | last post by:
Am I correct in assuming screen scraping is just the response text sent to the browser? If so, would that mean that this could not be screen scraped? function moi() { var tag = '<a href='; var tagType1 = '"mail'+'to:', tagType2 = '">', tagType3 = '<\/a>'; var user1 = 'web', user2 = 'master', user3 = '@'; var dom1 = 'danger', dom2 = 'ous', dom3 = 'ly'; var tld = '.us';...
1
1552
by: niv | last post by:
Hello, I would like to screen scrape certain parts of a webpage...how can I do this in asp.net For instance.... a stockticker thats embeded on a webpage.. I dont want the entire page.. I would like just the stock ticker...
4
3453
by: rachel | last post by:
Hello, I am currently contracted out by a real estate agent. He has a page that he has created himself that has a list of homes.. their images and data in html format. He wants me to take this page and reformat it so that it looks different. Do I use screen scraping to do this? Could someone please point me to a good screen scraping
2
386
by: Steve | last post by:
I am working on an application to screen scrape information from a web page. I have the base code working but the problem is I have to login before I can get the info I need. The page is hosted on my Router. When I go to the IP of the router I get the following page. <HTML> <head> <meta http-equiv="content-type" content="text/html;charset=iso-8859-1"> <title>Login</title> </head>
0
8386
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8903
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8592
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8661
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7421
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6216
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5686
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4213
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
2044
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.