473,699 Members | 2,576 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

FSO + XMLHTTP + reading large files + errr....

The application;

Service on my webserver that allows a user to upload their HOSTS file for functions to verify the contents are still valid.

Uses;

1. XMLHTTP (MSXML2)
2. FileSystemObjec t
3. CrazyBeavers Upload control (couldn't get the Dundas one to work)

How it's supposed to work;

1. User uploads file (test file = 1.10MB)
2. FSO saves file to server then prepares for reading
3. File is opened using the Read(n) method (line # 45 in res_upload.txt)
4. Content is parsed and the parts required, passed to a function (URLExists)
5. Status saved to report to later show to client

Files:

Test file: hpHosts - http://www.hosts-file.net

Example report: http://mysteryfcm.plus.com/?mode=Hos...0hosts.txt.htm

Code that uploads then processes the file: http://mysteryfcm.plus.com/res_upload.txt

The problem;

The file can contain anything from 10 lines to 20,000+ lines, each one is parsed and passed to the URLExists function. I'm not entirely sure whether the problem is due to the content itself, or the number of calls to the XMLHTTP object but, a file containing 10,000 lines (the test file) times out after around 1,000 lines.

The timeout set for the URLExists function is 5 seconds (the timeout takes well over an hour of reading/parsing, to occur), the script timeout is set to 5000

I've tried cutting alot of the content of the test file out so it's 206K instead of 1.10MB, but it's still taking forever to process, and then timing out anyway.

The question;

I've found some doc's online that show how to read and parse large files with a ton of content and have applied this to the application in question but for reasons best known to itself, it is still timing out. What I'm wondering is;

1. Would it be best to ditch the application server-side and make it a downloadable application instead?

It would be easier to work with in VB but thats the reason I didn't want to do it that way (I like a challenge).

2. Would it be viable to split the file into parts once uploaded, and process each part seperately instead of doing it the way I'm currently doing it?.

3. Is there a better alternative that I've simply not thought of?

I've probably not given enough info and apologies in advance if this is the case (got a million and one things going through my head atm). Thanks in advance for any advice/suggestions.

--
Regards

Steven Burn
Ur I.T. Mate Group
www.it-mate.co.uk

Keeping it FREE!
Jul 22 '05 #1
3 2283
[I don't understand why my news reader won't prefix the lines from the OP...
sorry for any confusion... I prefixed the short sections by hand...]
>>>>>>>>>>> >>>> "Steven Burn" <so*******@in-time.invalid> wrote in message
news:uT******** ******@TK2MSFTN GP15.phx.gbl...
The application;

Service on my webserver that allows a user to upload their HOSTS file for
functions to verify the contents are still valid.

[snip]

The problem;

The file can contain anything from 10 lines to 20,000+ lines, each one is
parsed and passed to the URLExists function. I'm not entirely sure whether
the problem is due to the content itself, or the number of calls to the
XMLHTTP object but, a file containing 10,000 lines (the test file) times out
after around 1,000 lines.

The timeout set for the URLExists function is 5 seconds (the timeout takes
well over an hour of reading/parsing, to occur), the script timeout is set
to 5000
I've tried cutting alot of the content of the test file out so it's 206K
instead of 1.10MB, but it's still taking forever to process, and then
timing
out anyway.

<<<<<<<<<<<<<<< <<<<

Are there really boxes out there with HOSTS files anywhere even near that
long? We're talking %windir%\system 32\drivers\etc\ HOSTS, yeah? Any time
I'm even tempted to put more than 25 lines in HOSTS [esp. the same
entries on more than one PC] I find somewhere reasonably convenient to
install BIND! Don't like 99.5% of the HOSTS files out there have just one
line?

127.0.0.1 localhost

Sorry if all that's beside the point, I'm mostly just curious as to whether
or not this commonly exists, and if so, why?


>>>>>>>> >>>>>> The question;

I've found some doc's online that show how to read and parse large files
with a ton of content and have applied this to the application in question
but for reasons best known to itself, it is still timing out. What I'm
wondering is;

1. Would it be best to ditch the application server-side and make it a
downloadable application instead?

<<<<<<<<<<<<<<< <<<<<<<

I would think so. You don't really care about the contents, do you? Aren't
you really only interested in whether or not the file has changed? I can
see potential value in storing the last confirmed copy off of the local
machine, to prevent tampering, but wouldn't a CRC be just a valid of a
check, without all the muss and fuss?

I would also consider setting a file system change hook, and then comparing
the contents [CRC] to a non-locally stored value just once per session, to
verify that it was not changed while your app wasn't running. After that,
you'll know instantly when any other changes are made.

Long story short, the only real value a server-side process can add to this
paradigm is off-site storage.

>>>>>>>>>>> >>>>>>>>>

It would be easier to work with in VB but thats the reason I didn't
want
to do it that way (I like a challenge).

2. Would it be viable to split the file into parts once uploaded, and
process each part seperately instead of doing it the way I'm currently
doing
it?.

3. Is there a better alternative that I've simply not thought of?

<<<<<<<<<<<<<<< <<<<<<<<<<<

If you really have a good reason to parse and store the contents
entry-for-entry, then I'd store them in a database and leverage that
technology, that's by far the easiest way to make this kind of thing scale.
-Mark

Jul 22 '05 #2
Gazing into my crystal ball I observed "Mark J. McGinty"
<mm******@spamf romyou.com> writing in
news:eu******** ******@TK2MSFTN GP14.phx.gbl:
Are there really boxes out there with HOSTS files anywhere even near
that long? We're talking %windir%\system 32\drivers\etc\ HOSTS, yeah?
Any time I'm even tempted to put more than 25 lines in HOSTS [esp. the
same entries on more than one PC] I find somewhere reasonably
convenient to install BIND! Don't like 99.5% of the HOSTS files out
there have just one line?

127.0.0.1 localhost

Sorry if all that's beside the point, I'm mostly just curious as to
whether or not this commonly exists, and if so, why?


My hosts file is 6351 lines. It contains listings of bad hosts, spyware
hosts, advertising hosts, etc. Because of this I see very little ads, and
have had no problems with spyware or viruses for at least five years. I go
to web sites with third party ads and see a nice, friendly 404. I even
changed my 404 to read "Doh! The website cannot be found" in red.

You might want to take a look at
<http://www.mvps.org/winhelp2002/hosts.htm> to see how this can help your
system.

Although my Hosts file is long, I have no lag time in requesting a page
that is not on the list. I love it!

--
Adrienne Boswell
http://www.cavalcade-of-coding.info
Please respond to the group so others can share
Jul 22 '05 #3
Thanks for your comments.

The reasoning for this is quite simply because of the hpHosts & mvps Hosts etc files, that contain server IP's/URL's that are no longer valid (my app simply detects and reports their validity). The contents themselves will not be stored unless the user asks my app to do so.

--
Regards

Steven Burn
Ur I.T. Mate Group
www.it-mate.co.uk

Keeping it FREE!

"Mark J. McGinty" <mm******@spamf romyou.com> wrote in message news:eu******** ******@TK2MSFTN GP14.phx.gbl...
[I don't understand why my news reader won't prefix the lines from the OP...
sorry for any confusion... I prefixed the short sections by hand...]
>>>>>>>>>>>> >>>> "Steven Burn" <so*******@in-time.invalid> wrote in message
news:uT******** ******@TK2MSFTN GP15.phx.gbl...
The application;

Service on my webserver that allows a user to upload their HOSTS file for
functions to verify the contents are still valid.

[snip]

The problem;

The file can contain anything from 10 lines to 20,000+ lines, each one is
parsed and passed to the URLExists function. I'm not entirely sure whether
the problem is due to the content itself, or the number of calls to the
XMLHTTP object but, a file containing 10,000 lines (the test file) times out
after around 1,000 lines.

The timeout set for the URLExists function is 5 seconds (the timeout takes
well over an hour of reading/parsing, to occur), the script timeout is set
to 5000
I've tried cutting alot of the content of the test file out so it's 206K
instead of 1.10MB, but it's still taking forever to process, and then
timing
out anyway.

<<<<<<<<<<<<<<< <<<<



Are there really boxes out there with HOSTS files anywhere even near that
long? We're talking %windir%\system 32\drivers\etc\ HOSTS, yeah? Any time
I'm even tempted to put more than 25 lines in HOSTS [esp. the same
entries on more than one PC] I find somewhere reasonably convenient to
install BIND! Don't like 99.5% of the HOSTS files out there have just one
line?

127.0.0.1 localhost

Sorry if all that's beside the point, I'm mostly just curious as to whether
or not this commonly exists, and if so, why?


>>>>>>>>>> >>>>>>

The question;

I've found some doc's online that show how to read and parse large files
with a ton of content and have applied this to the application in question
but for reasons best known to itself, it is still timing out. What I'm
wondering is;

1. Would it be best to ditch the application server-side and make it a
downloadable application instead?

<<<<<<<<<<<<<<< <<<<<<<



I would think so. You don't really care about the contents, do you? Aren't
you really only interested in whether or not the file has changed? I can
see potential value in storing the last confirmed copy off of the local
machine, to prevent tampering, but wouldn't a CRC be just a valid of a
check, without all the muss and fuss?

I would also consider setting a file system change hook, and then comparing
the contents [CRC] to a non-locally stored value just once per session, to
verify that it was not changed while your app wasn't running. After that,
you'll know instantly when any other changes are made.

Long story short, the only real value a server-side process can add to this
paradigm is off-site storage.



>>>>>>>>>>>> >>>>>>>>>

It would be easier to work with in VB but thats the reason I didn't
want
to do it that way (I like a challenge).

2. Would it be viable to split the file into parts once uploaded, and
process each part seperately instead of doing it the way I'm currently
doing
it?.

3. Is there a better alternative that I've simply not thought of?

<<<<<<<<<<<<<<< <<<<<<<<<<<



If you really have a good reason to parse and store the contents
entry-for-entry, then I'd store them in a database and leverage that
technology, that's by far the easiest way to make this kind of thing scale.


-Mark






Jul 22 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
12449
by: Dan Sikorsky | last post by:
Uploading from browser to server using Msxml2.XMLHTTP takes a long time about 15 minutes for a 1.5MB file at 37.2Kbps, although it does get there. Is there anyway to speed things up? here's the code ... var sFilename = (document.form1.myFileName.value); // create ADO-stream Object var ado_stream = new ActiveXObject("ADODB.Stream"); // create XML document with default header and primary node
7
2500
by: Fabri | last post by:
I'm trying to develop a way to include static files in htm pages with javascript. I'm trying to use XMLHTTP object this way: ---------------------------------------------------------------------- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head>
3
2076
by: Brad | last post by:
I'm working on a web app which will display LARGE tiff image files (e.g files 10-20+ mb). Files are hidden from users direct access. For other, smaller image files I have used FileStream to read in a file in a single Read and so my quesitons are: (1) What is a practical file size limit for reading using FileStream.Read (reading the file in a single read)...especially on a web server where I don't think I'd want to tax memory...
1
4923
by: Ike | last post by:
Ive copied an online example for writing out a php file, programmatically, then would like to re-display that data in a browswer window that automatically refreshes as the data file (getdata.php, which is the file I am programmatically rewriting to elsewhere) gets changed. I am trying to use AJAX and JSON to do this. I have copied an example of using HttpRequest Object as the backbone of this from...
1
7401
by: KoosJaspers | last post by:
I have a remarkable problem. Opening a file using xmlhttp works perfectly. The responseText output is read, since it can be assigned to an alert() message, as follows : alert(xmlhttp.resposeText) However, xmlhttp.responseTest can NOT be assigned to a normal string variable, UNLESS the file is already present in the temporary internet folder. This problem only occurs in the remote webserver context, not on my local system. It seems that a...
5
10770
by: Prasad | last post by:
Hi all, I wanted to know whether my page is connecting to server or not using xmlHttp request. For that , I am using one condition in onreadystatechange function as.. function xxx() { if(xmlhttp.readyState==4) {
5
14985
blazedaces
by: blazedaces | last post by:
Ok, so you know my problem, java is running out of memory reading with SAX, the event-based xml parser intended more-so than DOM for extremely large files. I'll try to explain what I've been doing and why I have to do it. Hopefully someone has a suggestion... Alright, so I'm using a gps-simulation program that outputs gps data, like longitude, lattitude, altitude, etc. (hundreds of terms, these are just the well known ones). In the newer...
7
2149
by: Dan Beanweed | last post by:
I am experimenting with XMLHTTP in a personal website. If I can understand it I would like to use it in a SVG application at work. But I don't get what's going on with my code, especially after reading the posts on this and other fora. I create an object using "xmlhttp = new XMLHttpRequest();", no problem. Then I have this script: function getContent(file){ if(file==""){ return; } xmlhttp.onReadyStateChange=stateChange();...
17
9922
by: byte8bits | last post by:
How does C++ safely open and read very large files? For example, say I have 1GB of physical memory and I open a 4GB file and attempt to read it like so: #include <iostream> #include <fstream> #include <string> using namespace std; int main () {
0
8705
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8628
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
8943
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7785
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6550
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5884
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
3075
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2362
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2016
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.