473,395 Members | 1,783 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,395 software developers and data experts.

Streamreader Alternative

I am writing an ASP page with a C# backend. This page uses a
StreamReader to input a file and pull out a specific section of the
text based on markers inside. While the code I have works, I get a 404
error on files larger than 10 mbs, which will be a common occurence. I
understand this might be a question better suited for an ASP forum, but
I was wondering if anyone out there knows of an alternative to the
StreamReader which will help me avoid this timout issue.

Thanks.

Aug 14 '06 #1
9 5535
Matt Bailey <ba*********@gmail.comwrote:
I am writing an ASP page with a C# backend. This page uses a
StreamReader to input a file and pull out a specific section of the
text based on markers inside. While the code I have works, I get a 404
error on files larger than 10 mbs, which will be a common occurence. I
understand this might be a question better suited for an ASP forum, but
I was wondering if anyone out there knows of an alternative to the
StreamReader which will help me avoid this timout issue.
I doubt that the issue is really with StreamReader - that's pretty
fast.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Aug 14 '06 #2
StreamReader itself should be quite capable of handling the file. How are
you parsing the file? Reading line by line? Reading the whole thing at
once? How are you then "separating" out the section you need?

Also, just to clarify - you say you are getting a 404 error. That is a
"Page Not Found" error, I don't believe a timeout would throw this...

--
Adam Clauss

"Matt Bailey" <ba*********@gmail.comwrote in message
news:11*********************@i42g2000cwa.googlegro ups.com...
>I am writing an ASP page with a C# backend. This page uses a
StreamReader to input a file and pull out a specific section of the
text based on markers inside. While the code I have works, I get a 404
error on files larger than 10 mbs, which will be a common occurence. I
understand this might be a question better suited for an ASP forum, but
I was wondering if anyone out there knows of an alternative to the
StreamReader which will help me avoid this timout issue.

Thanks.

Aug 14 '06 #3
Here is a snippit of the code...

private Hashtable readPDFile(string pdFile)
{
StreamReader reader = new StreamReader(File.OpenRead(pdFile));
Hashtable pdList = new Hashtable();

while (reader.ReadLine().Trim() != "NCE"){}

ArrayList entries = new ArrayList();
string entry;

entry = reader.ReadLine();

while(entry != "ZZZZ")
{
if(!entry.StartsWith("&"))
entries.Add(entry.Substring(0, 8));

entry = reader.ReadLine().Trim();
}

entries.Sort();

foreach(object o in entries)
{
string key = o.ToString().Substring(0, 7);

if(pdList.ContainsKey(key))
pdList[key] = Convert.ToInt32(pdList[key]) + 1;
else
pdList.Add(key, 1);
}

reader.Close();

return pdList;
}

Essentially, I am reading through a file which is seperated into
"chapters", which are denoted by headers. The section I want starts
with the header "NCE", so I am reading through the lines until I hit
that marker, and then read from the file until I hit the end of the
chapter, marked by "ZZZZ". From there, I parse through the list and
count the number of entries for each value, ignoring the final
character. In the example of the file that doesn't work, that is about
20,000 records, which this program can handle. The problem lies in
where the file itself contains more data further on in additional
chapters, which is padding the filesize to 10 mb. This code works with
smaller files with more records being read in (approx 30,000), so I
know it's the additional chapters that are the culprit.

And you are right, I am not getting a 404 error, but a "The page cannot
be displayed. The page you are looking for is currently unavailable.
The Web site might be experiencing technical difficulties, or you may
need to adjust your browser settings." error.

Hope this makes things clearer.

Thanks!

Adam Clauss wrote:
StreamReader itself should be quite capable of handling the file. How are
you parsing the file? Reading line by line? Reading the whole thing at
once? How are you then "separating" out the section you need?

Also, just to clarify - you say you are getting a 404 error. That is a
"Page Not Found" error, I don't believe a timeout would throw this...

--
Adam Clauss

"Matt Bailey" <ba*********@gmail.comwrote in message
news:11*********************@i42g2000cwa.googlegro ups.com...
I am writing an ASP page with a C# backend. This page uses a
StreamReader to input a file and pull out a specific section of the
text based on markers inside. While the code I have works, I get a 404
error on files larger than 10 mbs, which will be a common occurence. I
understand this might be a question better suited for an ASP forum, but
I was wondering if anyone out there knows of an alternative to the
StreamReader which will help me avoid this timout issue.

Thanks.
Aug 14 '06 #4
You might want to try turning off page buffering in your ASP.NET page
to see if that helps. It's turned on by default, and a 10Mb file will
likely be over that threshold.

Jono

Matt Bailey wrote:
Here is a snippit of the code...

private Hashtable readPDFile(string pdFile)
{
StreamReader reader = new StreamReader(File.OpenRead(pdFile));
Hashtable pdList = new Hashtable();

while (reader.ReadLine().Trim() != "NCE"){}

ArrayList entries = new ArrayList();
string entry;

entry = reader.ReadLine();

while(entry != "ZZZZ")
{
if(!entry.StartsWith("&"))
entries.Add(entry.Substring(0, 8));

entry = reader.ReadLine().Trim();
}

entries.Sort();

foreach(object o in entries)
{
string key = o.ToString().Substring(0, 7);

if(pdList.ContainsKey(key))
pdList[key] = Convert.ToInt32(pdList[key]) + 1;
else
pdList.Add(key, 1);
}

reader.Close();

return pdList;
}

Essentially, I am reading through a file which is seperated into
"chapters", which are denoted by headers. The section I want starts
with the header "NCE", so I am reading through the lines until I hit
that marker, and then read from the file until I hit the end of the
chapter, marked by "ZZZZ". From there, I parse through the list and
count the number of entries for each value, ignoring the final
character. In the example of the file that doesn't work, that is about
20,000 records, which this program can handle. The problem lies in
where the file itself contains more data further on in additional
chapters, which is padding the filesize to 10 mb. This code works with
smaller files with more records being read in (approx 30,000), so I
know it's the additional chapters that are the culprit.

And you are right, I am not getting a 404 error, but a "The page cannot
be displayed. The page you are looking for is currently unavailable.
The Web site might be experiencing technical difficulties, or you may
need to adjust your browser settings." error.

Hope this makes things clearer.

Thanks!

Adam Clauss wrote:
StreamReader itself should be quite capable of handling the file. How are
you parsing the file? Reading line by line? Reading the whole thing at
once? How are you then "separating" out the section you need?

Also, just to clarify - you say you are getting a 404 error. That is a
"Page Not Found" error, I don't believe a timeout would throw this...

--
Adam Clauss

"Matt Bailey" <ba*********@gmail.comwrote in message
news:11*********************@i42g2000cwa.googlegro ups.com...
>I am writing an ASP page with a C# backend. This page uses a
StreamReader to input a file and pull out a specific section of the
text based on markers inside. While the code I have works, I get a 404
error on files larger than 10 mbs, which will be a common occurence. I
understand this might be a question better suited for an ASP forum, but
I was wondering if anyone out there knows of an alternative to the
StreamReader which will help me avoid this timout issue.
>
Thanks.
>
Aug 14 '06 #5
Matt Bailey <ba*********@gmail.comwrote:
Here is a snippit of the code...
Rather than a snippet, have you tried producing a short but complete
program which demonstrates the problem? What happens if you try to call
the same method from a console app which just passes in the filename
from the command line?

A few pointers on the code, btw:

1) You're not using either a "using" statement or try/finally with the
StreamReader, so if anything goes wrong the file will stay open until
the GC kicks in and finalizes the underlying stream.

2) Your ToString call in the foreach is unnecessary - just declare o as
a string instead of as an object.

3) It's not clear why you're building a list in the first place - why
not just add them straight to the hashtable?

Finally, if you can't reproduce the problem in a console app, add some
logging to the method to see how far it's actually getting and how long
it's taking to get there.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Aug 14 '06 #6
Hi Jon, thanks for the replies.

The reason I am going to a list is so that I can sort the entries. The
Hashtable doesn't have that functionality as far as I know. As for the
try/catch, it's there, but not in the snippet. The code works in a
console app, but not over the web, which is where my problem lies.

Jon wrote:
Matt Bailey <ba*********@gmail.comwrote:
Here is a snippit of the code...

Rather than a snippet, have you tried producing a short but complete
program which demonstrates the problem? What happens if you try to call
the same method from a console app which just passes in the filename
from the command line?

A few pointers on the code, btw:

1) You're not using either a "using" statement or try/finally with the
StreamReader, so if anything goes wrong the file will stay open until
the GC kicks in and finalizes the underlying stream.

2) Your ToString call in the foreach is unnecessary - just declare o as
a string instead of as an object.

3) It's not clear why you're building a list in the first place - why
not just add them straight to the hashtable?

Finally, if you can't reproduce the problem in a console app, add some
logging to the method to see how far it's actually getting and how long
it's taking to get there.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Aug 15 '06 #7
I also neglected to mention in the previous post that I have tried to
do some debugging using both logging and breakpoints in the code, and
they never go off....

Aug 15 '06 #8
And in response to Jono, I have tried setting the Page.Response.Buffer
equal to false, assuming that's the buffer you are refering to. That
is to no avail.

Aug 15 '06 #9
Matt Bailey wrote:
I also neglected to mention in the previous post that I have tried to
do some debugging using both logging and breakpoints in the code, and
they never go off....
That's the first thing to fix then. You say it's working for small
files - so you should be able to get logging etc working on small
files. You can then try with the large file.

My guess is that you'll find your problem isn't in the area you think
it is.

Jon

Aug 15 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: oafyuf | last post by:
Hi, I'm having performanbce issues with StreamReader and was wondering what I could do to improve it... The following takes around 3 seconds to process! The content of the response is: ...
2
by: Keith Kingsley | last post by:
I'm using a StreamReader to read in several lines from an ASCII file. I'd like to know the StreamReader's "true" position-- that is, the number of bytes into the file that the StreamReader has...
3
by: Arno | last post by:
Hi, I'm using TcpClient for communication between two PC running a small piece of software. The protocol used has been designed internally and is HTTP similar (command line, headers, body). A...
2
by: Ian Oldbury | last post by:
I'm having a problem reading from a flat file, in the file "£" and "»" exist however when i view the contents of the variable LineContents these characters don't exist. Has anyone got any...
1
by: Matt Bailey | last post by:
Hi, I have an ASPX page with a C# backend in which I use a StreamReader to input a file so that I can parse through it. While the code I have works, I get a 404 error when I try to use an input...
3
by: Arpan | last post by:
A file can be read using only the StreamReader object like this: Dim sReader As StreamReader sReader = New StreamReader(Server.MapPath("File1.txt")) While(sReader.Peek -1)...
1
by: garyusenet | last post by:
>From MSDN I'm trying to learn more about streamreader. I'm working my way down the MSDN definition. The first entry is the Public Constructor StreamReader. I'm fairly happy I have a basic grasp on...
1
by: Sladan | last post by:
Im trying to read a xml-file with a StreamReader. For the moment I'm using the following code. streamReader = new StreamReader(stream, System.Text.Encoding.Default); string feedData =...
0
by: rajana | last post by:
Dear All, We have Ansi file with german characters (Ä / Ø) , We are using Streamreader to read the contents of the file. But Readline() not able to read the German characters. We tried all...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.