out of Memory execption

[Ganesan selvaraj]

I using C# .net. i want to split the text files based of the some condition.
my source text file size may be 4 kb to 4 gb. some time when i split the
i got the "out of memory exception. when i read from the file i read the
10mb and split the this content and clear the buffer and read the second 10
mb and so on.

How can i rectify the problem?

[Oliver Sturm]

Ganesan selvaraj wrote:

I using C# .net. i want to split the text files based of the some
condition.
my source text file size may be 4 kb to 4 gb. some time when i split
the
i got the "out of memory exception. when i read from the file i read the
10mb and split the this content and clear the buffer and read the second
10
mb and so on.

How can i rectify the problem?

Please show us the important parts of the code you're using.
Oliver Sturm
--
omnibus ex nihilo ducendis sufficit unum
Spaces inserted to prevent google email destruction:
MSN oliver @ sturmnet.org Jabber sturm @ amessage.de
ICQ 27142619 http://www.sturmnet.org/blog

[Ganesan selvaraj]

Hi Oliver
Here with i send the impotent code. Bellow i have given 2 methods.
i am getting this execption in these 2 methods.

In the FileReadIntoMemory method i read the 10mb of data from the source
file and i pass it to Split_Files method.

In the Split_Files method i am spliting the string and creating the small
files. and i am applying some properties to that method.

if you need any more info. pls let me know.

private bool FileReadIntoMemory(string filename, int
bufferSize)//,StringBuilder stringBuilder,long idx)
{
try
{

boolDeleteFile=true;
StringBuilder stringBuilder=new StringBuilder();
StringBuilder SBDreContent=new StringBuilder();
StringBuilder SBCntContent=new StringBuilder();
StringBuilder SBCntFilePath=new StringBuilder();
StringBuilder SBLogPath=new StringBuilder();

FileInfo fileInfo = new FileInfo(filename.Trim());
FileStream fileStream = new FileStream(filename.Trim(), FileMode.Open,
FileAccess.Read);
StreamReader streamReader = new StreamReader(fileStream);
string strLogcontent="";

// Reduce the buffer to fit the filesize, if the buffer is less than the
file
if ( (int) fileInfo.Length < bufferSize )
bufferSize = (int) fileInfo.Length;

char[] fileContents = new char[bufferSize];
int bytesRead = streamReader.Read(fileContents, 0, bufferSize );

// Can't do much with 0 bytes
if ( bytesRead == 0 )
throw new Exception("File is 0 bytes");

//StringBuilder stringBuilder = new StringBuilder();

int lastRead=0;
//int cnt=0;

while( bytesRead > 0 )
{

try
{

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<FileR eadIntoMemory>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<FileR eadIntoMemory>===========================>");

}
}
stringBuilder.Append(fileContents,0,bytesRead);
lastRead=bytesRead;
strLogcontent=Split_Files(stringBuilder,strLogcont ent,SBDreContent,SBCntContent,SBCntFilePath,SBLogP ath);
fileContents=null;
fileContents = new char[bufferSize];
bytesRead = streamReader.Read( fileContents, 0, bufferSize );
}

catch(Exception Ex)
{

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<FileR eadIntoMemory
Exception>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<FileR eadIntoMemory
Exception>===========================>");
}
}
Trace.WriteLine("Exception: " + Ex.ToString() + " :
method:FileReadIntoMemory : "
+ DateTime.Now.ToString());
return false;
}

}
if (bytesRead ==0)
{
boolFileReadStatus=false;
}

if(stringBuilder.Length>0)
{

strLogcontent=Split_Files(stringBuilder,strLogcont ent,SBDreContent,SBCntContent,SBCntFilePath,SBLogP ath);
}
createCNTFile( SBDreContent, SBCntContent,SBCntFilePath);
if (SBLogPath.Length>0)
{
WriteAllToDreFile(strLogcontent, SBLogPath.ToString());
SBLogPath.Remove(0,SBLogPath.Length);
}

SBDreContent=null;
SBCntContent=null;
SBCntFilePath=null;
stringBuilder=null;
streamReader.Close();
fileStream.Close();

//return stringBuilder;
}

catch(Exception Ex)
{
Trace.WriteLine("Exception: " + Ex.ToString() + " :
method:FileReadIntoMemory : "
+ DateTime.Now.ToString());
return false;
}

return true;
}


public string Split_Files(StringBuilder stringBuilder,string
Logcontent,StringBuilder SBDreContent,StringBuilder
SBCntContent,StringBuilder SBCntFilePath,StringBuilder SBLogPath)
//public bool Split_Files(string entiredata)
{
Regex reDreStart = new
Regex(ConfigurationSettings.AppSettings["SplitterBookmark1"].ToString()
+".*");
Regex reDreEnd = new
Regex(ConfigurationSettings.AppSettings["SplitterBookmark3"].ToString() );
Regex reDreContent = new
Regex(ConfigurationSettings.AppSettings["SplitterBookmark2"].ToString());
Regex reDreDelete = new
Regex(ConfigurationSettings.AppSettings["DeletedFiles"].ToString() +".*");
//this reDreSection for sectioning
Regex reDreSection = new
Regex(ConfigurationSettings.AppSettings["DreSection"].ToString() +".*");

MatchCollection mcDreStart = reDreStart.Matches(stringBuilder.ToString()
);
MatchCollection mcDreDreEnd = reDreEnd.Matches(stringBuilder.ToString());

string URLLog=null;
string strLogcontent=Logcontent;
int LastPos=stringBuilder.Length;
if (mcDreStart.Count>mcDreDreEnd.Count)
{
LastPos=mcDreStart[mcDreDreEnd.Count].Index ;
}

for(int i=0;i<mcDreStart.Count ;i++)
{
string dreDoc="";
string FileName="";
string cntDoc="";

try
{

GC.Collect();
}
catch(Exception Ex1)
{

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
}
}
Trace.WriteLine("Exception: " + Ex1.Message + " : Delete ref"
+ DateTime.Now.ToString());
}

try
{

if(i<mcDreDreEnd.Count)
{
//this mcDreSection for sectioning
int intStction=0;
dreDoc=stringBuilder.ToString().Substring(
mcDreStart[i].Index,mcDreDreEnd[i].Index-mcDreStart[i].Index);
MatchCollection mcDreSection= reDreSection.Matches(dreDoc);
if(mcDreSection.Count>0)
{
intStction=Convert.ToInt32(mcDreSection[0].ToString().Replace(ConfigurationSettings.AppSetti ngs["DreSection"].ToString(),null).Replace("\n",null).Replace("\r", null));
}
else
{
intStction=0;
}

if (intStction==0)
{

if(mcDreStart[i].ToString().IndexOf("www.mpa.gov.sg.")>=0)
{
int gh=1;
}

FileName=getDrePath(mcDreStart[i].ToString().Replace(@"\",@"/").Replace('\r',' ').Trim () );
FileName=FileName.Replace(@"\\",@"\");
if(i==0 && SBLogPath.Length==0 )
{

URLLog = ConfigurationSettings.AppSettings["IDXesFilePath"] +
GetDomainFolder(FileName.Replace("Y:" +
ConfigurationSettings.AppSettings["BranchVPath"].ToString().Replace("/",@"\")
, ""));

if(File.Exists(URLLog))
{
StreamReader ObjSRLog = new StreamReader(URLLog,
System.Text.Encoding.UTF8);
strLogcontent=ObjSRLog.ReadToEnd();
ObjSRLog.Close();
ObjSRLog=null;
}
SBLogPath.Append(URLLog);
}
if(JustTheFileName(FileName,false).ToUpper().Index Of(@"\CON.")==0)
{
FileName=FileName.Replace("CON.","CON_").Replace(" con.","con_");
}
}//if (intStction==0)
else
{
FileName=SBCntFilePath.ToString();
}//else of if(intStction==0)
MatchCollection mcDreContent = reDreContent.Matches(dreDoc);


if(mcDreContent.Count>0)
{
cntDoc=dreDoc.Substring(mcDreContent[0].Index+
mcDreContent[0].ToString().Length) ;
dreDoc=dreDoc.Substring(0,mcDreContent[0].Index+
mcDreContent[0].ToString().Length+1);
}
/// HERE IF THE FILE NAME STARTS WIHT "CON." WE CHANGE TO "CON_".

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<Split _Files>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" +
p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<Split _Files>===========================>");
}
}
if(intStction==0)
{
createCNTFile( SBDreContent, SBCntContent,SBCntFilePath);
if( SBDreContent.Length>0)
SBDreContent.Remove(0,SBDreContent.Length);
if( SBCntContent.Length>0)
SBCntContent.Remove(0,SBCntContent.Length);
if( SBCntFilePath.Length>0)
SBCntFilePath.Remove(0,SBCntFilePath.Length);
SBDreContent.Append(dreDoc);
SBCntContent.Append(cntDoc);
SBCntFilePath.Append(FileName+".cnt");
}
else
{
SBCntContent.Append(cntDoc);

}

if(strLogcontent.ToUpper().IndexOf((FileName+".cnt ").ToUpper())<0 &&
intStction==0)//&&
strLogcontent.IndexOf(FileName.Replace(Configurati onSettings.AppSettings["IDXesFilePath"],ConfigurationSettings.AppSettings["TeamSiteBasePath"]) +".cnt")<0)
{
TotalCnt++;
strLogcontent+=FileName+".cnt\n";

}
dreDoc=null;
cntDoc=null;

}
}
catch(Exception Ex)

{

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
}
}
Trace.WriteLine("Exception: " + Ex.Message + " : Filename: " + FileName
+ DateTime.Now.ToString());
}

}

try
{
MatchCollection mcDreDelete =
reDreDelete.Matches(stringBuilder.ToString());
string strDeletedDoc="";
for(int i=0;i<mcDreDelete.Count ;i++)
{
strDeletedDoc+=mcDreDelete[i].ToString()+"\n";
//SBDeleteContent.Add(mcDreDelete[i].ToString());
}

//this is new code for delteted files.
if(strDeletedDoc.Trim()!="")
{
strLogcontent=DeleteURLFiles(strDeletedDoc,strLogc ontent);
}

// if (URLLog==null || URLLog.Trim()=="" )
// { string LineString1="";
// Regex reDreDelete1 = new
Regex(ConfigurationSettings.AppSettings["DeletedFiles"].ToString() +".*");
// MatchCollection mcDreDelete1 = reDreDelete1.Matches(strDeletedDoc);
// if(mcDreDelete.Count>0)
// {
// LineString1=mcDreDelete1[0].ToString();
// string [] URLArr = LineString1.Split('+');
// string URL = null;
// if(URLArr.Length >0)
// {
//
// URL = HttpUtility.UrlDecode(URLArr[0].ToString().Replace(
// ConfigurationSettings.AppSettings["DeletedFiles"], null));
// URL = URL.Replace(@"?docs=",
null).Replace("https://",null).Replace("http://",null) ;
// string delfolder=URL.Substring(0,URL.IndexOf("/"));
// delfolder=delfolder.Replace("..","__");
// if(!Directory.Exists(ConfigurationSettings.AppSett ings["IDXesFilePath"].ToString() + delfolder) )
// {
// Directory.CreateDirectory(ConfigurationSettings.Ap pSettings["IDXesFilePath"].ToString() + delfolder);
// }
// URLLog=ConfigurationSettings.AppSettings["IDXesFilePath"].ToString() + delfolder + @"\" + delfolder + ".log";
// if(File.Exists(URLLog) && strLogcontent.Trim()=="" )
// {
// StreamReader ObjSRLog = new StreamReader(URLLog,
System.Text.Encoding.UTF8);
// strLogcontent=ObjSRLog.ReadToEnd();
// ObjSRLog.Close();
// ObjSRLog=null;
//
// }
//
//
//
// }
// }
//
// }
// strLogcontent=DeleteURLFiles(strDeletedDoc,strLogc ontent);

// WriteLineToFile(strDeletedDoc,
// ConfigurationSettings.AppSettings["IDXesFilePath"] +
// ConfigurationSettings.AppSettings["DeletedLog"]);
// if (URLLog!=null && URLLog.Trim()!="" )
// {
// if(boolDeleteFile==true)
// {
// WriteAllToDreFile(strDeletedDoc,
// URLLog.Substring(0,URLLog.LastIndexOf("\\")+1)+
// ConfigurationSettings.AppSettings["DeletedLog"]);
// }
// else
// {
// WriteAllToFile(strDeletedDoc,
// URLLog.Substring(0,URLLog.LastIndexOf("\\")+1)+
// ConfigurationSettings.AppSettings["DeletedLog"]);
//
// }
// WriteAllToDreFile(strLogcontent, URLLog);
// }
//
//
// URLLog="";
// URLLog=null;

}
catch(Exception Ex)
{
Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
}
}
Trace.WriteLine("Exception: " + Ex.Message + " : Delete ref"
+ DateTime.Now.ToString());
}
try
{
string tempString=stringBuilder.ToString().Substring(Last Pos);
if(tempString.Length>0)
{
stringBuilder.Replace(tempString,null);
}
stringBuilder.Remove(0,stringBuilder.Length);
stringBuilder.Append(tempString);
}
catch(Exception Ex)
{

Process[] ps= Process.GetProcesses() ;
foreach(Process p in ps)
{
if(p.ProcessName.ToString().Trim().ToUpper().Index Of("CSTMDV4")==0)
{
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
Trace.WriteLine("CSTMDv4 physical memory :" + p.WorkingSet.ToString());
Trace.WriteLine("CSTMDv4 virtual memory :" +
p.VirtualMemorySize.ToString());
Trace.WriteLine("<==========================<Split _Files
Exception>===========================>");
}
}
Trace.WriteLine("Exception: " + Ex.Message + " : Delete ref"
+ DateTime.Now.ToString());
}
//stringBuilder=null;
boolDeleteFile=false;
return strLogcontent;


}

with regards
Ganesan

[Jon Skeet [C# MVP]]

Ganesan selvaraj <Ga*************@discussions.microsoft.com> wrote:

Here with i send the impotent code. Bellow i have given 2 methods.
i am getting this execption in these 2 methods.

In the FileReadIntoMemory method i read the 10mb of data from the source
file and i pass it to Split_Files method.

In the Split_Files method i am spliting the string and creating the small
files. and i am applying some properties to that method.

if you need any more info. pls let me know.

A few suggestions:

1) Your bytesRead variable is misnamed - it's keeping track of
*characters* read, not bytes.

2) You don't need to create a new buffer (fileContents) on each
iteration - just reuse the previous one.

3) You appear to be processing the whole file as read so far in each
call to Split_Files (which would conventionally be called SplitFiles) -
this means that the first buffer's data will be processed on *every*
iteration. Is that really what you want to do? I would expect you to
either process a single buffer at a time, or read the whole thing in
and then process it in one go.

4) I believe you're calling ToUpper on the whole of the log content
read - that's going to be fairly expensive if it's large.

5) You're creating a new set of regular expressions based on the app
settings on every iteration. This is much less efficient than creating
them once and reusing them. You could also make them compiled, to
improve efficiency.

6) You *really* need to break your methods up more - it's very hard to
tell what's going on at the moment.

7) You're using string concatenation in a loop, which is probably
hurting your efficiency.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Oliver Sturm]

Ganesan selvaraj wrote:

Here with i send the impotent code. Bellow i have given 2 methods.
i am getting this execption in these 2 methods.

In the FileReadIntoMemory method i read the 10mb of data from the source
file and i pass it to Split_Files method.

In the Split_Files method i am spliting the string and creating the small
files. and i am applying some properties to that method.

if you need any more info. pls let me know.

So, in addition to everything Jon has already said:

- You're using a char array of the length given by your bufferSize. As you are comparing this value with the one given by FileInfo.Length, I assume that this is supposed to be a size in bytes. As a char has two bytes, your array ends up being double the size you intended it to be. Of course you need the char array for the StreamReader, but you aren't calculating the needed size correctly.

- There are circumstances where the StringBuilder.ToString() method will make a copy of the whole contents before returning a string. At a glance, you are calling this method at least five times during the Split_Files method, so if the content should happen to be copied, that would account for huge memory use.

- Why are you doing things like this: dreDoc=stringBuilder.ToString().Substring(mcDreSta rt[i].Index,mcDreDreEnd[i].Index-mcDreStart[i].Index);
You are using the indexes given by the regular expression match to call
Substring? Why don't you just read the result text out of the regular
expression match, that's what it's for.
Running a regex on such a huge string may not be very efficient
memory-wise, I haven't tested this. But given the fact that you only use
the position where a certain string starts, you'd probably be better of
with String.IndexOf().

So, this is all I can guess at - Jon's right, your code looks horrible,
man :-) Try pulling out all that Process output stuff, getting rid of the
comments, taking out all the stuff you put in only for debugging - it'll
greatly increase the chances of people reading your code and of anybody
actually finding a problem for you.


Oliver Sturm
--
omnibus ex nihilo ducendis sufficit unum
Spaces inserted to prevent google email destruction:
MSN oliver @ sturmnet.org Jabber sturm @ amessage.de
ICQ 27142619 http://www.sturmnet.org/blog

[Ganesan selvaraj]

Hi
acutely i want to split some source file. I have attached the sample file.
in that i want to split the #DRECONTENT part for each url and create a file.
so the file size can be 3 kb to 3 GB.
already i send you the code. in that i am reading 10 mb(configurable)
content and i split that one. then i remove the first 10mb from memory and
read next 10mb content on so on.
This is what my code is doing. when i run this some times i got the out of
memory exception.

when i run this code i log the memory utilization. the virtual memory is
keep on increasing.

my sample input file:
==>test.idx <==
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/index.htm
#DRETITLE
Pulse Of The Web Sponsored Links Sample Web.
#DRESECTION 0
#DREDATE 1121339978
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-1.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Pulse Of The Web Sponsored Links Sample Web."
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/index.htm"
#DREFIELD SPIDERDATE="1121339978"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="0"
#DREFIELD SPIDERPAGELINKS="9"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="2146"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="1741"
#DREFIELD IMPORTMETALEN="102"
#DREFIELD IMPORTLINKLEN="54"
#DREFIELD IMPORTTITLELEN="0"
#DREFIELD IMPORTQUALITY="107"
#DREFIELD DREDATE="1121339978"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-1.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="d1ca19e1b3bac33ba7818d99f4c54952"
#DREFIELD summary="Pulse Of The Web Sponsored Links Sample Web./pages Long
Distance Courses- sample web./pages and chapters This web"
#DRECONTENT
Pulse Of The Web
Sponsored Links
Sample Web./pages
Long Distance Courses- sample web./pages and chapters
This web./page provides a complete list of CSET courses and sample
web./pages and chapters. CSET 1200: GUI Programming w/Visual Basic. NET.
Index of /class/cs454/cs412.sp2000/sample.web./pages
Index of /class/cs454/cs412.sp2000/sample.web./pages. Name Last modified
Size Description Parent Directory - sample.index.html 14-Jan
SEUL: Sample Web./pages for SEUL-PUB (fwd)
SEUL: Sample Web./pages for SEUL-PUB (fwd). To: se******@seul.org; Subject:
SEUL: Sample Web./pages for SEUL-PUB (fwd); From: Roger R Dingledine ;
The We*****@servenet.com. Sample Website ./pages.
Sample Web./pages Just as a house consists of rooms varying in size and
function, a website consists of web./pages varying in length and purpose.
Build Your Own Web ./page
Do It Yourself or Professional Web Design - Build the Site You Want!
Re: SEUL: Sample Web./pages for SEUL-PUB. To: se**********@seul.org;
Subject: Re: SEUL: Sample Web./pages for SEUL-PUB; From: Roger R Dingledine ;
Tamil Script Code for Information Interchange (TSCII) Internet Usage
Guidelines and Sample Web./pages. Tamil Script Code for Information
Interchange (TSCII).
Click Here to go. to Rawhide Design. Home ./page. Sample Web./pages. Below
are some sample WEB./pageS to help give you a better idea of your design
layout choices when ordering your web./page!
If you would like to view some Opening Sample Web./pages. you may wish to
have me program center circle marked. To Sample ./pages or click here: To
Sample ./pages
Some Sample Web./pages.
All trademarks and copyrighted information contained herein are the property
of their respective owners

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page9.htm
#DRETITLE
Page8
#DRESECTION 0
#DREDATE 1121340006
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-10.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page8"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page9.htm"
#DREFIELD SPIDERDATE="1121340006"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="508"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="447"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="112"
#DREFIELD DREDATE="1121340006"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-10.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="0d349c3068ccca79a880fa6020b159e7"
#DREFIELD summary="But you made a mistake yourself! We all do, from time to
time. If you think you’ve found an error in my own writing, first read the
“Commonly Made Suggestions� page, then follow the instructions on that page
if you still think I need correcting"
#DRECONTENT
But you made a mistake yourself!
We all do, from time to time. If you think you’ve found an error in my own
writing, first read the “Commonly Made Suggestions� page, then follow the
instructions on that page if you still think I need correcting. I’ve changed
many aspects of these pages in response to such mail; even if I disagree with
you, I try to do so politely. If you write me, please don’t call me “Brian.�
My given name is Paul.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page1.htm
#DRETITLE
Over 4.4 million visitors since 1997. In its first three y...
#DRESECTION 0
#DREDATE 1121339982
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-2.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Over 4.4 million visitors since 1997. In its first three
y..."
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page1.htm"
#DREFIELD SPIDERDATE="1121339982"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="2"
#DREFIELD SPIDERPAGESIZE="3446"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="3397"
#DREFIELD IMPORTMETALEN="102"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="0"
#DREFIELD IMPORTQUALITY="118"
#DREFIELD DREDATE="1121339982"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-2.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="75a2b7033a1c1ec55c4121243b872140"
#DREFIELD summary="Over 4.4 million visitors since 1997. In its first three
years this site was visited more than a million times"
#DRECONTENT
Over 4.4 million visitors since 1997.
In its first three years this site was visited more than a million times.
Because of various technical and design problems which caused it to lag far
behind the actual traffic, the counter has since been removed from this page.
After the first million hits, a link to a more sophisticated counter was
installed on the next page, where most visitors go directly. Click on the
“Site Meter� icon at the bottom of that page and add 1 million to get a fair
idea of how many people have dropped in. Note that this counter measures
discrete visits, not just “hits.�
If you search for the word “English� in Google, which gives a measure of
popularity by ranking its results in order of the number of links other
people have created to them, my site turns up twice, both high in the list of
hits, because many people still link to the old, many years out-of-date
address www.wsu.edu:8080/~brians/errors/ instead of the correct current
address at www.wsu.edu/~brians/errors/. Add the two together, and this would
seem to be the most commonly linked-to site on the Web for �English.� Thanks,
folks!
Recommended as an “Incredibly Useful Site� in Yahoo Internet Life Magazine ,
July, 1997, pp. 82-83 and cited as a Yahoo “Site of the Week� and recommended
by Netsurfer Digest March 5, 1999. It has also been recommended in the pages
of The Weekend Australian, The Bangkok Post, the Los Angeles Times (a David
Colker column widely reprinted around the U.S.), the Seattle Times, the
Philadelphia Inquirer, the Halifax Chronicle-Herald, Ziff-Davis� Inside the
Internet, newsletter The Web magazine, and March 15, 1999 and many other and
periodicals.
Also recommended by
Over 4.4 million visitors since 1997.
In its first three years this site was visited more than a million times.
Because of various technical and design problems which caused it to lag far
behind the actual traffic, the counter has since been removed from this page.
After the first million hits, a link to a more sophisticated counter was
installed on the next page, where most visitors go directly. Click on the
“Site Meter� icon at the bottom of that page and add 1 million to get a fair
idea of how many people have dropped in. Note that this counter measures
discrete visits, not just “hits.�
If you search for the word “English� in Google, which gives a measure of
popularity by ranking its results in order of the number of links other
people have created to them, my site turns up twice, both high in the list of
hits, because many people still link to the old, many years out-of-date
address www.wsu.edu:8080/~brians/errors/ instead of the correct current
address at www.wsu.edu/~brians/errors/. Add the two together, and this would
seem to be the most commonly linked-to site on the Web for �English.� Thanks,
folks!
Recommended as an “Incredibly Useful Site� in Yahoo Internet Life Magazine ,
July, 1997, pp. 82-83 and cited as a Yahoo “Site of the Week� and recommended
by Netsurfer Digest March 5, 1999. It has also been recommended in the pages
of The Weekend Australian, The Bangkok Post, the Los Angeles Times (a David
Colker column widely reprinted around the U.S.), the Seattle Times, the
Philadelphia Inquirer, the Halifax Chronicle-Herald, Ziff-Davis� Inside the
Internet, newsletter The Web magazine, and March 15, 1999 and many other and
periodicals.
Also recommended by

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page2.htm
#DRETITLE
Page2
#DRESECTION 0
#DREDATE 1121339984
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-3.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page2"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page2.htm"
#DREFIELD SPIDERDATE="1121339984"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="553"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="498"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="113"
#DREFIELD DREDATE="1121339984"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-3.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="96ff801d321ba516fd0622fe4d9b2fc0"
#DREFIELD summary="What is an error in English? The concept of language
errors is a fuzzy one. I’ll leave to linguists the technical definitions"
#DRECONTENT
What is an error in English?
The concept of language errors is a fuzzy one. I’ll leave to linguists the
technical definitions. Here we’re concerned only with deviations from the
standard use of English as judged by sophisticated users such as professional
writers, editors, teachers, and literate executives and personnel officers.
The aim of this site is to help you avoid low grades, lost employment
opportunities, lost business, and titters of amusement at the way you write
or speak.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page3.htm
#DRETITLE
Page3
#DRESECTION 0
#DREDATE 1121339989
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-4.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page3"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page3.htm"
#DREFIELD SPIDERDATE="1121339989"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="492"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="431"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="112"
#DREFIELD DREDATE="1121339989"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-4.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="473cb1af8c69401cf4125e12d9d92753"
#DREFIELD summary="But isn’t one person’s mistake another’s standard usage?
Often enough, but if your standard usage causes other people to consider you
stupid or ignorant, you may want to consider changing it. You have the right
to express yourself in any manner you please, but if you wish to communicate
effectively, you should use nonstandard English only when you intend to,
rather than fall into it because you don’t know any better"
#DRECONTENT
But isn’t one person’s mistake another’s standard usage?
Often enough, but if your standard usage causes other people to consider you
stupid or ignorant, you may want to consider changing it. You have the right
to express yourself in any manner you please, but if you wish to communicate
effectively, you should use nonstandard English only when you intend to,
rather than fall into it because you don’t know any better.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page4.htm
#DRETITLE
Page3
#DRESECTION 0
#DREDATE 1121339991
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-5.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page3"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page4.htm"
#DREFIELD SPIDERDATE="1121339991"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="581"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="520"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="113"
#DREFIELD DREDATE="1121339991"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-5.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="205843e949e8b6186ef1281d2a7ccfbf"
#DREFIELD summary="Aren’t some of these points awfully picky? This is a
relative matter. One person’s gaffe is another’s peccadillo"
#DRECONTENT
Aren’t some of these points awfully picky?
This is a relative matter. One person’s gaffe is another’s peccadillo. Some
common complaints about usage strike me as too persnickety, but I’m just
discussing mistakes in English that happen to bother me. Feel free to create
your own page listing your own pet peeves, but I welcome suggestions for
additions to these pages. First, read the Commonly Made Suggestions page, and
if you still want to write me, please do so, after reading the instructions
on that page.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page5.htm
#DRETITLE
Page3
#DRESECTION 0
#DREDATE 1121339995
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-6.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page3"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page5.htm"
#DREFIELD SPIDERDATE="1121339995"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="2"
#DREFIELD SPIDERPAGESIZE="461"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="398"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="112"
#DREFIELD DREDATE="1121339995"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-6.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="669ec873d9f7eb5e92ba0de8f6319c5b"
#DREFIELD summary="What gives you the right to say what an error in English
is? I could take the easy way out and say I’m a professor of English and do
this sort of thing for a living. True, but my Ph"
#DRECONTENT
What gives you the right to say what an error in English is?
I could take the easy way out and say I’m a professor of English and do this
sort of thing for a living. True, but my Ph.D. is in comparative literature,
not composition or linguistics, and I teach courses in the history of ideas
rather than language as such. But I admire good writing and try to encourage
it in my students.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page6.htm
#DRETITLE
Page6
#DRESECTION 0
#DREDATE 1121339997
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-7.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page6"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page6.htm"
#DREFIELD SPIDERDATE="1121339997"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="669"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="608"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="113"
#DREFIELD DREDATE="1121339997"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-7.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="0b566384350fb75e436e74ff3dec49fb"
#DREFIELD summary="found a word you criticized in the dictionary! You will
find certain words or phrases criticized here listed in dictionaries. Note
carefully labels like dial"
#DRECONTENT
found a word you criticized in the dictionary!
You will find certain words or phrases criticized here listed in
dictionaries. Note carefully labels like dial. (dialectical), nonstandard,
and obsolete before assuming that the dictionary is endorsing them. The
primary job of a dictionary is to track how people actually use language.
Dictionaries differ among themselves on how much guidance to usage they
provide; but the goal of a usage guide like this is substantially different:
to protect you against patterns which are regarded by substantial numbers of
well-educated people as nonstandard.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page7.htm
#DRETITLE
Page6
#DRESECTION 0
#DREDATE 1121340000
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-8.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page6"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page7.htm"
#DREFIELD SPIDERDATE="1121340000"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="809"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="746"
#DREFIELD IMPORTMETALEN="101"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="114"
#DREFIELD DREDATE="1121340000"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-8.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="23be374a5f0dda03c82a49c7bdce9166"
#DREFIELD summary="Why do you discuss mainly American usage? Because I’m an
American, my students are mostly American, most English-speaking Web users
are Americans, and American English is quickly becoming an international
standard. I am slowly reworking the site to take note of American deviations
from standard British practice"
#DRECONTENT
Why do you discuss mainly American usage?
Because I’m an American, my students are mostly American, most
English-speaking Web users are Americans, and American English is quickly
becoming an international standard. I am slowly reworking the site to take
note of American deviations from standard British practice. However, the job
is complicated by the fact that Canadians, Australians, and many others often
follow patterns somewhere between the two. If the standard usage where you
are differs from what is described here, tell me about it; and if I think
it’s important to do so, I’ll note that fact. Meanwhile, just assume that
this site is primarily about American English. If you feel tempted to argue
with me, click here first.

#DREENDDOC
#DREREFERENCE http://15.220.107.161:81/www.hptest.com/page8.htm
#DRETITLE
Page8
#DRESECTION 0
#DREDATE 1121340004
#DREDBNAME SGMS
#DREFILENAME C:\Work\Test\Test\HPTest-9.html
#DRESTORECONTENT yes
#DREFIELD DRETITLE="Page8"
#DREFIELD DREREFERENCE="http://15.220.107.161:81/www.hptest.com/page8.htm"
#DREFIELD SPIDERDATE="1121340004"
#DREFIELD SPIDERDOMAIN="15.220.107.161:81"
#DREFIELD SPIDERPAGEDEPTH="1"
#DREFIELD SPIDERPAGELINKS="0"
#DREFIELD SPIDERPAGEDURATION="1"
#DREFIELD SPIDERPAGESIZE="1599"
#DREFIELD CONTENTTYPE="text/html"
#DREFIELD IMPORTBODYLEN="1536"
#DREFIELD IMPORTMETALEN="102"
#DREFIELD IMPORTLINKLEN="0"
#DREFIELD IMPORTTITLELEN="6"
#DREFIELD IMPORTQUALITY="116"
#DREFIELD DREDATE="1121340004"
#DREFIELD DREFILENAME="C:\Work\Test\Test\HPTest-9.html"
#DREFIELD BIAS="100"
#DREFIELD VISIBLE="FALSE"
#DREFIELD CHECKSUM="e3f3e951ff0b4f5496fb5d3043ac760d"
#DREFIELD summary="Does it oppress immigrants and subjugated minorities to
insist on the use of standard English? Language standards can certainly be
used for oppressive purposes, but most speakers and writers of all races and
classes want to use language in a way that will impress others. It is
interesting that in the debate over Oakland, California’s proposed “ebonics�
policy, African-American parents were especially outspoken in arguing that to
allow students to regard street slang as legitimate in an educational setting
was to limit them and worsen their oppressed status"
#DRECONTENT
Does it oppress immigrants and subjugated minorities to insist on the use of
standard English?
Language standards can certainly be used for oppressive purposes, but most
speakers and writers of all races and classes want to use language in a way
that will impress others. It is interesting that in the debate over Oakland,
California’s proposed “ebonics� policy, African-American parents were
especially outspoken in arguing that to allow students to regard street slang
as legitimate in an educational setting was to limit them and worsen their
oppressed status. The fact is that the world is full of teachers, employers,
and other authorities who may penalize you for your nonstandard use of the
English language. Not a few employers automatically discard any job
application that they notice contains a usage or spelling error. Feel free to
denounce these people if you wish; but if you need their good opinion to get
ahead, you’d be wise to learn standard English. Note that I often suggest
differing usages as appropriate depending on the setting: spoken vs. written,
informal vs. formal; slang is often highly appropriate. In fact, most of the
errors discussed on this site are common in the writing of privileged
middle-class Americans, and some are characteristic of people with advanced
degrees and considerable intellectual attainments. However you come down on
this issue, note that the great advantage of an open Web-based educational
site like this is that it’s voluntary: take what you want and leave the rest.

#DREENDDOC

[Oliver Sturm]

Ganesan selvaraj wrote:

acutely i want to split some source file. I have attached the sample file.
in that i want to split the #DRECONTENT part for each url and create a
file.
so the file size can be 3 kb to 3 GB.
already i send you the code. in that i am reading 10 mb(configurable)
content and i split that one. then i remove the first 10mb from memory and
read next 10mb content on so on.
This is what my code is doing. when i run this some times i got the out of
memory exception.

when i run this code i log the memory utilization. the virtual memory is
keep on increasing.

I'm sorry, mate. Jon and I sent you a lot of suggestions about your code,
assuming you might want to act on them. I'm not going to reiterate all
this now with a sample file at hand.

Why don't you rework your code in some of the ways we suggested, see if
some of the problems go away, possibly repost your new, hopefully much
shorter and more readable sample code if there are still questions?
Oliver Sturm
--
Expert programming and consulting services available
See http://www.sturmnet.org (try /blog as well)