MatchCollection Memory Issue

[Kini]

In the program below, when the control comes back to the Main program,
a considerable amount of memory is still associated with the program.
Is this normal? Is it due to repeated generation of MatchCollection
objects?

This program is a sample of a larger problem I am facing with a
document retrieval application. The algorithm is as follows:
1. Recusively traverse a directory
2. Obtain the text of every file in a string
3. Extract "terms" in the string using the expression in Regex
4. Loop through MatchCollection to obtain number of occurences of each
"term"

Through a memory profiling tool, I noticed considerable profusion of
strings, which I think is due to MatchCollection . How do I ensure that
memory occupied by MatchCollection is reclaimed often?

Thanks, Kini

*** Code ***

using System;
using System.Collecti ons;
using System.Text.Reg ularExpressions ;

namespace DemoRegex
{
class MainClass
{
[STAThread]
static void Main(string[] args)
{
for(int i=0;i<10000;i++ )
{
FileConvert();
}
Console.WriteLi ne("Press Any Key To End");
Console.ReadLin e();
}
static void FileConvert()
{
string testString = "Fractions can be expressed as a
numerator over a denominator; however, storing them as a floating-point
value might be necessary. Storing fractions as floating-point values
introduces rounding errors that make it difficult to perform
comparisons. Expressing the value as a fraction (e.g., 1/6) allows the
maximum precision. Expressing the value as a floating-point value
(e.g., 0.16667) can limit the precision of the value. In this case, the
precision depends on the number of digits that the developer decides to
use to the right of the decimal point.";

Regex regEx = new
Regex(@"[a-zA-Z0-9]*[.\-_]*[.a-zA-Z\-_][.a-zA-Z0-9\-_]*[a-zA-Z0-9]");
MatchCollection matchesInDoc = regEx.Matches(t estString);
}
}
}

[Mark R. Dawson]

Hi Kini,
usually the Garbage Collector will execute when it thinks it is time to
reclaim unused memory. Are you seeing some kind of degredation in the
program as a result of large amounts of memory being consumed, how much is
being used compared to the amount of RAM you have?

You can force the Garbage collector to explicitly run at any time in your
code by calling GC.Collect();

Mark.

"Kini" wrote:

In the program below, when the control comes back to the Main program,
a considerable amount of memory is still associated with the program.
Is this normal? Is it due to repeated generation of MatchCollection
objects?

This program is a sample of a larger problem I am facing with a
document retrieval application. The algorithm is as follows:
1. Recusively traverse a directory
2. Obtain the text of every file in a string
3. Extract "terms" in the string using the expression in Regex
4. Loop through MatchCollection to obtain number of occurences of each
"term"

Through a memory profiling tool, I noticed considerable profusion of
strings, which I think is due to MatchCollection . How do I ensure that
memory occupied by MatchCollection is reclaimed often?

Thanks, Kini

*** Code ***

using System;
using System.Collecti ons;
using System.Text.Reg ularExpressions ;

namespace DemoRegex
{
class MainClass
{
[STAThread]
static void Main(string[] args)
{
for(int i=0;i<10000;i++ )
{
FileConvert();
}
Console.WriteLi ne("Press Any Key To End");
Console.ReadLin e();
}
static void FileConvert()
{
string testString = "Fractions can be expressed as a
numerator over a denominator; however, storing them as a floating-point
value might be necessary. Storing fractions as floating-point values
introduces rounding errors that make it difficult to perform
comparisons. Expressing the value as a fraction (e.g., 1/6) allows the
maximum precision. Expressing the value as a floating-point value
(e.g., 0.16667) can limit the precision of the value. In this case, the
precision depends on the number of digits that the developer decides to
use to the right of the decimal point.";

Regex regEx = new
Regex(@"[a-zA-Z0-9]*[.\-_]*[.a-zA-Z\-_][.a-zA-Z0-9\-_]*[a-zA-Z0-9]");
MatchCollection matchesInDoc = regEx.Matches(t estString);
}
}
}

[Kini]

Hello Mark,

I have 500MB of RAM on my PC.

For a particular example, using 59 files of net size 43MB, my
application's memory consumption peaks to 111MB. At the place where I
provide a breakpoint, I know that the only objects in scope are not
more than 10MB in size. I do not understand why the system does not let
go of the remaining memory :-? :(

Thanks, Kini

[Mark R. Dawson]

Hi Kini,
the nature of Garbage collection is that it is non-deterministic to the
application you do not know when it will be run. The GC has an algorithm
sitting behind it that decides when it is time to clean up the managed heap.

If you have 500MB of RAM and you are using 111MB and don't having other
apps competing for the RAM resource then the GC may decide that there is no
need to run. You don't want the Garbage Collector to be running too often
because this may hurt performance more than using more RAM, because while the
Garbage Collector is running no other threads are allowed to run, this is
because the garbage collector is removing items from the managed heap and
moving objects around in memory when it finishes it then has to reset all the
pointers to those objects to the objects new memory location before you are
allowed to continue using them.

In general the Garbage Collector is probably doing an optimum job and I
would not worry about the amount of memory you used. It might be interesting
to open up a lot of applications on your computer such that a lot of RAM is
being used and see if your application still grows to 111MB or if the GC
tidies up before that due to the limited resources.

Mark.

"Kini" wrote:

Hello Mark,

I have 500MB of RAM on my PC.

For a particular example, using 59 files of net size 43MB, my
application's memory consumption peaks to 111MB. At the place where I
provide a breakpoint, I know that the only objects in scope are not
more than 10MB in size. I do not understand why the system does not let
go of the remaining memory :-? :(

Thanks, Kini

[Willy Denoyette [MVP]]

Not sure what you mean with a considerable amount of memory, when I run your
code it's working set starts with ~5MB and grows to ~6MB. The GC ran ~30
times to collect Gen0 and once to collect Gen1. That doesn't look like a
considerable amount though.
Note that you have to be more explicit when talking about memory, what kind
of memory are you talking about, there is managed heap memory and there is
native heap memory. All the different memory types have their own
performance counters and you can watch them using perfmon.

Willy.

"Kini" <ki********@gma il.com> wrote in message
news:11******** **************@ g49g2000cwa.goo glegroups.com.. .

In the program below, when the control comes back to the Main program,
a considerable amount of memory is still associated with the program.
Is this normal? Is it due to repeated generation of MatchCollection
objects?

This program is a sample of a larger problem I am facing with a
document retrieval application. The algorithm is as follows:
1. Recusively traverse a directory
2. Obtain the text of every file in a string
3. Extract "terms" in the string using the expression in Regex
4. Loop through MatchCollection to obtain number of occurences of each
"term"

Through a memory profiling tool, I noticed considerable profusion of
strings, which I think is due to MatchCollection . How do I ensure that
memory occupied by MatchCollection is reclaimed often?

Thanks, Kini

*** Code ***

using System;
using System.Collecti ons;
using System.Text.Reg ularExpressions ;

namespace DemoRegex
{
class MainClass
{
[STAThread]
static void Main(string[] args)
{
for(int i=0;i<10000;i++ )
{
FileConvert();
}
Console.WriteLi ne("Press Any Key To End");
Console.ReadLin e();
}
static void FileConvert()
{
string testString = "Fractions can be expressed as a
numerator over a denominator; however, storing them as a floating-point
value might be necessary. Storing fractions as floating-point values
introduces rounding errors that make it difficult to perform
comparisons. Expressing the value as a fraction (e.g., 1/6) allows the
maximum precision. Expressing the value as a floating-point value
(e.g., 0.16667) can limit the precision of the value. In this case, the
precision depends on the number of digits that the developer decides to
use to the right of the decimal point.";

Regex regEx = new
Regex(@"[a-zA-Z0-9]*[.\-_]*[.a-zA-Z\-_][.a-zA-Z0-9\-_]*[a-zA-Z0-9]");
MatchCollection matchesInDoc = regEx.Matches(t estString);
}
}
}

[Kini]

Hello Mark and Willy,

Thank you for taking time to look into this issue and offering
suggestions. After introducing Disposer methods for some of my classes
(which includes explicit calls to garbage collector), the memory
consumtpion (I guess managed heap) appears to be modest. This is with
repsect to my application's code.

Thanks, Kini

[Willy Denoyette [MVP]]

"Kini" <ki********@gma il.com> wrote in message
news:11******** *************@g 14g2000cwa.goog legroups.com...

Hello Mark and Willy,

Thank you for taking time to look into this issue and offering
suggestions. After introducing Disposer methods for some of my classes
(which includes explicit calls to garbage collector), the memory
consumtpion (I guess managed heap) appears to be modest. This is with
repsect to my application's code.

Thanks, Kini

Well, this is exactly what you should NOT do, there are very few reasons to
call GC.Collect() from user code. If your classes don't own unmanaged
resources, there is no reason to implement the disposable pattern either.

Willy.