IStream.Read Memory Leak?

[KWienhold]

I'm currently writing an application (using Visual Studio 2003 SP1 and
C#) that stores files and additional information in a single compound
file using IStorage/IStream.
Since files in a compound file aren't really useful to a user, I use
the IStream::Read function together with a StreamWriter to extract
single files from my compound document.
When I first tested these functions everything seemed to work fine (and
basically, it does), however the memory usage of the app would
skyrocket every time I extracted a large file and never come down until
I closed the application.
At first I thought I had simply forgotten to clear my buffer, leaving a
global reference behind that stopped the GC from cleaning it up (or
something similar), after more in-depth testing this does not appear to
be the case.
Stepping through the code line by line and watching the memory usage
carefully I could see that memory would be reserved every time I called
the Read-function of an IStream (which is to be expected), but it would
not be freed when I nulled the buffer-array I had passed as the out
parameter.
I would have expected IStream::Read to use the buffer array I pass and
not require much memory beyond that, but even after calling the
Garbage-Collector manually after clearing my objects, the memory stays
in use.
I have tried pretty much anything I can think of, calling
IStream::Release after every read (which is not really feasible in any
case, since I want to read my data in chunks and not realease the
stream after every chunk), using Marshal.ReleaseComObject on the
stream, getting the IntPtr from my buffer array and calling
Marshal.FreeCoTaskMem on it manually, but to no effect, the memory
stays used.
I'm at my wits end with this, I can't possibly live with a memory leak
this enormous, but I also can't think of any way to get my data without
using IStream::Read.
Since I'm currently working with VS 2003/.Net 1.1 I use my own
IStream-Implementation, not the one Microsoft supplies with .Net 2.0.
I did convert my project to VS2005 just to try their IStream-class, but
it seems that that works even less, since their implementation of
IStream::Read takes the last parameter (which should return the number
of actually read bytes) as ByVal and thusly always returns 0, not a
good thing if you want to read data in chunks and need to know when you
are done.
Since I haven't worked with IStream before, it is absolutely possible
that I am simply missing something here, but I have no idea what.
Has anyone else ever experienced behavior like this and could offer
some help on how to make sure Read doesn't gobble up my memory?
Any reply would be greatly appreciated.

Sincerely,
Kevin Wienhold

[Mattias Sjögren]

Kevin,

It's hard to comment on your problem without seeing any code. But I
have a hard time imagining a memory leak in the system's
implementation of IStream::Read. It's so widely used, so such a
problem would have been discovered by now.

>I did convert my project to VS2005 just to try their IStream-class, but
it seems that that works even less, since their implementation of
IStream::Read takes the last parameter (which should return the number
of actually read bytes) as ByVal and thusly always returns 0, not a
good thing if you want to read data in chunks and need to know when you
are done.

The last parameter is an IntPtr in this case. You have to pass in a
pointer to a location (sizeof(int) = 4 byte buffer) where the result
should be written and then read it from there.

It's defined this way because it's legal for caller's of Read to pass
a NULL pointer if they don't care about the information.
Mattias

--
Mattias Sjögren [C# MVP] mattias @ mvps.org
http://www.msjogren.net/dotnet/ | http://www.dotnetinterop.com
Please reply only to the newsgroup.

[KWienhold]

Thank you for your response.

It's hard to comment on your problem without seeing any code. But I
have a hard time imagining a memory leak in the system's
implementation of IStream::Read. It's so widely used, so such a
problem would have been discovered by now.

I realize that IStream is widely used, that's why I assumed I did
something wrong.
I was just hoping that someone knew some general advice concerning
this. The actual function that does the reading is pretty trivial, so I
guess the error lies elsewhere, but since I have no idea where, it is
hard to decide which parts of the code I should post.

The last parameter is an IntPtr in this case. You have to pass in a
pointer to a location (sizeof(int) = 4 byte buffer) where the result
should be written and then read it from there.

It's defined this way because it's legal for caller's of Read to pass
a NULL pointer if they don't care about the information.

If I use the same implementation .Net 2.0 uses, pass an IntPtr and get
its value after the Read-call using IntPtr.ToInt32(), the value is
always 0.
If I pass the IntPtr as a reference it seems to work fine.
I haven't really worked with many COM objects in C#, so I assumed that
the ToInt32-Function would do exactly what you suggested, reading 4
bytes from memory from where the pointer points to and convert them
into an integer.
I'll see what happens when I read the memory manually, it would be
better to stick to the suggested interface declaration without the ref.

Sincerely,
Kevin Wienhold

[KWienhold]

A small addition:
After getting an IntPtr through Marshal.AllocCoTaskMem(sizeof(int)) I
was able to read a non-zero value from the pointer using ToInt32, I
didn't initialize the IntPtr properly before.
However the value this returns doesn't match the bytes read.
I read data from the stream in chunks 1 MB in size (so just over 1
million bytes), the value returned from IStream::Read after reading one
chunk is over 68 million and never becomes zero, even if I pass 0 as
the second parameter.