Garbage collection, Unmanaged code and SafeArrays

[R. MacDonald]

Hello, all,

I am currently working on a .Net (VB) application that invokes routines
in unmanaged (Fortran) DLLs. The unmanaged routines then communicate
with the .Net application by means of a call-back mechanism. These
calls pass a string that contains a "command" and a pointer to a
SafeArray that (depending on the command) either receives data from the
..Net application or provides data to the .Net application.

This mechanism is working fine.

However, there might be a problem related to garbage collection. Here
is what I was trying to do:

1. Create a SafeArray in the unmanaged code.
2. Pass a pointer to the SafeArray to the .Net call-back routine for
processing.
3. After the .Net call-back is finished, the unmanaged code
"destroys" the SafeArray to free its resources.

Step 3 fails! On examination, it appeared there was no longer a
SafeArray at the original pointer location. What I discovered was that
the value of the pointer to the SafeArray was being changed by the call
to the .Net routine. There WAS a SafeArray at this changed location.
It appeared to have the same contents as the original SafeArray and,
from unmanaged code, I can "destroy" the SafeArray at this new location.

Some research at:

http://msdn.microsoft.com/library/de...ingpinning.asp

revealed:

<quote>======== ==============
Reference Types
.... Reference types have the following conditional behavior:

If a reference type is passed by value and it has members of
non-blittable types, the types are converted twice:
When an argument is passed to the unmanaged side.
On return from the call.

<end-quote>========= ==========

Now, I'm passing the SafeArray in the opposite direction. I.e. from
unmanaged code to managed code. But if something similar applies, then
it would seem that on return I have received a different SafeArray than
the one I passed. And the one I passed seems to no longer exist!

The only immediate complication that I had from this was from a table of
allocated resources I was keeping (so that I could be sure these get
cleaned up properly before returning from the unmanaged routine). In
this table the pointers for SafeArrays that had been passed to .Net were
no longer valid (so freeing the corresponding SafeArrays failed). (I
can see that there could have been additional consequences had I wanted
to do anything more serious with the original SafeArrays.)

But my real question now is: "What happened to the original SafeArray?"
Did (or will) .Net's Garbage Collector take care of it, or have I now
got a memory leak?

Secondary questions are:

Will the .Net Garbage Collector clean-up the SafeArray that it has
created/substituted for the original? If so, how will it know that the
unmanaged code has finished with it? (The only .Net reference is in the
no-longer-active call-back routine.)

If I destroy the substituted SafeArray in unmanaged code, will this
cause some angst for the .Net Garbage Collector?

I sure wish I had a clear picture of how all this was working. Any
enlightenment (or pointers to helpful documentation) will be much
appreciated.

Cheers,
Randy

[Kevin Spencer]

Good research.

I'm not sure that one of your assumptions is correct, however:

Step 3 fails! On examination, it appeared there was no longer a SafeArray
at the original pointer location. What I discovered was that the value of
the pointer to the SafeArray was being changed by the call to the .Net
routine. There WAS a SafeArray at this changed location. It appeared to
have the same contents as the original SafeArray and, from unmanaged code,
I can "destroy" the SafeArray at this new location.
Are ou usre that the original SafeArray isn't still at the original
location? I find it hard to believe that the managed code can actually
change the location of the unmanaged array. Instead, it sounds to me like
the managed code is making a copy of the original SafeArray. If so, the
unmanaged code should still be able to deallocate the original SafeArray,
and the .Net garbage collection will take care of the copy.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer

Presuming that God is "only an idea" -
Ideas exist.
Therefore, God exists.

"R. MacDonald" <sc****@NO-SP-AMcips.ca> wrote in message
news:44******** *************** @news.wanadoo.n l... Hello, all,

I am currently working on a .Net (VB) application that invokes routines in
unmanaged (Fortran) DLLs. The unmanaged routines then communicate with
the .Net application by means of a call-back mechanism. These calls pass
a string that contains a "command" and a pointer to a SafeArray that
(depending on the command) either receives data from the .Net application
or provides data to the .Net application.

This mechanism is working fine.

However, there might be a problem related to garbage collection. Here is
what I was trying to do:

1. Create a SafeArray in the unmanaged code.
2. Pass a pointer to the SafeArray to the .Net call-back routine for
processing.
3. After the .Net call-back is finished, the unmanaged code "destroys"
the SafeArray to free its resources.

Step 3 fails! On examination, it appeared there was no longer a SafeArray
at the original pointer location. What I discovered was that the value of
the pointer to the SafeArray was being changed by the call to the .Net
routine. There WAS a SafeArray at this changed location. It appeared to
have the same contents as the original SafeArray and, from unmanaged code,
I can "destroy" the SafeArray at this new location.

Some research at:

http://msdn.microsoft.com/library/de...ingpinning.asp

revealed:

<quote>======== ==============
Reference Types
... Reference types have the following conditional behavior:

If a reference type is passed by value and it has members of non-blittable
types, the types are converted twice:
When an argument is passed to the unmanaged side.
On return from the call.

<end-quote>========= ==========

Now, I'm passing the SafeArray in the opposite direction. I.e. from
unmanaged code to managed code. But if something similar applies, then it
would seem that on return I have received a different SafeArray than the
one I passed. And the one I passed seems to no longer exist!

The only immediate complication that I had from this was from a table of
allocated resources I was keeping (so that I could be sure these get
cleaned up properly before returning from the unmanaged routine). In this
table the pointers for SafeArrays that had been passed to .Net were no
longer valid (so freeing the corresponding SafeArrays failed). (I can see
that there could have been additional consequences had I wanted to do
anything more serious with the original SafeArrays.)

But my real question now is: "What happened to the original SafeArray?"
Did (or will) .Net's Garbage Collector take care of it, or have I now got
a memory leak?

Secondary questions are:

Will the .Net Garbage Collector clean-up the SafeArray that it has
created/substituted for the original? If so, how will it know that the
unmanaged code has finished with it? (The only .Net reference is in the
no-longer-active call-back routine.)

If I destroy the substituted SafeArray in unmanaged code, will this cause
some angst for the .Net Garbage Collector?

I sure wish I had a clear picture of how all this was working. Any
enlightenment (or pointers to helpful documentation) will be much
appreciated.

Cheers,
Randy

[R. MacDonald]

Hello, Kevin,

Thanks for your reply.

Re:

... I find it hard to believe that the managed code can actually
change the location of the unmanaged array. ...
I agree that this is hard to believe. Actually, I had thought .Net had
done away with the original SafeArray, and I found that equally hard to
believe.

It turns out that your disbelief was right! My assumption that the
original SafeArray was gone was incorrect.

Your reply prompted me to go back and have a closer look. (Sometimes
these "problems" seem to vanish when I look a second time, making me
suspect that I am subject to hallucinations. ) This time however, the
"problem" appeared to be the same, so I looked a little deeper.

My conclusion that the original SafeArray was gone was based on the fact
that I received an error code (-2147352563) when I tried to "destroy"
it. This is the same error code I receive if I try to destroy the same
SafeArray a second time, so I believed that it meant the original
SafeArray no longer existed.

But I seem to have misinterpreted the meaning of this code. When I
convert this value to hex (8002000D) I discover that it maps to the
symbol "DISP_E_ARRAYIS LOCKED".

This is much more believable than the array has been moved or doesn't
exist. Since I had already removed the lock that I had put on this
SafeArray prior to my attempt to destroy it, I now believe I must be
applying a second lock inadvertently, and I will begin stepping through
my code to try to find it.

During my experiments however, I discovered that if I remove all the
locks before the SafeArray is passed to the .Net call-back, then the
pointer doesn't get changed. That is, it passes back the original, or
it passes back its (second) copy in the same location that was occupied
by the original. I suspect that it is the latter. When the original
SafeArray is locked, then .Net can't replace it so it creates its second
copy in a new location and updates the pointer to that new location.

This leads me to believe that after the call (if the original SafeArray
was unlocked) then it has been destroyed by .Net and replaced by a copy.
I then wonder: "Who owns that copy?" I.e. Who should de-allocate it
-- the unmanaged code, or the .Net garbage collector?

For now, I will de-allocate it in the unmanaged code. It created
something, so it should delete something. And I'll hope that this
doesn't give the .Net Garbage Collector any headaches.

Once again, thanks for your post. Your expression of doubt was what I
needed to get on the right track.

Cheers,
Randy
Kevin Spencer wrote:
Good research.

I'm not sure that one of your assumptions is correct, however:

Step 3 fails! On examination, it appeared there was no longer a SafeArray
at the original pointer location. What I discovered was that the value of
the pointer to the SafeArray was being changed by the call to the .Net
routine. There WAS a SafeArray at this changed location. It appeared to
have the same contents as the original SafeArray and, from unmanaged code,
I can "destroy" the SafeArray at this new location.

Are ou usre that the original SafeArray isn't still at the original
location? I find it hard to believe that the managed code can actually
change the location of the unmanaged array. Instead, it sounds to me like
the managed code is making a copy of the original SafeArray. If so, the
unmanaged code should still be able to deallocate the original SafeArray,
and the .Net garbage collection will take care of the copy.

[Kevin Spencer]

Hi Randy,

I wish I had more time to look into this. But let me give you a few more
intuitive suggestions, based on my understanding of .Net. First, I would not
expect that the managed code should be expected to do any clean-up work in
the unmanaged code, and vice versa. It just doesn't make sense.

In addition, I'm not sure that passing a pointer to the SafeArray is the
best way to go, unless you can't avoid it. First, as you've noticed, the
..Net Framework is already making a copy of the array. Second, allowing the
..Net Framework to perform any operations that change the contents of the
unmanaged memory just seems like a dangerous idea. Interesting to be sure,
and no doubt an educational experience, but probably dangerous. At any rate,
when pointers are passed back and forth, I would tend to disallow either of
the apps to modify data in the other. Even in terms of good ol' OOP
principles, objects should manage themselves whenever possible. That's what
encapsulation is all about.

I'd love to hear what you do find out as you go, though!

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer

Presuming that God is "only an idea" -
Ideas exist.
Therefore, God exists.

"R. MacDonald" <sc****@NO-SP-AMcips.ca> wrote in message
news:44******** *************** @news.wanadoo.n l...

Hello, Kevin,

Thanks for your reply.

Re:

... I find it hard to believe that the managed code can actually
change the location of the unmanaged array. ...

I agree that this is hard to believe. Actually, I had thought .Net had
done away with the original SafeArray, and I found that equally hard to
believe.

It turns out that your disbelief was right! My assumption that the
original SafeArray was gone was incorrect.

Your reply prompted me to go back and have a closer look. (Sometimes
these "problems" seem to vanish when I look a second time, making me
suspect that I am subject to hallucinations. ) This time however, the
"problem" appeared to be the same, so I looked a little deeper.

My conclusion that the original SafeArray was gone was based on the fact
that I received an error code (-2147352563) when I tried to "destroy" it.
This is the same error code I receive if I try to destroy the same
SafeArray a second time, so I believed that it meant the original
SafeArray no longer existed.

But I seem to have misinterpreted the meaning of this code. When I
convert this value to hex (8002000D) I discover that it maps to the symbol
"DISP_E_ARRAYIS LOCKED".

This is much more believable than the array has been moved or doesn't
exist. Since I had already removed the lock that I had put on this
SafeArray prior to my attempt to destroy it, I now believe I must be
applying a second lock inadvertently, and I will begin stepping through my
code to try to find it.

During my experiments however, I discovered that if I remove all the locks
before the SafeArray is passed to the .Net call-back, then the pointer
doesn't get changed. That is, it passes back the original, or it passes
back its (second) copy in the same location that was occupied by the
original. I suspect that it is the latter. When the original SafeArray
is locked, then .Net can't replace it so it creates its second copy in a
new location and updates the pointer to that new location.

This leads me to believe that after the call (if the original SafeArray
was unlocked) then it has been destroyed by .Net and replaced by a copy. I
then wonder: "Who owns that copy?" I.e. Who should de-allocate it -- the
unmanaged code, or the .Net garbage collector?

For now, I will de-allocate it in the unmanaged code. It created
something, so it should delete something. And I'll hope that this doesn't
give the .Net Garbage Collector any headaches.

Once again, thanks for your post. Your expression of doubt was what I
needed to get on the right track.

Cheers,
Randy
Kevin Spencer wrote:

Good research.

I'm not sure that one of your assumptions is correct, however:

Step 3 fails! On examination, it appeared there was no longer a
SafeArray at the original pointer location. What I discovered was that
the value of the pointer to the SafeArray was being changed by the call
to the .Net routine. There WAS a SafeArray at this changed location. It
appeared to have the same contents as the original SafeArray and, from
unmanaged code, I can "destroy" the SafeArray at this new location.

Are ou usre that the original SafeArray isn't still at the original
location? I find it hard to believe that the managed code can actually
change the location of the unmanaged array. Instead, it sounds to me like
the managed code is making a copy of the original SafeArray. If so, the
unmanaged code should still be able to deallocate the original SafeArray,
and the .Net garbage collection will take care of the copy.

[R. MacDonald]

Hello, Kevin,

Thanks for your comments. I agree with all you have said.

Re:

... I'm not sure that passing a pointer to the SafeArray is the
best way to go, unless you can't avoid it.
I haven't been able to come up with any other "practical" scheme for
this. There are some performance concerns associated with this
application (which is why the unmanaged code remains unmanaged). I
think this precludes transferring the data via the file system. I'm
open to any ideas that others may have though. (The languages involved
are VB.Net and Fortran.)

Re: ...allowing the Net Framework to perform any operations that
change the contents of the unmanaged memory just seems like
a dangerous idea. ... I would tend to disallow either of
the apps to modify data in the other.
I agree with you. In this case (i.e. if the SafeArray is unlocked) the
Net Framework doesn't actually change the contents of unmanaged memory.
(I guess it replaces it with an identical copy of the original.) But
it could just as easily have replaced it with a modified copy. That's
just a matter of how the data transfers have been structured/programmed.
I am happier to have clients pulling data from servers, rather than
having the servers pushing the data to the client, so that's the way I
have done it here.

When the unmanaged (Fortran) needs data from the .Net side, I create the
SafeArray in .Net and pass the pointer to Fortran. I could update the
data from the Fortran side, or even deallocate the SafeArray. But I
don't (for the same reasons that concern you). In this case .Net should
deallocate the SafeArray, and I am trusting the magic of the Garbage
Collector to do that.

In reverse, when the unmanaged (Fortran) must send data to the .Net
side, I create the SafeArray in .Fortran and pass the pointer to .Net.
I let .Net update its own data structures, and I don't want .Net to
modify the Fortran data directly. Likewise, I expect to deallocate the
SafeArray in the Fortran DLL.

That's the way I would like it to be, anyway. My conundrum is that I'm
not sure whether or not .Net "thinks" it or Fortran "owns" the copy of
the (unlocked) SafeArray that it has replaced. I'll try to investigate
this further as time permits. (I know you understand what that means.)
:-(

In the interim I have decided to keep a lock on the Fortran created
SafeArrays. This way, I believe there is no doubt about ownership and
responsibility for clean-up. The .Net created copy is then placed in a
different memory location than the original, and (unless I learn
otherwise) I will trust .Net's Garbage Collector to deallocate that
copy. Fortran can deallocate the original.

I'll post again if I learn anything more about this.

Again, thanks for your interest and input.

Cheers,
Randy
Kevin Spencer wrote:
Hi Randy,

I wish I had more time to look into this. But let me give you a few more
intuitive suggestions, based on my understanding of .Net. First, I would not
expect that the managed code should be expected to do any clean-up work in
the unmanaged code, and vice versa. It just doesn't make sense.

In addition, I'm not sure that passing a pointer to the SafeArray is the
best way to go, unless you can't avoid it. First, as you've noticed, the
.Net Framework is already making a copy of the array. Second, allowing the
.Net Framework to perform any operations that change the contents of the
unmanaged memory just seems like a dangerous idea. Interesting to be sure,
and no doubt an educational experience, but probably dangerous. At any rate,
when pointers are passed back and forth, I would tend to disallow either of
the apps to modify data in the other. Even in terms of good ol' OOP
principles, objects should manage themselves whenever possible. That's what
encapsulation is all about.

I'd love to hear what you do find out as you go, though!

[Kevin Spencer]

Hi Randy,

That's the way I would like it to be, anyway. My conundrum is that I'm
not sure whether or not .Net "thinks" it or Fortran "owns" the copy of the
(unlocked) SafeArray that it has replaced. I'll try to investigate this
further as time permits. (I know you understand what that means.) :-(
I don't think you need to worry about this. The .Net Framework is actually
working with a pointer to the pointer, and it will Garbage-Collect that, but
not touch the original unmanaged data.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer

Presuming that God is "only an idea" -
Ideas exist.
Therefore, God exists.

"R. MacDonald" <sc****@NO-SP-AMcips.ca> wrote in message
news:44******** *************** @news.wanadoo.n l... Hello, Kevin,

Thanks for your comments. I agree with all you have said.

Re:

... I'm not sure that passing a pointer to the SafeArray is the
best way to go, unless you can't avoid it.

I haven't been able to come up with any other "practical" scheme for this.
There are some performance concerns associated with this application
(which is why the unmanaged code remains unmanaged). I think this
precludes transferring the data via the file system. I'm open to any
ideas that others may have though. (The languages involved are VB.Net and
Fortran.)

Re:

...allowing the Net Framework to perform any operations that
change the contents of the unmanaged memory just seems like
a dangerous idea. ... I would tend to disallow either of
the apps to modify data in the other.

I agree with you. In this case (i.e. if the SafeArray is unlocked) the
Net Framework doesn't actually change the contents of unmanaged memory. (I
guess it replaces it with an identical copy of the original.) But it
could just as easily have replaced it with a modified copy. That's just a
matter of how the data transfers have been structured/programmed. I am
happier to have clients pulling data from servers, rather than having the
servers pushing the data to the client, so that's the way I have done it
here.

When the unmanaged (Fortran) needs data from the .Net side, I create the
SafeArray in .Net and pass the pointer to Fortran. I could update the
data from the Fortran side, or even deallocate the SafeArray. But I don't
(for the same reasons that concern you). In this case .Net should
deallocate the SafeArray, and I am trusting the magic of the Garbage
Collector to do that.

In reverse, when the unmanaged (Fortran) must send data to the .Net side,
I create the SafeArray in .Fortran and pass the pointer to .Net. I let
.Net update its own data structures, and I don't want .Net to modify the
Fortran data directly. Likewise, I expect to deallocate the SafeArray in
the Fortran DLL.

That's the way I would like it to be, anyway. My conundrum is that I'm
not sure whether or not .Net "thinks" it or Fortran "owns" the copy of the
(unlocked) SafeArray that it has replaced. I'll try to investigate this
further as time permits. (I know you understand what that means.) :-(

In the interim I have decided to keep a lock on the Fortran created
SafeArrays. This way, I believe there is no doubt about ownership and
responsibility for clean-up. The .Net created copy is then placed in a
different memory location than the original, and (unless I learn
otherwise) I will trust .Net's Garbage Collector to deallocate that copy.
Fortran can deallocate the original.

I'll post again if I learn anything more about this.

Again, thanks for your interest and input.

Cheers,
Randy
Kevin Spencer wrote:

Hi Randy,

I wish I had more time to look into this. But let me give you a few more
intuitive suggestions, based on my understanding of .Net. First, I would
not expect that the managed code should be expected to do any clean-up
work in the unmanaged code, and vice versa. It just doesn't make sense.

In addition, I'm not sure that passing a pointer to the SafeArray is the
best way to go, unless you can't avoid it. First, as you've noticed, the
.Net Framework is already making a copy of the array. Second, allowing
the .Net Framework to perform any operations that change the contents of
the unmanaged memory just seems like a dangerous idea. Interesting to be
sure, and no doubt an educational experience, but probably dangerous. At
any rate, when pointers are passed back and forth, I would tend to
disallow either of the apps to modify data in the other. Even in terms of
good ol' OOP principles, objects should manage themselves whenever
possible. That's what encapsulation is all about.

I'd love to hear what you do find out as you go, though!