Bool return value changing magically from unmanaged to managed function call.

[DaTurk]

Hi, I have a c# application that needs to access c++ libraries, so it
does this by using a managed layer of c++ CLI.

Anyway, in the CLI function call, that calls the unmanaged function it
expects a boolean return value, which it gets, but the value changes!
I walked throuhg the code, and the unmanged code is returning false,
but when I look at the variable in the managed code it's return value
it true. Even after I initialized the variable accepting that value
as false. There are no other threads accesing this code. Anyone have
any ideas?

[William DePalo [MVP VC++]]

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@l53g2000cwa.googlegr oups.com...

Anyway, in the CLI function call, that calls the unmanaged function it
expects a boolean return value, which it gets, but the value changes!
I walked throuhg the code, and the unmanged code is returning false,
but when I look at the variable in the managed code it's return value
it true. Even after I initialized the variable accepting that value
as false. There are no other threads accesing this code. Anyone have
any ideas?

It's a bug in the way the runtime marshals and unmarshals booleans across
the boundary between native and managed code.

There are other ways of solving the problem, but this one

bool unManagedFunction(...)
{
bool ok;

// set ok

ok = ...;

// Insure the 3 high order bytes of the accumulator will be zero
// Then set its low order byte appropriately

_asm xor eax, eax
return ok;
}

imposes a single instruction penalty.

Regards,
Will
www.ivrforbeginners.com

[DaTurk]

On Feb 7, 9:16 am, "William DePalo [MVP VC++]"
<[e-mail]:

"DaTurk" <[e-mail] in message

news:11**********************@l53g2000cwa.googlegr oups.com...

Anyway, in the CLI function call, that calls the unmanaged function it
expects a boolean return value, which it gets, but the value changes!
I walked throuhg the code, and the unmanged code is returning false,
but when I look at the variable in the managed code it's return value
it true. Even after I initialized the variable accepting that value
as false. There are no other threads accesing this code. Anyone have
any ideas?

It's a bug in the way the runtime marshals and unmarshals booleans across
the boundary between native and managed code.

There are other ways of solving the problem, but this one

bool unManagedFunction(...)
{
bool ok;

// set ok

ok = ...;

// Insure the 3 high order bytes of the accumulator will be zero
// Then set its low order byte appropriately

_asm xor eax, eax
return ok;

}

imposes a single instruction penalty.

Regards,
Willwww.ivrforbeginner

THanks for the reply. I'm not entirely sure I understand this command
_asm xor eax, eax enough to use it, would you mind elaborating on it?
In the unmanaged code, it already uses a similar command, how will the
one you propose effect this one __asm mov eax,100?? Or is this the
problem?

[DaTurk]

Just a heads up, I used that command _asm xor eax, eax, and removed
the other one, and it didn't help. What are some of the other
solutions?

[William DePalo [MVP VC++]]

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@s48g2000cws.googlegr oups.com...

THanks for the reply.

You are welcome.

I'm not entirely sure I understand this command
_asm xor eax, eax enough to use it, would you mind
elaborating on it?

It sets the value of the 32 bit accumulator register (EAX) to 0. C/C++
functions whose return values fit into 32 bits are returned to the caller in
EAX.

Now where you have native code that looks like this

bool a = boolFunc();

if ( a )
{
}

the function sets only the low order byte and the if statement checks only
the low byte. All is well.

However, across the managed / native boundary I _think_ that booleans are
larger than a single byte. So whatever random garbage you have in the some
or all of the three high order bytes of a function's return value get passed
back to the caller. If they are not all 0, you get a bogus TRUE value.

In the unmanaged code, it already uses a similar command, how will the
one you propose effect this one __asm mov eax,100?? Or is this the
problem?

IIUC, that will return TRUE. Is that want you want?

One way or another you probably want to clear the high 3 bytes of EAX and
drop the return value into the low byte. You can change all the return
values of your boolean functions from bool (1 byte) to BOOL (4 bytes) which
will do the trick, you can do what I do, or you can do as a knowledge base
article suggests and call a function which returns 0 to EAX _immediately_
before the real return.

Regards,
Will
www.ivrforbeginners.com

[DaTurk]

On Feb 7, 9:51 am, "William DePalo [MVP VC++]"
<[e-mail]:

"DaTurk" <[e-mail] in message

news:11**********************@s48g2000cws.googlegr oups.com...

THanks for the reply.

You are welcome.

I'm not entirely sure I understand this command
_asm xor eax, eax enough to use it, would you mind
elaborating on it?

It sets the value of the 32 bit accumulator register (EAX) to 0. C/C++
functions whose return values fit into 32 bits are returned to the caller in
EAX.

Now where you have native code that looks like this

bool a = boolFunc();

if ( a )
{
}

the function sets only the low order byte and the if statement checks only
the low byte. All is well.

However, across the managed / native boundary I _think_ that booleans are
larger than a single byte. So whatever random garbage you have in the some
or all of the three high order bytes of a function's return value get passed
back to the caller. If they are not all 0, you get a bogus TRUE value.

In the unmanaged code, it already uses a similar command, how will the
one you propose effect this one __asm mov eax,100?? Or is this the
problem?

IIUC, that will return TRUE. Is that want you want?

What do you mean that this will return true? Do you mean that because
of this statement, no matter what, the function will return true?

>
One way or another you probably want to clear the high 3 bytes of EAX and
drop the return value into the low byte. You can change all the return
values of your boolean functions from bool (1 byte) to BOOL (4 bytes) which
will do the trick, you can do what I do, or you can do as a knowledge base
article suggests and call a function which returns 0 to EAX _immediately_
before the real return.

Regards,
Willwww.ivrforbeginners.com

[William DePalo [MVP VC++]]

"DaTurk" <mm******@hotmail.comwrote in message
news:11*********************@a34g2000cwb.googlegro ups.com...

Just a heads up, I used that command _asm xor eax, eax, and removed
the other one, and it didn't help. What are some of the other
solutions?

You'll find more wordy explanations here:

http://www.codeproject.com/buglist/virtualboolbug.asp

http://support.microsoft.com/default.aspx?kbid=823071

But, if it is the bug I think it is then the crux of the issue is that some
number of the high 24 bits of EAX are set on return.

You can check that in the debugger. Put a breakpoint on the right brace that
ends your boolean function. When your application stops there take a look at
EAX. If it is larger than 0x000000FF you failed to clear the high order 24
bits. The

_asm xor eax, eax

instruction clears all 32 of them and if you put it _just before_ the place
in your code where you return the boolean value, that return should set
_only_ the low order eight bits.

Regards,
Will
www.ivrforbeginners.com

[DaTurk]

On Feb 7, 10:28 am, "William DePalo [MVP VC++]"
<[e-mail]:

"DaTurk" <[e-mail] in message

news:11*********************@a34g2000cwb.googlegro ups.com...

Just a heads up, I used that command _asm xor eax, eax, and removed
the other one, and it didn't help. What are some of the other
solutions?

You'll find more wordy explanations here:

http://www.codeproject.com/buglist/virtualboolbug.asp

http://support.microsoft.com/default.aspx?kbid=823071

But, if it is the bug I think it is then the crux of the issue is that some
number of the high 24 bits of EAX are set on return.

You can check that in the debugger. Put a breakpoint on the right brace that
ends your boolean function. When your application stops there take a look at
EAX. If it is larger than 0x000000FF you failed to clear the high order 24
bits. The

_asm xor eax, eax

instruction clears all 32 of them and if you put it _just before_ the place
in your code where you return the boolean value, that return should set
_only_ the low order eight bits.

Regards,
Willwww.ivrforbeginners.com

Your right, at the end of the function the
EAX 0x1000b300 unsigned long

[DaTurk]

But here's something interesting.

I run the program with _asm xor eax, eax in it. stepping through it.
After this line I see the Eax go to 0, then the next line is the
return statement. I step onto the return statement, and when I step
again, it steps onto the return statement again, you know you have to
step twice to get off of the statement, and on that second step the
EAX gets garbled again. Any insights as to why it would take two
steps to get passed the return statement?

[William DePalo [MVP VC++]]

"DaTurk" <mm******@hotmail.comwrote in message
news:11********************@v45g2000cwv.googlegrou ps.com...

I run the program with _asm xor eax, eax in it. stepping through it.
After this line I see the Eax go to 0, then the next line is the
return statement. I step onto the return statement, and when I step
again, it steps onto the return statement again, you know you have to
step twice to get off of the statement, and on that second step the
EAX gets garbled again. Any insights as to why it would take two
steps to get passed the return statement?

I suggest that you put a breakpoint at the entry to your function and switch
to the disassembly view in the IDE. Then single step through the machine
code, not the source code. That should tell you exactly what is going on.

Regards,
Will
www.ivrforbeginners.com

[DaTurk]

On Feb 7, 11:02 am, "William DePalo [MVP VC++]"
<[e-mail]:

"DaTurk" <[e-mail] in message

news:11********************@v45g2000cwv.googlegrou ps.com...

I run the program with _asm xor eax, eax in it. stepping through it.
After this line I see the Eax go to 0, then the next line is the
return statement. I step onto the return statement, and when I step
again, it steps onto the return statement again, you know you have to
step twice to get off of the statement, and on that second step the
EAX gets garbled again. Any insights as to why it would take two
steps to get passed the return statement?

I suggest that you put a breakpoint at the entry to your function and switch
to the disassembly view in the IDE. Then single step through the machine
code, not the source code. That should tell you exactly what is going on.

Regards,
Willwww.ivrforbeginners.com

Sounds like a good idea, I started loking at it, when I realized the
last itme I looked at assembly was about 15 years ago, and I sucked at
it then. WOuld you mind taking a peek at it? This is right at the
end of the function.

1000B3BA call _RTC_CheckEsp (10028A20h)
//__asm mov eax,100
_asm xor eax, eax;
1000B3BF xor eax,eax
return false;
1000B3C1 mov byte ptr [ebp-2ADh],0
1000B3C8 mov byte ptr [ebp-4],0Ch
1000B3CC mov esi,esp
1000B3CE lea ecx,[err]
1000B3D4 call dword ptr
[__imp_std::basic_string<char,std::char_traits<char >,std::allocator<char>

>::~basic_string<char,std::char_traits<char>,std:: allocator<char

(1003007Ch)]
1000B3DA cmp esi,esp
1000B3DC call _RTC_CheckEsp (10028A20h)
1000B3E1 mov eax,offset $L84050 (1000B3E7h)
1000B3E6 ret
$L84050:
1000B3E7 mov al,byte ptr [ebp-2ADh]
}
}

Thanks again

[William DePalo [MVP VC++]]

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@q2g2000cwa.googlegro ups.com...

Sounds like a good idea, I started loking at it, when I realized the
last itme I looked at assembly was about 15 years ago, and I sucked at
it then. WOuld you mind taking a peek at it? This is right at the
end of the function.

OK.

1000B3BA call _RTC_CheckEsp (10028A20h)
//__asm mov eax,100
_asm xor eax, eax;
1000B3BF xor eax,eax
return false;
1000B3C1 mov byte ptr [ebp-2ADh],0
1000B3C8 mov byte ptr [ebp-4],0Ch
1000B3CC mov esi,esp
1000B3CE lea ecx,[err]
1000B3D4 call dword ptr
[__imp_std::basic_string<char,std::char_traits<char >,std::allocator<char>

>>::~basic_string<char,std::char_traits<char>,std: :allocator<char

(1003007Ch)]
1000B3DA cmp esi,esp
1000B3DC call _RTC_CheckEsp (10028A20h)
1000B3E1 mov eax,offset $L84050 (1000B3E7h)
1000B3E6 ret
$L84050:
1000B3E7 mov al,byte ptr [ebp-2ADh]
}
}

It may be that the destructor for an "automatic" std::string object runs
just after my

_asm xor eax, eax

and before the actual

ret

leaving EAX non-zero. You can step through the assembly code to find out.

If that's the case you can either dynamically allocate it and delete it
before the xor (that's nasty and brittle) or change the return type to BOOL
or int.

Perhaps someone else has a better alternative.

Thanks again

You are welcome.

Regards,
Will
www.ivrforbeginners.com

[Marcus Heege]

Hi DaTurk,

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@l53g2000cwa.googlegr oups.com...

Hi, I have a c# application that needs to access c++ libraries, so it
does this by using a managed layer of c++ CLI.

Just to avoid misunderstandings, can you please confirm you are using
C++/CLI (Visual Studio 2005) and no earlier version.

Marcus

[Ben Voigt]

"William DePalo [MVP VC++]" <wi***********@mvps.orgwrote in message
news:eD**************@TK2MSFTNGP06.phx.gbl...

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@q2g2000cwa.googlegro ups.com...

>Sounds like a good idea, I started loking at it, when I realized the
last itme I looked at assembly was about 15 years ago, and I sucked at
it then. WOuld you mind taking a peek at it? This is right at the
end of the function.

OK.

>1000B3BA call _RTC_CheckEsp (10028A20h)
//__asm mov eax,100
_asm xor eax, eax;
1000B3BF xor eax,eax
return false;
1000B3C1 mov byte ptr [ebp-2ADh],0
1000B3C8 mov byte ptr [ebp-4],0Ch
1000B3CC mov esi,esp
1000B3CE lea ecx,[err]
1000B3D4 call dword ptr
[__imp_std::basic_string<char,std::char_traits<char >,std::allocator<char>

>>>::~basic_string<char,std::char_traits<char>,std ::allocator<char

(1003007Ch)]
1000B3DA cmp esi,esp
1000B3DC call _RTC_CheckEsp (10028A20h)
1000B3E1 mov eax,offset $L84050 (1000B3E7h)
1000B3E6 ret
$L84050:
1000B3E7 mov al,byte ptr [ebp-2ADh]
}
}

It may be that the destructor for an "automatic" std::string object runs
just after my

_asm xor eax, eax

and before the actual

ret

leaving EAX non-zero. You can step through the assembly code to find out.

If that's the case you can either dynamically allocate it and delete it
before the xor (that's nasty and brittle) or change the return type to
BOOL or int.

Perhaps someone else has a better alternative.

You can force the stack destructor to run early using an extra enclosing
scope "{}". However, use of inline assembly to adjust EAX is brittle. You
should make the managed and unmanaged definitions match, both one byte (C++
bool and C# byte) or four byte (C++ BOOL and C# int). Relying on nothing
else to touch EAX is a very bad idea, and there are some calling conventions
where EAX isn't used for the return value.

>

>Thanks again

You are welcome.

Regards,
Will
www.ivrforbeginners.com

[William DePalo [MVP VC++]]

"Ben Voigt" <rb*@nospam.nospamwrote in message
news:ea**************@TK2MSFTNGP02.phx.gbl...

However, use of inline assembly to adjust EAX is brittle.

Well, life is full of compromises. Like a lot of people I encountered the
bug well after my API was designed. I could have gone back and changed all
of the signatures of the boolean functions but for me and my application it
was far less risky to add a single line immediately before the single return
instruction in all of the functions. YMMV.

You should make the managed and unmanaged definitions match, both one byte
(C++ bool and C# byte)

For me, when I think bool I think binary I think on/off, yes/no, true/false.
When I think byte I think small number. As I see it, this is a case of the
cure being worse than the disease. YMMV.

or four byte (C++ BOOL and C# int).

As above.

and there are some calling conventions where EAX isn't used for the return
value.

Hmm. I didn't know that. Under what conditions does a bool return value not
get returned in EAX?

Regards,
Will

[DaTurk]

On Feb 7, 12:44 pm, "Marcus Heege" <[e-mail]:

Hi DaTurk,

"DaTurk" <[e-mail] in message

news:11**********************@l53g2000cwa.googlegr oups.com...

Hi, I have a c# application that needs to access c++ libraries, so it
does this by using a managed layer of c++ CLI.

Just to avoid misunderstandings, can you please confirm you are using
C++/CLI (Visual Studio 2005) and no earlier version.

Marcus

My apologies, I'm using Mnaged C++, .net 1.1.

[Marcus Heege]

"DaTurk" <mm******@hotmail.comwrote in message
news:11**********************@v45g2000cwv.googlegr oups.com...

On Feb 7, 12:44 pm, "Marcus Heege" <[e-mail]:

>Hi DaTurk,

"DaTurk" <[e-mail] in message

news:11**********************@l53g2000cwa.googleg roups.com...

Hi, I have a c# application that needs to access c++ libraries, so it
does this by using a managed layer of c++ CLI.

Just to avoid misunderstandings, can you please confirm you are using
C++/CLI (Visual Studio 2005) and no earlier version.

Marcus

My apologies, I'm using Mnaged C++, .net 1.1.

OK. Now it makes sense. The bug you discribe is very likely the one Wiliam
DePalo has mentioned. Have you solved the problem now?