x64 and BSTR allocation, what has changed?

[Egbert Nierop \(MVP for IIS\)]

In win32 mode, a BSTR was UINT length prefixed and terminated exactly by a
zero char.

So if you allocated "Hello World" that would allocate 28 bytes.

In x64 and (IA64 as well) it would become 32 bytes, because of the fact that
a pointer, is 8 bytes instead of 4 bytes.

The length prefix -still- is a UINT however and not a UINT_PTR.

But it seems that I'm still not quite complete on par. Is there any other
info or rules that have changed on top off the win32 BSTR allocations?

Thanks!

[David Lowndes]

>In win32 mode, a BSTR was UINT length prefixed and terminated exactly by a

zero char.

So if you allocated "Hello World" that would allocate 28 bytes.

In x64 and (IA64 as well) it would become 32 bytes, because of the fact that
a pointer, is 8 bytes instead of 4 bytes.

How do you come to that conclusion? Where does a pointer come into a
BSTR?

Dave

[Egbert Nierop \(MVP for IIS\)]

"David Lowndes" <Da****@example.invalid> wrote in message
news:6o********************************@4ax.com...

In win32 mode, a BSTR was UINT length prefixed and terminated exactly by
a
zero char.

So if you allocated "Hello World" that would allocate 28 bytes.

In x64 and (IA64 as well) it would become 32 bytes, because of the fact
that
a pointer, is 8 bytes instead of 4 bytes.
How do you come to that conclusion? Where does a pointer come into a
BSTR?

ok,

I'll explain this.
Not the pointer -itself- comes into but the pointer is to the UINT length
prefix.

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

But but... however, if you want the length, you need a UINT prefix, not a
UINT_PTR prefix.

I'm not stating this, just hesitating. Maybe someone knows the real facts...

BSTR __stdcall SysAllocString(const OLECHAR * input)
{
PWSTR retval = NULL;

if (input != NULL)
{
UINT_PTR slen = lstrlenW(input);
UINT_PTR newlen = slen * sizeof(OLECHAR)+ sizeof(UINT_PTR) +
sizeof(OLECHAR);
PWSTR temp = NULL;
temp = (PWSTR)::CoTaskMemAlloc(newlen);

if (temp != NULL)
{
UINT* plen = (UINT*)temp;
plen[0] = (UINT)slen * sizeof(OLECHAR);
retval = &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];
if (slen > 0)
CopyMemory(retval, input, (slen + 1) * sizeof(OLECHAR));
}
}
return retval;
}
Dave

[David Lowndes]

>> How do you come to that conclusion? Where does a pointer come into a

BSTR?

ok,

I'll explain this.
Not the pointer -itself- comes into but the pointer is to the UINT length
prefix.

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

I don't see any evidence for that assumption in the MSDN documentation
or the header files. SysStringLen is defined to return an UINT not
UINT_PTR.

Dave

[Jochen Kalmbach [MVP]]

Hi David!

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...

--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/

[Alexander Nickolov]

Let's not forget that the memory must be 8-byte aligned.
I would guess it allocates 8 bytes for the length prefix, and
ignores the first four. And also let's not forget that the
allocation strategy for BSTR is officially undocumented...
Only the layout is documented.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"David Lowndes" <Da****@example.invalid> wrote in message
news:km********************************@4ax.com...

How do you come to that conclusion? Where does a pointer come into a
BSTR?

ok,

I'll explain this.
Not the pointer -itself- comes into but the pointer is to the UINT length
prefix.

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

I don't see any evidence for that assumption in the MSDN documentation
or the header files. SysStringLen is defined to return an UINT not
UINT_PTR.

Dave

[Igor Tandetnik]

Jochen Kalmbach [MVP] <no********************@holzma.de> wrote:

A BSTR does not have a "length portion"... it is just a pointer...

Does too. The length in bytes is stored in the 4 bytes preceding the
byte referred to by the pointer. How do you expect SysStringLen to work
otherwise? Recall that BSTR may contain embedded NULs.
--
With best wishes,
Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not
necessarily a good idea. It is hard to be sure where they are going to
land, and it could be dangerous sitting under them as they fly
overhead. -- RFC 1925

[Egbert Nierop \(MVP for IIS\)]

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:uG**************@TK2MSFTNGP14.phx.gbl...

Hi David!

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

Yes, I'm saying that on 64-bit platforms, the length portion is 8 bytes!
But only the lower 4 bytes are effective.

I can prove this since I used a BSTR stub. This had special reasons. I had a
benchmark, that has proven that avoiding BSTR caching, was good on a Wintel
platform I did this by redefining all SysAlloc* strings to my own written
replacements.

On the 64-bit platform, this stub is working in a stand-alone EXE but not
inside a service and I can't figure why.

[Egbert Nierop \(MVP for IIS\)]

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:uk*************@TK2MSFTNGP09.phx.gbl...

Let's not forget that the memory must be 8-byte aligned.
I would guess it allocates 8 bytes for the length prefix, and
ignores the first four. And also let's not forget that the
allocation strategy for BSTR is officially undocumented...
Only the layout is documented.

Your guess is right about the 8 bytes length prefix.

The BSTR is just allocated by CoTaskMemAlloc and and CoTaskMemRealloc that
is using RtlHeap functions.

I have no source code, but my COM components, have been working for 3 years
on this. Except on the 64-bit platform that I am testing now.

[Egbert Nierop \(MVP for IIS\)]

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:uG**************@TK2MSFTNGP14.phx.gbl...

Hi David!

On X64 and IA64, the pointer is -not- to the UINT length prefix, but to a
UINT_PTR length prefix.

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...

You missed a lot by thinking this :)

[Alexander Nickolov]

Oh, I know how it's allocated :). I'm just pointing out that
this is officially undocumented.

So in Win64 we have:

byte len content
-8 4 unused
-4 4 length
0 n string

Or did I miss something?

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Egbert Nierop (MVP for IIS)" <eg***********@nospam.invalid> wrote in
message news:Ot**************@TK2MSFTNGP12.phx.gbl...

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:uk*************@TK2MSFTNGP09.phx.gbl...

Let's not forget that the memory must be 8-byte aligned.
I would guess it allocates 8 bytes for the length prefix, and
ignores the first four. And also let's not forget that the
allocation strategy for BSTR is officially undocumented...
Only the layout is documented.

Your guess is right about the 8 bytes length prefix.

The BSTR is just allocated by CoTaskMemAlloc and and CoTaskMemRealloc that
is using RtlHeap functions.

I have no source code, but my COM components, have been working for 3
years on this. Except on the 64-bit platform that I am testing now.

[David Lowndes]

>Let's not forget that the memory must be 8-byte aligned.

I would guess it allocates 8 bytes for the length prefix, and
ignores the first four.

OK.

.... so why does this matter to Egbert?

Dave

[Egbert Nierop \(MVP for IIS\)]

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:OS**************@TK2MSFTNGP11.phx.gbl...

Oh, I know how it's allocated :). I'm just pointing out that
this is officially undocumented.

So in Win64 we have:

byte len content
-8 4 unused
-4 4 length
0 n string and add a '\0' after the n string.
(unicode)
Or did I miss something?

It's working. I had still somewhere an invalid memory location calculation
(-4 instead of -8)

#include "stdafx.h"
#include "bstrnocache.h"

void __stdcall FreeString(BSTR * theString) throw()
{
if (theString != NULL && *theString != NULL )
{
SysFreeString2(*theString);
*theString = NULL;
}
}

int __stdcall SysReAllocStringByteLen2(BSTR * bstr, const char* input, UINT
cch) throw()
{
// ATLASSERT(bstr != NULL);
if (*bstr == NULL)
*bstr = SysAllocStringByteLen(input, cch);
else
{
UINT copyIntoLen = input != NULL ? lstrlenA(input): 0;
if (copyIntoLen > cch)
copyIntoLen = cch;
UINT_PTR newlen = cch + sizeof(OLECHAR) + sizeof(UINT_PTR);

PCHAR temp = (PCHAR)(*bstr) - sizeof(UINT_PTR);

temp = (PCHAR)::CoTaskMemRealloc(temp, newlen);

if (temp != NULL)
{
UINT_PTR* plen = (UINT_PTR*)temp;
UINT* plen2 = (UINT*)temp;
plen2[sizePtrCorrection] = cch ; //asign bytelength for BSTR
if (copyIntoLen > 0) //copy chars
memcpy(&plen[1],
input,
(copyIntoLen + 1));

temp[sizeof(UINT_PTR) + cch ] = 0; //terminate LPWSTR with a wide_string
temp[sizeof(UINT_PTR) + cch + 1] = 0;
*bstr = (BSTR)&temp[sizeof(UINT_PTR)];
}
}
return *bstr == NULL ? FALSE : TRUE;
}
int __stdcall SysReAllocStringLen2(BSTR * bstr, const OLECHAR* input, UINT
cch) throw()
{
//ATLASSERT(bstr != NULL);
if (*bstr == NULL)
{
*bstr = SysAllocStringLen2(input, cch);
}
else
{
UINT_PTR copyIntoLen = input != NULL ? lstrlenW(input): 0;
if (copyIntoLen > cch)
copyIntoLen = cch;
UINT_PTR newlen = cch * sizeof(OLECHAR) + sizeof(OLECHAR) +
sizeof(UINT_PTR);

PWSTR temp = *bstr - sizeof(UINT_PTR) / sizeof(OLECHAR);

temp = (PWSTR)::CoTaskMemRealloc(temp, newlen);

if (temp != NULL)
{
UINT_PTR* plen = (UINT_PTR*)temp;
UINT* plen2 = (UINT*)temp;
plen2[sizePtrCorrection] = cch * sizeof(OLECHAR); //asign bytelength for
BSTR
if (copyIntoLen > 0) //copy values
memcpy(&temp[sizeof(UINT_PTR) / sizeof(OLECHAR)],
input,
(copyIntoLen + 1) * sizeof(OLECHAR));

temp[sizeof(UINT_PTR) / sizeof(OLECHAR) + cch ] = 0; //terminate LPWSTR
*bstr = &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];

}

}
return *bstr == NULL ? FALSE : TRUE;
}
BSTR __stdcall SysAllocStringLen2(const OLECHAR * input, UINT cch) throw()
{
UINT_PTR copyIntoLen = input != NULL ? lstrlenW(input) : 0;
if (copyIntoLen > cch)
copyIntoLen = cch;

UINT_PTR newlen = cch * sizeof(OLECHAR) + sizeof(OLECHAR) +
sizeof(UINT_PTR);

PWSTR temp = (PWSTR)::CoTaskMemAlloc(newlen);
if (temp != NULL)
{
UINT_PTR* plen = (UINT_PTR*)temp;
UINT*plen2 = (UINT*)temp;
plen2[0] =0;
plen2[sizePtrCorrection] = cch * sizeof(OLECHAR);
if (copyIntoLen > 0)
memcpy(&plen[1],
input,
(copyIntoLen + 1) * sizeof(OLECHAR));

temp[sizeof(UINT_PTR) / sizeof(OLECHAR) + cch ] = 0;
return &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];
}
else
return NULL;

}

BSTR __stdcall SysAllocStringByteLen2(const char* input, UINT cch)
throw()
{
UINT_PTR copyIntoLen = input != NULL ? lstrlenA(input) : 0;
if (copyIntoLen > cch)
copyIntoLen = cch;

UINT_PTR newlen = cch + sizeof(OLECHAR) + sizeof(UINT_PTR);

PWSTR temp = (PWSTR)::CoTaskMemAlloc(newlen);
if (temp != NULL)
{
UINT_PTR* plen = (UINT_PTR*)temp;
UINT* plen2 = (UINT*)temp;
plen2[sizePtrCorrection] = cch;
if (copyIntoLen > 0)
memcpy(&temp[sizeof(UINT_PTR) / sizeof(OLECHAR)],
input,
copyIntoLen + 1);
char* zeroit = (char*)temp;
zeroit[sizeof(UINT_PTR) + cch ] = 0; //terminate LPWSTR with a wide_string
zeroit[sizeof(UINT_PTR) + cch + 1] = 0;
//temp[sizeof(unsigned int) / sizeof(wchar_t) + cch ] = 0;
return &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];
}
else
return NULL;

}
///<summary>
/// Identical to SysAllocString
///</summary>
BSTR __stdcall SysAllocString2(const OLECHAR * input) throw()
{
PWSTR retval = NULL;

if (input != NULL)
{
UINT_PTR slen = lstrlenW(input);
UINT_PTR newlen = slen * sizeof(OLECHAR)+ sizeof(UINT_PTR) +
sizeof(OLECHAR);
PWSTR temp = NULL;
temp = (PWSTR)::CoTaskMemAlloc(newlen);

if (temp != NULL)
{
UINT* plen = (UINT*)temp;
plen[sizePtrCorrection] = (UINT)slen * sizeof(OLECHAR);
retval = &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];
if (slen > 0)
memcpy(retval, input, (slen + 1) * sizeof(OLECHAR));
}
}
return retval;
}

int __stdcall SysReAllocString2(BSTR * bstr, const OLECHAR * input) throw()
{
//ATLASSERT(bstr != NULL);
bool didEmpty = FALSE;
if (*bstr == NULL)
{
*bstr = SysAllocString2(input);
}
else
{

if (input == NULL)
{
FreeString(bstr);
didEmpty = true;
}
else
{
UINT_PTR slen = lstrlenW(input);
UINT_PTR newlen = slen * sizeof(OLECHAR) + sizeof(OLECHAR) +
sizeof(UINT_PTR);

BSTR temp = *bstr - sizeof(INT_PTR) / sizeof(OLECHAR); //get real address
temp = (BSTR)::CoTaskMemRealloc(temp, newlen);

if (temp != NULL)
{
UINT_PTR* plen = (UINT_PTR*)temp;
UINT* plen2 = (UINT*)temp;
plen2[sizePtrCorrection] = (UINT)slen * sizeof(OLECHAR);
*bstr = &temp[sizeof(UINT_PTR) / sizeof(OLECHAR)];
memcpy(*bstr, input, (slen + 1) * sizeof(OLECHAR));
}
}

}
return *bstr == NULL && didEmpty == false? FALSE : TRUE;
}

UINT __stdcall SysStringByteLen2(BSTR theString) throw()
{
UINT retval = 0;
if (theString != NULL)
{
UINT* temp = (UINT*)theString;
temp--;
retval = *temp;
}
return retval;

}

UINT __stdcall SysStringLen2(BSTR theString) throw()
{
UINT retval = 0;
if (theString != NULL)
{
UINT* temp = (UINT*)theString;
temp--;
retval = *temp / sizeof(OLECHAR);
}
return retval;
}

[Jochen Kalmbach [MVP]]

Hi Egbert!

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...

You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

(As "Alexander said: "And also let's not forget that the
allocation strategy for BSTR is officially undocumented...")

Greetings
Jochen

[Egbert Nierop \(MVP for IIS\)]

"David Lowndes" <Da****@example.invalid> wrote in message
news:jk********************************@4ax.com...

Let's not forget that the memory must be 8-byte aligned.
I would guess it allocates 8 bytes for the length prefix, and
ignores the first four.

OK.

... so why does this matter to Egbert?

The prefix was concernig my question, the alignment not.

CoTaskMemAlloc nowhere states in the docs that you must 'align' memory. I
think that RtlHeap* etc do align the allocations by themselves but I'm not
that guru to know this.

[Egbert Nierop \(MVP for IIS\)]

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Hi Egbert!

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...
You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

Sure, it's your party, but BSTR is _not_ just a wchar_t* is documented!

(As "Alexander said: "And also let's not forget that the
allocation strategy for BSTR is officially undocumented...")

This is not rocket science :)
If you need an 11 wchar length string, what else must you store but the
length _and_ the compatibility with wchar_t* ie a zero terminator?

The other thing you need to know, is that all oleautomation memory allocs,
should go through CoTaskMemAlloc. Now I'm a rocket scientist :)

I started to hesitate on the x64 because of GPs but now I fixed it.

[David Lowndes]

>> ... so why does this matter to Egbert?

The prefix was concernig my question, the alignment not.

I still don't see why any of this was of any significance to you. What
are you doing that you need to know this for?

Dave

[Egbert Nierop \(MVP for IIS\)]

"David Lowndes" <Da****@example.invalid> wrote in message
news:0p********************************@4ax.com...

... so why does this matter to Egbert?

The prefix was concernig my question, the alignment not.

I still don't see why any of this was of any significance to you. What
are you doing that you need to know this for?

Well, in fact, in this thread I explained it. What I do? I'm a rocket
scientist :)

This explains it all...
http://technolog.nl/eprogrammer/

[David Lowndes]

>>>> ... so why does this matter to Egbert?

The prefix was concernig my question, the alignment not.
I still don't see why any of this was of any significance to you. What
are you doing that you need to know this for?

Well, in fact, in this thread I explained it. What I do? I'm a rocket
scientist :)

I read all the thread and *I* couldn't work out why you needed to
know.
This explains it all...
http://technolog.nl/eprogrammer/

Aha, the key bit being:

"I was managing my own BSTR allocation replacements"

I'd not sussed that from this thread.

Dave

[Eberhard Schefold]

Jochen Kalmbach [MVP] wrote:

BSTR is just "wchar_t*"...

You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

But you do realize that you must create, manipulate and destroy a BSTR
only in the documented ways, essentially using the Sysxxx functions? You
can provide a BSTR in instances where a LPCWSTR is expected, but not the
other way round. The preceding length value is an implementation detail,
and therefore we should not make any assumptions -- agreed. But if you
fail to remember that a BSTR is /not/ just a wchar_t array, you'll get
into trouble.

[Jochen Kalmbach [MVP]]

Hi Eberhard!

You missed a lot by thinking this :)
No problem for me... I never rely on undecomented features...

But you do realize that you must create, manipulate and destroy a BSTR
only in the documented ways, essentially using the Sysxxx functions?

Yes. That´s why I do not care about the internals of BSTR...
The preceding length value is an implementation detail,
and therefore we should not make any assumptions -- agreed. But if you
fail to remember that a BSTR is /not/ just a wchar_t array, you'll get
into trouble.

Yes. You are right. For BSTR you should only use the SysXxx functions.

Greetings
Jochen

[Scherbina Vladimir]

The internals are always good to know, especially for the OP case - if you
need to control allocations ...

--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:eV**************@TK2MSFTNGP12.phx.gbl...

Hi Eberhard!

You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

But you do realize that you must create, manipulate and destroy a BSTR
only in the documented ways, essentially using the Sysxxx functions?

Yes. That´s why I do not care about the internals of BSTR...

The preceding length value is an implementation detail, and therefore we
should not make any assumptions -- agreed. But if you fail to remember
that a BSTR is /not/ just a wchar_t array, you'll get into trouble.

Yes. You are right. For BSTR you should only use the SysXxx functions.

Greetings
Jochen

[Skywing]

The fact that BSTRs have a length prefix is fully and publicly documented <
see
http://msdn.microsoft.com/library/de...663a57d2b5.asp >

This is a very important thing that you need to know and understand if you
ever manage BSTRs yourself. I don't know where you got the idea that this
is an undocumented, secret implementation detail - but it's absolutely not.
If you've done nontrivial code that works with BSTRs, I'm not sure how you
could get by without knowing these sorts of things.

Additionally, even the allocation strategy used for BSTRs by oleaut32 is
also documented (see the above link).

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Hi Egbert!

So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
length portion rather than a 4 byte length? I'd be surprised if it
was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...

You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

(As "Alexander said: "And also let's not forget that the
allocation strategy for BSTR is officially undocumented...")

Greetings
Jochen

[Jochen Kalmbach [MVP]]

Hi Skywing!

The fact that BSTRs have a length prefix is fully and publicly documented <
see
http://msdn.microsoft.com/library/de...663a57d2b5.asp >
Great!
This is a very important thing that you need to know and understand if you
ever manage BSTRs yourself.
I don't see the need to know the internals of this BSTR...
I don't know where you got the idea that this
is an undocumented, secret implementation detail - but it's absolutely not.

Maybe I was not yet in the situation to handle "nontrivial" BSTR code...
But in general: It is always good to known the the underlying system is
doing...

--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/

[Egbert Nierop \(MVP for IIS\)]

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:u2**************@TK2MSFTNGP11.phx.gbl...

Hi Skywing!

The fact that BSTRs have a length prefix is fully and publicly documented
< see
http://msdn.microsoft.com/library/de...663a57d2b5.asp >

Great!

This is a very important thing that you need to know and understand if
you ever manage BSTRs yourself.

I don't see the need to know the internals of this BSTR...

Who told you that you need to?

When I ask a question, you just can assume that I see need and if you don't
know to answer, please refrain from vain comments then :)

In fact, I wanted a mechanisme, that has 'disabled' BSTR caching, without
affecting the whole process or using a sneaky API to disable caching.

Caching of strings in a process, is bad for performance for a Service.

You don't see a need, so don't waste time about that.

[Alexander Nickolov]

Since I started it, let me clarify. The layout is documented
to the extent that the 4 bytes preceding the pointer are
its length (in bytes I might add). It is also documented
that you should use SysAllocString and friends to allocate
BSTRs. What is not documented, however, is how SysAlloc
functions allocate the raw memory. It was common knowledge
that on Win32 the allocated pointer is 4 bytes before the returned
pointer, but this does not make it documented! (It is also public
knowledge that CoTaskMemAlloc is used for the memory
allocation of course.) Apparently, this allowed Microsoft to
change the allocation strategy to reflect the alignment requirement
for Win64 and return a 64-bit aligned pointer from SysAlloc
functions. Of course this breaks the aforementioned public
knowledge (and we extended it obviously...). There's nothing
wrong with knowing undocumented aspects. What's wrong is
to expect that these assumptions will continue to be valid
going forward.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Skywing" <sk**************@valhallalegends.com> wrote in message
news:Oh**************@TK2MSFTNGP11.phx.gbl...

The fact that BSTRs have a length prefix is fully and publicly documented
< see
http://msdn.microsoft.com/library/de...663a57d2b5.asp >

This is a very important thing that you need to know and understand if you
ever manage BSTRs yourself. I don't know where you got the idea that this
is an undocumented, secret implementation detail - but it's absolutely
not. If you've done nontrivial code that works with BSTRs, I'm not sure
how you could get by without knowing these sorts of things.

Additionally, even the allocation strategy used for BSTRs by oleaut32 is
also documented (see the above link).

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in
message news:%2****************@tk2msftngp13.phx.gbl...

Hi Egbert!

> So, you're saying that on 64-bit platforms, a BSTR has an 8 byte
> length portion rather than a 4 byte length? I'd be surprised if it
> was.

A BSTR does not have a "length portion"... it is just a pointer...

Or did I miss something?

BSTR is just "wchar_t*"...

You missed a lot by thinking this :)

No problem for me... I never rely on undecomented features...

(As "Alexander said: "And also let's not forget that the
allocation strategy for BSTR is officially undocumented...")

Greetings
Jochen

[Pavel Lebedinsky [MSFT]]

"Egbert Nierop (MVP for IIS)" wrote:

In fact, I wanted a mechanisme, that has 'disabled' BSTR caching, without
affecting the whole process or using a sneaky API to disable caching.

Caching of strings in a process, is bad for performance for a Service.

What are the symptoms you're seeing? High contention/cpu usage/
memory usage/something else? Does setting OA_NOCACHE=1 help?

What OS is this?

--
This posting is provided "AS IS" with no warranties, and confers no
rights.

[Pavel Lebedinsky [MSFT]]

BTW, I'm not suggesting OA_NOCACHE as a permanent workaround.
But it would be useful to know if it helps alleviate whatever symptoms
you are seeing.

--
This posting is provided "AS IS" with no warranties, and confers no
rights.

"Pavel Lebedinsky [MSFT]" wrote:

In fact, I wanted a mechanisme, that has 'disabled' BSTR caching, without
affecting the whole process or using a sneaky API to disable caching.

Caching of strings in a process, is bad for performance for a Service.

What are the symptoms you're seeing? High contention/cpu usage/
memory usage/something else? Does setting OA_NOCACHE=1 help?

What OS is this?

[Egbert Nierop \(MVP for IIS\)]

"Pavel Lebedinsky [MSFT]" <pa***@online.microsoft.com> wrote in message
news:e4**************@TK2MSFTNGP10.phx.gbl...

"Egbert Nierop (MVP for IIS)" wrote:

In fact, I wanted a mechanisme, that has 'disabled' BSTR caching, without
affecting the whole process or using a sneaky API to disable caching.

Caching of strings in a process, is bad for performance for a Service.

What are the symptoms you're seeing? High contention/cpu usage/
memory usage/something else? Does setting OA_NOCACHE=1 help?

What OS is this?

Any server version of windows (starting with windows 2000).

I very simple program showed me that BSTR caching, only works within -the
same- thread. So, in IIS, when you use a lot of oleautomation (COM &
scripting) and each thread creates its own garbage etc, it is really useless
to create a caching mechanism. In terms of scalability, using thread local
storage, is evil :)

And OA_NOCACHE only helps as I understood, for a debug DLL as the
documentation says.

And there is also a new API call, starting from XP Sp1 which disables
caching. But my component, should not disable caching in a process that I do
not host.

Thanks anyway for your comments.

[Egbert Nierop \(MVP for IIS\)]

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:e$**************@TK2MSFTNGP14.phx.gbl...

Since I started it, let me clarify. The layout is documented
to the extent that the 4 bytes preceding the pointer are
its length (in bytes I might add). It is also documented
that you should use SysAllocString and friends to allocate
BSTRs. What is not documented, however, is how SysAlloc
functions allocate the raw memory. It was common knowledge
that on Win32 the allocated pointer is 4 bytes before the returned
pointer, but this does not make it documented! (It is also public
knowledge that CoTaskMemAlloc is used for the memory
allocation of course.) Apparently, this allowed Microsoft to
change the allocation strategy to reflect the alignment requirement
for Win64 and return a 64-bit aligned pointer from SysAlloc
functions. Of course this breaks the aforementioned public
knowledge (and we extended it obviously...). There's nothing
wrong with knowing undocumented aspects. What's wrong is
to expect that these assumptions will continue to be valid
going forward.

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this has
nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my shoes
on this...

:)

[Jochen Kalmbach [MVP]]

Hi Egbert!

Since I started it, let me clarify. The layout is documented
to the extent that the 4 bytes preceding the pointer are
its length (in bytes I might add). It is also documented
that you should use SysAllocString and friends to allocate
BSTRs. What is not documented, however, is how SysAlloc
functions allocate the raw memory. It was common knowledge
that on Win32 the allocated pointer is 4 bytes before the returned
pointer, but this does not make it documented! (It is also public
knowledge that CoTaskMemAlloc is used for the memory
allocation of course.) Apparently, this allowed Microsoft to
change the allocation strategy to reflect the alignment requirement
for Win64 and return a 64-bit aligned pointer from SysAlloc
functions. Of course this breaks the aforementioned public
knowledge (and we extended it obviously...). There's nothing
wrong with knowing undocumented aspects. What's wrong is
to expect that these assumptions will continue to be valid
going forward.

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...

The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats
the BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would
only be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.
But this behaviour was never documented and might be change in the future...

Documented is only:
BSTR-4: Len
BSTR: String

Greetings
Jochen

[Egbert Nierop \(MVP for IIS\)]

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:%2***************@TK2MSFTNGP10.phx.gbl...

Hi Egbert!

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this has
nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...
The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats the
BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would only
be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.

The pointer to every thing in 64-bit windows, =is= just a pointer. What it
points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with alignment.

Another possibility is that some design guys at MS decided that BSTRs should
not be limited by the 2GB / 4GB limit and after this decision, they found
that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The BSTR
is persisted by value, including the length-prefix! WHile a LPWSTR (if
interoped) would not have that problem, it is simply for instance a 5 GB
string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String

There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly rewrote
QuickBasic and PDS. Now he 's so rich, he can play cello and do what he
wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years on a
64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...

[Alexander Nickolov]

I think you misunderstand alignment. A pointer alignment deals
with the address the pointer points to, whereas data alignment
deals with the address a member of a struct is aligned on. The size
of the struct member dictates the minimum alignment required,
but pointer alignment is fixed - at least the CPU addressing
width, which in case of Win64 is 8 bytes. Since SysAlloc
functions return pointers, they must ensure proper alignment
as per the architecture. Note the alignment has only slight
performance implication on x86 architecture systems, but
misalignment can lead to program crash on other platforms.
Imagine you store a (properly aligned) struct in the buffer
returned by SysAllocStringByteLen and access it after type-
casting. If the pointer is misaligned, so will be the struct members!

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Egbert Nierop (MVP for IIS)" <eg***********@nospam.invalid> wrote in
message news:%2****************@TK2MSFTNGP11.phx.gbl...

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in
message news:%2***************@TK2MSFTNGP10.phx.gbl...

Hi Egbert!

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...

The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats the
BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would only
be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.

The pointer to every thing in 64-bit windows, =is= just a pointer. What it
points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with alignment.

Another possibility is that some design guys at MS decided that BSTRs
should not be limited by the 2GB / 4GB limit and after this decision, they
found that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The
BSTR is persisted by value, including the length-prefix! WHile a LPWSTR
(if interoped) would not have that problem, it is simply for instance a 5
GB string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String

There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly rewrote
QuickBasic and PDS. Now he 's so rich, he can play cello and do what he
wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years on
a 64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...

[Alexander Nickolov]

Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:eq**************@TK2MSFTNGP11.phx.gbl...

I think you misunderstand alignment. A pointer alignment deals
with the address the pointer points to, whereas data alignment
deals with the address a member of a struct is aligned on. The size
of the struct member dictates the minimum alignment required,
but pointer alignment is fixed - at least the CPU addressing
width, which in case of Win64 is 8 bytes. Since SysAlloc
functions return pointers, they must ensure proper alignment
as per the architecture. Note the alignment has only slight
performance implication on x86 architecture systems, but
misalignment can lead to program crash on other platforms.
Imagine you store a (properly aligned) struct in the buffer
returned by SysAllocStringByteLen and access it after type-
casting. If the pointer is misaligned, so will be the struct members!

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Egbert Nierop (MVP for IIS)" <eg***********@nospam.invalid> wrote in
message news:%2****************@TK2MSFTNGP11.phx.gbl...

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in
message news:%2***************@TK2MSFTNGP10.phx.gbl...

Hi Egbert!

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...

The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats
the BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would
only be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.

The pointer to every thing in 64-bit windows, =is= just a pointer. What
it points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with
alignment.

Another possibility is that some design guys at MS decided that BSTRs
should not be limited by the 2GB / 4GB limit and after this decision,
they found that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The
BSTR is persisted by value, including the length-prefix! WHile a LPWSTR
(if interoped) would not have that problem, it is simply for instance a 5
GB string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String

There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly
rewrote QuickBasic and PDS. Now he 's so rich, he can play cello and do
what he wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years
on a 64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...

[Egbert Nierop \(MVP for IIS\)]

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:eb**************@tk2msftngp13.phx.gbl...

Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...

Thanks for the clarification.

Because you said this, I found out (by using CoGetMalloc), that the
allocation (length) alignment is 16 bytes, always. Applied to 32-bit or 64
bit systems.

[Alexander Nickolov]

Yes, that's to make sure you are alignment-compatible with
future processors for quite some time... :) I don't imagine we'll
be porting to 256-bit processors any time soon (these would
require 32-byte alignment).

However, SysAlloc functions are only 4-byte aligned since
they only need to support 32-bit CPUs. With Win64 their
alignment has to be upped up a bit to 8 bytes.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"Egbert Nierop (MVP for IIS)" <eg***********@nospam.invalid> wrote in
message news:%2******************@tk2msftngp13.phx.gbl...

"Alexander Nickolov" <ag********@mvps.org> wrote in message
news:eb**************@tk2msftngp13.phx.gbl...

Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...

Thanks for the clarification.

Because you said this, I found out (by using CoGetMalloc), that the
allocation (length) alignment is 16 bytes, always. Applied to 32-bit or 64
bit systems.

[Pavel Lebedinsky [MSFT]]

"Egbert Nierop (MVP for IIS)" wrote:

I very simple program showed me that BSTR caching, only works within -the
same- thread. So, in IIS, when you use a lot of oleautomation (COM &
scripting) and each thread creates its own garbage etc, it is really
useless
to create a caching mechanism. In terms of scalability, using thread local
storage, is evil :)
I believe the original goal of BSTR caching was to reduce contention
on the process heap by keeping a small number of freed BSTRs
in TLS. It mostly succeeds in this, though the effect can usually be
seen only in artificial benchmarks that allocate tons of BSTRs from
multiple threads.

The real problem with BSTR caching is that it can result in very
inefficient memory usage for certain types of workloads. If you
set OA_NOCACHE=1 and your Process\Private Bytes drop
significantly then you're probably affected by this. Otherwise it
most likely makes no difference.
And OA_NOCACHE only helps as I understood, for a debug DLL as the
documentation says.

On XP and later OA_NOCACHE works with retail oleaut32.dll
(but it still should only be used for debugging purposes).

--
This posting is provided "AS IS" with no warranties, and confers no
rights.

[Egbert Nierop \(MVP for IIS\)]

"Pavel Lebedinsky [MSFT]" <pa***@online.microsoft.com> wrote in message
news:Oq**************@TK2MSFTNGP09.phx.gbl...

"Egbert Nierop (MVP for IIS)" wrote:

I very simple program showed me that BSTR caching, only works within -the
same- thread. So, in IIS, when you use a lot of oleautomation (COM &
scripting) and each thread creates its own garbage etc, it is really
useless
to create a caching mechanism. In terms of scalability, using thread
local
storage, is evil :)
I believe the original goal of BSTR caching was to reduce contention
on the process heap by keeping a small number of freed BSTRs
in TLS. It mostly succeeds in this, though the effect can usually be
seen only in artificial benchmarks that allocate tons of BSTRs from
multiple threads.

I understand. I've seen a more flat performance and memory usage on a HTTP
benchmark. In the tested script lots of variables were created and
persisted.
For some reason, reallocating is faster than allocating/deleting so my
string heap management is based on reallocation, instead of using a cache.

However, this benchmark was done on a PIII 600. I've never rebenched this.
The real problem with BSTR caching is that it can result in very
inefficient memory usage for certain types of workloads. If you
set OA_NOCACHE=1 and your Process\Private Bytes drop
significantly then you're probably affected by this. Otherwise it
most likely makes no difference.

This is true. Just because of the fact that IIS and ASP is -pure- scripting.

And OA_NOCACHE only helps as I understood, for a debug DLL as the
documentation says.

On XP and later OA_NOCACHE works with retail oleaut32.dll
(but it still should only be used for debugging purposes).

A component builder never should rely on this or set it for customers.