473,836 Members | 1,509 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

x64 and BSTR allocation, what has changed?

In win32 mode, a BSTR was UINT length prefixed and terminated exactly by a
zero char.

So if you allocated "Hello World" that would allocate 28 bytes.

In x64 and (IA64 as well) it would become 32 bytes, because of the fact that
a pointer, is 8 bytes instead of 4 bytes.

The length prefix -still- is a UINT however and not a UINT_PTR.

But it seems that I'm still not quite complete on par. Is there any other
info or rules that have changed on top off the win32 BSTR allocations?

Thanks!

Mar 21 '06
37 5489
Hi Egbert!
Since I started it, let me clarify. The layout is documented
to the extent that the 4 bytes preceding the pointer are
its length (in bytes I might add). It is also documented
that you should use SysAllocString and friends to allocate
BSTRs. What is not documented, however, is how SysAlloc
functions allocate the raw memory. It was common knowledge
that on Win32 the allocated pointer is 4 bytes before the returned
pointer, but this does not make it documented! (It is also public
knowledge that CoTaskMemAlloc is used for the memory
allocation of course.) Apparently, this allowed Microsoft to
change the allocation strategy to reflect the alignment requirement
for Win64 and return a 64-bit aligned pointer from SysAlloc
functions. Of course this breaks the aforementioned public
knowledge (and we extended it obviously...). There's nothing
wrong with knowing undocumented aspects. What's wrong is
to expect that these assumptions will continue to be valid
going forward.


I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...


The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats
the BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would
only be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.
But this behaviour was never documented and might be change in the future...

Documented is only:
BSTR-4: Len
BSTR: String

Greetings
Jochen
Mar 23 '06 #31

"Jochen Kalmbach [MVP]" <no************ ********@holzma .de> wrote in message
news:%2******** *******@TK2MSFT NGP10.phx.gbl.. .
Hi Egbert!
I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this has
nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...
The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats the
BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would only
be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.


The pointer to every thing in 64-bit windows, =is= just a pointer. What it
points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with alignment.

Another possibility is that some design guys at MS decided that BSTRs should
not be limited by the 2GB / 4GB limit and after this decision, they found
that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The BSTR
is persisted by value, including the length-prefix! WHile a LPWSTR (if
interoped) would not have that problem, it is simply for instance a 5 GB
string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String


There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly rewrote
QuickBasic and PDS. Now he 's so rich, he can play cello and do what he
wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years on a
64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...

Mar 23 '06 #32
I think you misunderstand alignment. A pointer alignment deals
with the address the pointer points to, whereas data alignment
deals with the address a member of a struct is aligned on. The size
of the struct member dictates the minimum alignment required,
but pointer alignment is fixed - at least the CPU addressing
width, which in case of Win64 is 8 bytes. Since SysAlloc
functions return pointers, they must ensure proper alignment
as per the architecture. Note the alignment has only slight
performance implication on x86 architecture systems, but
misalignment can lead to program crash on other platforms.
Imagine you store a (properly aligned) struct in the buffer
returned by SysAllocStringB yteLen and access it after type-
casting. If the pointer is misaligned, so will be the struct members!

--
=============== =============== =======
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps .org
MVP VC FAQ: http://www.mvps.org/vcfaq
=============== =============== =======

"Egbert Nierop (MVP for IIS)" <eg***********@ nospam.invalid> wrote in
message news:%2******** ********@TK2MSF TNGP11.phx.gbl. ..

"Jochen Kalmbach [MVP]" <no************ ********@holzma .de> wrote in
message news:%2******** *******@TK2MSFT NGP10.phx.gbl.. .
Hi Egbert!
I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...


The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats the
BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would only
be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.


The pointer to every thing in 64-bit windows, =is= just a pointer. What it
points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with alignment.

Another possibility is that some design guys at MS decided that BSTRs
should not be limited by the 2GB / 4GB limit and after this decision, they
found that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The
BSTR is persisted by value, including the length-prefix! WHile a LPWSTR
(if interoped) would not have that problem, it is simply for instance a 5
GB string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String


There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly rewrote
QuickBasic and PDS. Now he 's so rich, he can play cello and do what he
wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years on
a 64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...

Mar 23 '06 #33
Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...

--
=============== =============== =======
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps .org
MVP VC FAQ: http://www.mvps.org/vcfaq
=============== =============== =======

"Alexander Nickolov" <ag********@mvp s.org> wrote in message
news:eq******** ******@TK2MSFTN GP11.phx.gbl...
I think you misunderstand alignment. A pointer alignment deals
with the address the pointer points to, whereas data alignment
deals with the address a member of a struct is aligned on. The size
of the struct member dictates the minimum alignment required,
but pointer alignment is fixed - at least the CPU addressing
width, which in case of Win64 is 8 bytes. Since SysAlloc
functions return pointers, they must ensure proper alignment
as per the architecture. Note the alignment has only slight
performance implication on x86 architecture systems, but
misalignment can lead to program crash on other platforms.
Imagine you store a (properly aligned) struct in the buffer
returned by SysAllocStringB yteLen and access it after type-
casting. If the pointer is misaligned, so will be the struct members!

--
=============== =============== =======
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps .org
MVP VC FAQ: http://www.mvps.org/vcfaq
=============== =============== =======

"Egbert Nierop (MVP for IIS)" <eg***********@ nospam.invalid> wrote in
message news:%2******** ********@TK2MSF TNGP11.phx.gbl. ..

"Jochen Kalmbach [MVP]" <no************ ********@holzma .de> wrote in
message news:%2******** *******@TK2MSFT NGP10.phx.gbl.. .
Hi Egbert!

I don't quite understand this.
Imagine, that the total prefix, still was 4 bytes, instead of 8, this
has nothing to do with alignment imho.
CoTaskMemAlloc, just returns a 64-bit pointer and in 32-bit windows, a
32-bit pointer. If alignment _plays_ a role, then not the SysAlloc*
functions deal with that, but CoTask* deals with it! I dare to bet my
shoes on this...

The problem is that the allocated pointer will be 8-byte aligned
(CoTaskMemAlloc will make this sure).

And now many functions pass the BSTR-Pointer to function which treats
the BSTR-Pointer as wchar_t*. And now if the "prefix" of the BSTR would
only be 4 bytes, then this BSTR-Pointer would be misaligned (not 8 byte
alligned).

That's IMHO the reason why the BSTR is now padding a 4-byte free-space
after the allocation-unit.


The pointer to every thing in 64-bit windows, =is= just a pointer. What
it points to, can even be 1 byte!.

I think the 4 byte waste, has to do with future plans, not with
alignment.

Another possibility is that some design guys at MS decided that BSTRs
should not be limited by the 2GB / 4GB limit and after this decision,
they found that a lot of interop (COM/RPC) would not work anymore.

Imagine, you do COM interop (ie RPC interop) with a 32-bit machine. The
BSTR is persisted by value, including the length-prefix! WHile a LPWSTR
(if interoped) would not have that problem, it is simply for instance a 5
GB string ( a little bit long, but though, it's the idea) would just be 0
terminated...

But this behaviour was never documented and might be change in the
future...

Documented is only:
BSTR-4: Len
BSTR: String


There are guys, who don't dare to bet on technology and guys who do.

For instance, NORTON was a programmer, who found undocumented IBM PC BIOS
function calls and used them as well. I know some other guys.
For instance, Ethan Whiner, an ASM programmer who wrote and nearly
rewrote QuickBasic and PDS. Now he 's so rich, he can play cello and do
what he wants the rest of his live.

THe ones who bet, sometimes get hurt, but get a much wider horizon.

Anyway, my product works for years, and it now also will work for years
on a 64-bit platform.
The nextplatform with 128-bit pointers :)) , would I still live then?
All that time, my product has the best scalability, and that's what I
prefer...


Mar 23 '06 #34

"Alexander Nickolov" <ag********@mvp s.org> wrote in message
news:eb******** ******@tk2msftn gp13.phx.gbl...
Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...


Thanks for the clarification.

Because you said this, I found out (by using CoGetMalloc), that the
allocation (length) alignment is 16 bytes, always. Applied to 32-bit or 64
bit systems.


Mar 23 '06 #35
Yes, that's to make sure you are alignment-compatible with
future processors for quite some time... :) I don't imagine we'll
be porting to 256-bit processors any time soon (these would
require 32-byte alignment).

However, SysAlloc functions are only 4-byte aligned since
they only need to support 32-bit CPUs. With Win64 their
alignment has to be upped up a bit to 8 bytes.

--
=============== =============== =======
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: ag********@mvps .org
MVP VC FAQ: http://www.mvps.org/vcfaq
=============== =============== =======

"Egbert Nierop (MVP for IIS)" <eg***********@ nospam.invalid> wrote in
message news:%2******** **********@tk2m sftngp13.phx.gb l...

"Alexander Nickolov" <ag********@mvp s.org> wrote in message
news:eb******** ******@tk2msftn gp13.phx.gbl...
Perhaps I wasn't very clear - struct member alignment is relative,
whereas pointer alignment is absolute. I hope this carries out
my message in more clear terms...


Thanks for the clarification.

Because you said this, I found out (by using CoGetMalloc), that the
allocation (length) alignment is 16 bytes, always. Applied to 32-bit or 64
bit systems.

Mar 24 '06 #36
"Egbert Nierop (MVP for IIS)" wrote:
I very simple program showed me that BSTR caching, only works within -the
same- thread. So, in IIS, when you use a lot of oleautomation (COM &
scripting) and each thread creates its own garbage etc, it is really
useless
to create a caching mechanism. In terms of scalability, using thread local
storage, is evil :)
I believe the original goal of BSTR caching was to reduce contention
on the process heap by keeping a small number of freed BSTRs
in TLS. It mostly succeeds in this, though the effect can usually be
seen only in artificial benchmarks that allocate tons of BSTRs from
multiple threads.

The real problem with BSTR caching is that it can result in very
inefficient memory usage for certain types of workloads. If you
set OA_NOCACHE=1 and your Process\Private Bytes drop
significantly then you're probably affected by this. Otherwise it
most likely makes no difference.
And OA_NOCACHE only helps as I understood, for a debug DLL as the
documentation says.


On XP and later OA_NOCACHE works with retail oleaut32.dll
(but it still should only be used for debugging purposes).

--
This posting is provided "AS IS" with no warranties, and confers no
rights.
Mar 24 '06 #37

"Pavel Lebedinsky [MSFT]" <pa***@online.m icrosoft.com> wrote in message
news:Oq******** ******@TK2MSFTN GP09.phx.gbl...
"Egbert Nierop (MVP for IIS)" wrote:
I very simple program showed me that BSTR caching, only works within -the
same- thread. So, in IIS, when you use a lot of oleautomation (COM &
scripting) and each thread creates its own garbage etc, it is really
useless
to create a caching mechanism. In terms of scalability, using thread
local
storage, is evil :)
I believe the original goal of BSTR caching was to reduce contention
on the process heap by keeping a small number of freed BSTRs
in TLS. It mostly succeeds in this, though the effect can usually be
seen only in artificial benchmarks that allocate tons of BSTRs from
multiple threads.


I understand. I've seen a more flat performance and memory usage on a HTTP
benchmark. In the tested script lots of variables were created and
persisted.
For some reason, reallocating is faster than allocating/deleting so my
string heap management is based on reallocation, instead of using a cache.

However, this benchmark was done on a PIII 600. I've never rebenched this.
The real problem with BSTR caching is that it can result in very
inefficient memory usage for certain types of workloads. If you
set OA_NOCACHE=1 and your Process\Private Bytes drop
significantly then you're probably affected by this. Otherwise it
most likely makes no difference.


This is true. Just because of the fact that IIS and ASP is -pure- scripting.
And OA_NOCACHE only helps as I understood, for a debug DLL as the
documentation says.


On XP and later OA_NOCACHE works with retail oleaut32.dll
(but it still should only be used for debugging purposes).


A component builder never should rely on this or set it for customers.

Mar 24 '06 #38

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
7129
by: Gilad Walden | last post by:
I use C# in .NET framework. I have an ActiveX implemented in C++ that has a COM interface method that gets as it’s out parameter a BSTR* . The interop translates this BSTR* into C# string. From my managed code, I am calling that function with ‘out’ parameter of ‘string’ type. For normal size strings it works fine, but when the string is very big (> 50,000 bytes), I get a null exception from the COM interop. What is the reason?...
5
4855
by: Karthik | last post by:
Hello! I am not a wizard in this area! Just need some help out in this. I am trying to convert bstr string to new character string. Here is the snippet of my code. **** Code Start**** GlobalInterfacePtr->GetName(bstr, bstrTNamePtr, &retVal);
5
2668
by: bluter | last post by:
We have server components which were created by a third party and compiled in VC++5 (sp3). They run fine on NT4 and 2000, however during testing of our migration to Server 2003, these components have been failing when performing Data Access functions. In particular we have been having problems with BSTR and _bstr_t. _bstr_t(BSTR*, 1) is the original code that the problem started with. Other findings are - 1) Converting a BSTR* to a...
12
3378
by: yufufi | last post by:
Hello, How does delete know how much memory to deallocate from the given pointer? AFAIK this informations is put there by new. new puts the size of the allocated memory before the just before the beginning of the array. But I couldn't find this information by looking at the memory. (Via VS2005 - C++) My second questions is, if there is a mechanism to know how much memory is allocated for the array, why don't we use it for things like...
0
9665
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10834
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10541
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10584
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10248
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7782
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5817
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4446
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4006
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.