Allocating memory for part of a C struct

[Chris Saunders]

Here is the declaration of a struct from WinIoCtl.h:

//
// Structures for FSCTL_TXFS_READ _BACKUP_INFORMA TION
//

typedef struct _TXFS_READ_BACK UP_INFORMATION_ OUT {
union {

//
// Used to return the required buffer size if return code is
STATUS_BUFFER_O VERFLOW
//

DWORD BufferLength;

//
// On success the data is copied here.
//

BYTE Buffer[1];
} DUMMYUNIONNAME;
} TXFS_READ_BACKU P_INFORMATION_O UT, *PTXFS_READ_BAC KUP_INFORMATION _OUT;

Now Buffer can be various lengths and I know of only one way to allocate
memory for it (by allocating memory for the whole struct). I have seen
other similar structs that are not a union and have multiple arrays with a
similar declaration. Is there any way to allocate memory for such arrays?

Regards
Chris Saunders

[Pavel Minaev]

Chris Saunders wrote:

Here is the declaration of a struct from WinIoCtl.h:

//
// Structures for FSCTL_TXFS_READ _BACKUP_INFORMA TION
//

typedef struct _TXFS_READ_BACK UP_INFORMATION_ OUT {
union {

//
// Used to return the required buffer size if return code is
STATUS_BUFFER_O VERFLOW
//

DWORD BufferLength;

//
// On success the data is copied here.
//

BYTE Buffer[1];
} DUMMYUNIONNAME;
} TXFS_READ_BACKU P_INFORMATION_O UT, *PTXFS_READ_BAC KUP_INFORMATION _OUT;

Now Buffer can be various lengths and I know of only one way to allocate
memory for it (by allocating memory for the whole struct). I have seen
other similar structs that are not a union and have multiple arrays with a
similar declaration. Is there any way to allocate memory for such arrays?

Allocating memory is easy - you just use Marshal.AllocHG lobal or
stackalloc (though you have to do the size calculation yourself,
accounting for extra bytes, and mind the layout, particularly the
padding at the end).

The problem is working with such a thing. .NET P/Invoke marshaller
doesn't understand VLA structs, so the best you can get is the same
trick as in C++ - declare a "fixed" single-element array at the end,
and then, after you allocate enough memory for the struct, treat it as
N-element array. C# doesn't do any bound checks for fixed arrays, so
you can do it just fine.

[Chris Saunders]

My thanks to Pavel Minaev.

Regards
Chris Saunders

[Ben Voigt [C++ MVP]]

Pavel Minaev wrote:

Chris Saunders wrote:

>Here is the declaration of a struct from WinIoCtl.h:

//
// Structures for FSCTL_TXFS_READ _BACKUP_INFORMA TION
//

typedef struct _TXFS_READ_BACK UP_INFORMATION_ OUT {
union {

//
// Used to return the required buffer size if return code is
STATUS_BUFFER_ OVERFLOW
//

DWORD BufferLength;

//
// On success the data is copied here.
//

BYTE Buffer[1];
} DUMMYUNIONNAME;
} TXFS_READ_BACKU P_INFORMATION_O UT,
*PTXFS_READ_BA CKUP_INFORMATIO N_OUT;

Now Buffer can be various lengths and I know of only one way to
allocate memory for it (by allocating memory for the whole struct).
I have seen other similar structs that are not a union and have
multiple arrays with a similar declaration. Is there any way to
allocate memory for such arrays?

Allocating memory is easy - you just use Marshal.AllocHG lobal or
stackalloc (though you have to do the size calculation yourself,
accounting for extra bytes, and mind the layout, particularly the
padding at the end).

The problem is working with such a thing. .NET P/Invoke marshaller
doesn't understand VLA structs, so the best you can get is the same
trick as in C++ - declare a "fixed" single-element array at the end,
and then, after you allocate enough memory for the struct, treat it as
N-element array. C# doesn't do any bound checks for fixed arrays, so
you can do it just fine.

Why the C# and p/invoke information in this group, and why use
Marshal.AllocHG lobal?

C++ interop is so much easier.

[Pavel Minaev]

"Ben Voigt [C++ MVP]" <rb*@nospam.nos pamwrote in message
news:ub******** ******@TK2MSFTN GP02.phx.gbl...

Why the C# and p/invoke information in this group, and why use
Marshal.AllocHG lobal?

I apologize - the groups come right after another, and the question is
rather atypical for a C/C++ developer (they usually just use malloc/new), so
I got confused there.

C++ interop is so much easier.

In fact, re-reading the original post, I see now that it doesn't even
reference any kind of interop at all - not even C++/CLI. So it was probably
a generic C++ question, which means that it boils down to the usual
non-standard-but-who-cares trick:

TXFS_READ_BACKU P_INFORMATION_O UT* trbio =
(TXFS_READ_BACK UP_INFORMATION_ OUT*)malloc(siz eof(TXFS_READ_B ACKUP_INFORMATI ON_OUT)
+ extra_bytes);

[Ben Voigt [C++ MVP]]

Pavel Minaev wrote:

"Ben Voigt [C++ MVP]" <rb*@nospam.nos pamwrote in message
news:ub******** ******@TK2MSFTN GP02.phx.gbl...

>Why the C# and p/invoke information in this group, and why use
Marshal.AllocH Global?

I apologize - the groups come right after another, and the question is
rather atypical for a C/C++ developer (they usually just use
malloc/new), so I got confused there.

>C++ interop is so much easier.

In fact, re-reading the original post, I see now that it doesn't even
reference any kind of interop at all - not even C++/CLI. So it was

True, but this is the C++/CLI group. So including some information on
interop might be useful (e.g. you can't use a buffer on the managed heap
unless you also pin it).

probably a generic C++ question, which means that it boils down to the
usual
non-standard-but-who-cares trick:

TXFS_READ_BACKU P_INFORMATION_O UT* trbio =

(TXFS_READ_BACK UP_INFORMATION_ OUT*)malloc(siz eof(TXFS_READ_B ACKUP_INFORMATI ON_OUT)
+ extra_bytes);

I don't see anything nonstandard about that. It's exactly what I would do,
except that in C++ I would use a cast keyword instead of the C-style cast.

[Pavel Minaev]

"Ben Voigt [C++ MVP]" <rb*@nospam.nos pamwrote in message
news:e%******** ********@TK2MSF TNGP04.phx.gbl. ..

True, but this is the C++/CLI group. So including some information on
interop might be useful (e.g. you can't use a buffer on the managed heap
unless you also pin it).

Well, there's always stackalloc... er, _malloca

>probably a generic C++ question, which means that it boils down to the
usual
non-standard-but-who-cares trick:

TXFS_READ_BACK UP_INFORMATION_ OUT* trbio =

(TXFS_READ_BAC KUP_INFORMATION _OUT*)malloc(si zeof(TXFS_READ_ BACKUP_INFORMAT ION_OUT)
+ extra_bytes);

I don't see anything nonstandard about that. It's exactly what I would
do, except that in C++ I would use a cast keyword instead of the C-style
cast.

It is non-standard in a sense that the ISO C++ standard prescribes undefined
behavior for this construct. It is precisely why it was treated specially in
C99 in form of open-sized arrays, to guarantee that this idiom was actually
blessed by the Standard.

[Ben Voigt [C++ MVP]]

Pavel Minaev wrote:

"Ben Voigt [C++ MVP]" <rb*@nospam.nos pamwrote in message
news:e%******** ********@TK2MSF TNGP04.phx.gbl. ..

>True, but this is the C++/CLI group. So including some information
on interop might be useful (e.g. you can't use a buffer on the
managed heap unless you also pin it).

Well, there's always stackalloc... er, _malloca

That wouldn't put a buffer on the managed heap.

>

>>probably a generic C++ question, which means that it boils down to
the usual
non-standard-but-who-cares trick:

TXFS_READ_BAC KUP_INFORMATION _OUT* trbio =

(TXFS_READ_BA CKUP_INFORMATIO N_OUT*)malloc(s izeof(TXFS_READ _BACKUP_INFORMA TION_OUT)
+ extra_bytes);

I don't see anything nonstandard about that. It's exactly what I
would do, except that in C++ I would use a cast keyword instead of
the C-style cast.

It is non-standard in a sense that the ISO C++ standard prescribes
undefined behavior for this construct. It is precisely why it was
treated specially in C99 in form of open-sized arrays, to guarantee
that this idiom was actually blessed by the Standard.

Well, I'm pretty sure you are allowed to use the memory returned by malloc
as the TXFS_READ_BACKU P_INFORMATION_O UT structure, as long as the size is at
least sizeof TXFS_READ_BACKU P_INFORMATION_O UT. I suppose the issue is that
the compiler is not required to layout the structure with the array at the
end, so you might be walking over a footer or some other field when you
reference using the array name? Explicitly referencing the memory
immediately past the structure should be ok by the standard as long as the
requirement that the extra space be allocated by the caller is stated,
right?

[Pavel Minaev]

On Sep 8, 5:30*pm, "Ben Voigt [C++ MVP]" <[e-mail] spamwrote:

"Ben Voigt [C++ MVP]" <[e-mail] spamwrote in message
news:e%******** ********@TK2MSF TNGP04.phx.gbl. ..

True, but this is the C++/CLI group. *So including some information
on interop might be useful (e.g. you can't use a buffer on the
managed heap unless you also pin it).

Well, there's always stackalloc... er, _malloca

That wouldn't put a buffer on the managed heap.

Why would it need to be on the managed heap, if it's just used for
interop?

It is non-standard in a sense that the ISO C++ standard prescribes
undefined behavior for this construct. It is precisely why it was
treated specially in C99 in form of open-sized arrays, to guarantee
that this idiom was actually blessed by the Standard.

Well, I'm pretty sure you are allowed to use the memory returned by malloc
as the TXFS_READ_BACKU P_INFORMATION_O UT structure, as long as the size isat
least sizeof TXFS_READ_BACKU P_INFORMATION_O UT. *I suppose the issue is that
the compiler is not required to layout the structure with the array at the
end, so you might be walking over a footer or some other field when you
reference using the array name? *Explicitly referencing the memory
immediately past the structure should be ok by the standard as long as the
requirement that the extra space be allocated by the caller is stated,
right?

Unfortunately, no. The Standard explicitly states that accessing an
array using an out-of-bounds index is U.B., without giving any
exceptions (such as the fact that memory accessed is allocated, etc).
In fact, you don't even have to dereference to get U.B. - it's already
happening at the pointer arithmetic stage. Recall that (p + i), where
p is pointer, and i is index, is only valid pointer arithmetics when
you're within the same object (object here defined as either array or
an individual value that the pointer is pointing to - not the
allocated memory block to which it belongs).

[Ben Voigt [C++ MVP]]

Pavel Minaev wrote:

On Sep 8, 5:30 pm, "Ben Voigt [C++ MVP]" <[e-mail] spamwrote:

>>"Ben Voigt [C++ MVP]" <[e-mail] spamwrote in message
news:e%****** **********@TK2M SFTNGP04.phx.gb l...
True, but this is the C++/CLI group. So including some information
on interop might be useful (e.g. you can't use a buffer on the
managed heap unless you also pin it).

>>Well, there's always stackalloc... er, _malloca

That wouldn't put a buffer on the managed heap.

Why would it need to be on the managed heap, if it's just used for
interop?

You presented it as a counter-example to my statement that you can't use a
buffer on the managed heap without pinning. Or that's how I understood it.

>

>>It is non-standard in a sense that the ISO C++ standard prescribes
undefined behavior for this construct. It is precisely why it was
treated specially in C99 in form of open-sized arrays, to guarantee
that this idiom was actually blessed by the Standard.

Well, I'm pretty sure you are allowed to use the memory returned by
malloc as the TXFS_READ_BACKU P_INFORMATION_O UT structure, as long as
the size is at least sizeof TXFS_READ_BACKU P_INFORMATION_O UT. I
suppose the issue is that the compiler is not required to layout the
structure with the array at the end, so you might be walking over a
footer or some other field when you reference using the array name?
Explicitly referencing the memory immediately past the structure
should be ok by the standard as long as the requirement that the
extra space be allocated by the caller is stated, right?

Unfortunately, no. The Standard explicitly states that accessing an
array using an out-of-bounds index is U.B., without giving any
exceptions (such as the fact that memory accessed is allocated, etc).
In fact, you don't even have to dereference to get U.B. - it's already
happening at the pointer arithmetic stage. Recall that (p + i), where
p is pointer, and i is index, is only valid pointer arithmetics when
you're within the same object (object here defined as either array or
an individual value that the pointer is pointing to - not the
allocated memory block to which it belongs).

I disagree. You can cast back to the original allocated object type, which
is array-of-char, and then all indices within the originally requested size
are legal and well-defined, including those past the sub-range of memory
which has been used as an object of class type.