MarshalAs Problems with Arrays

[O.B.]

Below is a program that shows a test for marshaling data from a byte
array to a class structure. Unfortunately, there are two annoying
problems (bugs?) that I can't seem to get around.

The first is that if I set the Articulation array to an offset of 22
in the EntityState class instead of 24, I get the following error.
The thing is, it is properly aligned. And the amount of data to be
copied is smaller than the index of 22 anyway.

System.TypeLoad Exception: Could not load type 'Test.EntitySta te' from
assembly 'MUSE_Common, Version=7.4.0.2 5116, Culture=neutral ,
PublicKeyToken= null' because it contains an object field at offset 22
that is incorrectly aligned or overlapped by a non-object field.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

Help?

-------------------------------------------------

using System;
using System.Runtime. InteropServices ;

namespace MarshalTest
{
[StructLayout(La youtKind.Explic it, Size = 16)]
public struct Articulation
{
[FieldOffset(0)]
public byte parameterTypeDe signator;

[FieldOffset(1)]
public byte change;

[FieldOffset(2)]
public ushort idAttachedTo;

[FieldOffset(4)]
public uint parameterType;

[FieldOffset(8)]
public uint parameterValue1 ;

[FieldOffset(12)]
public uint parameterValue2 ;
}

[StructLayout(La youtKind.Explic it, Size = 20)]
unsafe public struct DeadReckoning
{
[FieldOffset(0)]
public byte header;

[FieldOffset(1)]
public fixed byte other[15];

[FieldOffset(16)]
public uint footer;
}

[StructLayout(La youtKind.Explic it)]
public class EntityState
{
[FieldOffset(0)]
public byte mySize;

[FieldOffset(1)]
public DeadReckoning deadRecking;

[FieldOffset(21)]
public byte temp1;

// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(Unman agedType.ByValA rray, ArraySubType =
UnmanagedType.S truct)]
public Articulation[] articulations;

public EntityState(byt e[] rawData)
{
int thisSize = rawData.Length;

IntPtr pData = Marshal.AllocHG lobal(thisSize) ;
Marshal.Copy(ra wData, 0, pData, thisSize);
Marshal.PtrToSt ructure(pData, this);
Marshal.FreeHGl obal(pData);

mySize = (byte)thisSize;
}

public byte[] ToRaw()
{
byte[] byteArray = new byte[mySize];
IntPtr pointer = Marshal.AllocHG lobal(mySize);
Marshal.Structu reToPtr(this, pointer, false);
Marshal.Copy(po inter, byteArray, 0, mySize);
Marshal.FreeHGl obal(pointer);
return byteArray;
}
}

class Program
{
static void Main(string[] args)
{
// Simulate raw data read from hardware
int dataSize = 22;
byte[] rawData = new byte[dataSize];
rawData[0] = (byte)dataSize;
for (int i = 0; i < dataSize; i++)
{
rawData[i] = (byte)(i + 1);
}

// Convert raw data to class
EntityState entity = new EntityState(raw Data);

// Compare original data with data stored in class
byte[] rawData2 = entity.ToRaw();

if (rawData.Length != rawData2.Length )
{
Console.WriteLi ne("**** ERROR *****: Data not the same
length.");
return;
}
bool dataIdentical = true;

// BUG: Offsets 3 - 16 are not identical
for (int i = 1; i < rawData.Length; i++)
{
if (rawData[i] != rawData2[i])
{
Console.WriteLi ne(i + ":\t" + rawData[i] +
" != " + rawData2[i] + "
*****");
dataIdentical = false;
}
else
{
Console.WriteLi ne(i + ":\t" + rawData[i] + " == "
+ rawData2[i]);
}
}
if (!dataIdentical )
{
Console.WriteLi ne("**** ERROR *****: Data not the
same.");
}
}
}
}

[Jeroen Mostert]

O.B. wrote:

Below is a program that shows a test for marshaling data from a byte
array to a class structure. Unfortunately, there are two annoying
problems (bugs?) that I can't seem to get around.

The first is that if I set the Articulation array to an offset of 22
in the EntityState class instead of 24, I get the following error.
The thing is, it is properly aligned.

I'm afraid not. Under default packing the framework requires reference
fields to be aligned to a word boundary, that's 4 bytes in this case. So 24
bytes is OK, 22 is not. You can get around this by using SequentialLayou t
and specifying a pack size of 1 (that is, no alignment padding). Using
ExplicitLayout won't work because the packing size is ignored if you do
that. I don't know why; it doesn't seem to be an entirely logical restriction.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

<snip>

>
// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(Unman agedType.ByValA rray, ArraySubType =
UnmanagedType.S truct)]
public Articulation[] articulations;

This isn't going to work. Marshalling as .ByValArray requires that you
specify the size of the array in the SizeConst parameter. If you don't know
the size in advance, you can't use .ByValArray. So you'll have to make clear
what you're going for, here. Is "articulati ons" a variable-sized array or a
fixed-size array? If it's variable-sized, what indicates the number of
elements and how?

OK, snipping the rest of the code. These declarations work:

[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public struct Articulation {
public byte parameterTypeDe signator;
public byte change;
public ushort idAttachedTo;
public uint parameterType;
public uint parameterValue1 ;
public uint parameterValue2 ;
}

[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public struct DeadReckoning {
public byte header;
[MarshalAs(Unman agedType.ByValA rray, SizeConst = 15)]
public byte[] other;
public uint footer;
}

[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public class EntityState {
public byte mySize;
public DeadReckoning deadRecking;
public byte temp1;

// This is probably wrong. You'll have to marshal this array manually
if it's variable-sized.
[MarshalAs(Unman agedType.ByValA rray, ArraySubType =
UnmanagedType.S truct, SizeConst = 1)]
public Articulation[] articulations;
}

It appears fixed arrays in substructures aren't marshalled correctly, or
else I just haven't found the right way to tickle the marshaller. Using a
regular array and marshalling it as .ByValArray works, though.

--
J.

[O.B.]

On Jul 13, 4:16 pm, Jeroen Mostert <jmost...@xs4al l.nlwrote:

O.B. wrote:

Below is a program that shows a test for marshaling data from a byte
array to a class structure. Unfortunately, there are two annoying
problems (bugs?) that I can't seem to get around.

The first is that if I set the Articulation array to an offset of 22
in the EntityState class instead of 24, I get the following error.
The thing is, it is properly aligned.

I'm afraid not. Under default packing the framework requires reference
fields to be aligned to a word boundary, that's 4 bytes in this case. So 24
bytes is OK, 22 is not. You can get around this by using SequentialLayou t
and specifying a pack size of 1 (that is, no alignment padding). Using
ExplicitLayout won't work because the packing size is ignored if you do
that. I don't know why; it doesn't seem to be an entirely logical restriction.

Thanks. You are a blessing! In my real code, the offsets are not
sequential, but I can insert some "dummy" offsets to get the
sequential behavior to work.

The next error is that when I set the FieldOffset to 24 for the
articulations array, it runs great ... except that only the first byte
within the "other" array in DeadReckoning is set, they rest of the
bytes are 0 in the array. If I comment out the articulations array,
all the bytes of the "other" array in DeadReckoning are copied
correctly.

<snip>

// BUG: Cannot set this offset to 22 without a runtime error.
Why?
[FieldOffset(24)]
[MarshalAs(Unman agedType.ByValA rray, ArraySubType =
UnmanagedType.S truct)]
public Articulation[] articulations;

This isn't going to work. Marshalling as .ByValArray requires that you
specify the size of the array in the SizeConst parameter. If you don't know
the size in advance, you can't use .ByValArray. So you'll have to make clear
what you're going for, here. Is "articulati ons" a variable-sized array or a
fixed-size array? If it's variable-sized, what indicates the number of
elements and how?

It is actually variable sized. There is another attribute in the
EntityState class that specifies the number of articulations. I have
modified the test program (below) to include this attribute and how
I'm manually marshaling the data (see the ctor of EntityState). What
is odd is that the Marshal.SizeOf operation is returning a size
smaller than the actual array size returned by ToRaw(). Am I safe to
assume that the data returned by ToRaw is legit and Marshal's SizeOf
operation cannot handle dynamic arrays? Or is the SizeOf operation
correct and I am overwriting memory I shouldn't be?

using System;
using System.Runtime. InteropServices ;

namespace MarshalTest
{
[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public struct Articulation
{
public byte parameterTypeDe signator;
public byte change;
public ushort idAttachedTo;
public uint parameterType;
public uint parameterValue1 ;
public uint parameterValue2 ;
}

[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public struct DeadReckoning
{
public byte header;
[MarshalAs(Unman agedType.ByValA rray, SizeConst = 15)]
public byte[] other;
public uint footer;
}

[StructLayout(La youtKind.Sequen tial, Pack = 1)]
public class EntityState
{
public byte mySize; // Offset 0
public DeadReckoning deadRecking; // Offset 1
public byte numArticulation s; // Offset 21
[MarshalAs(Unman agedType.ByValA rray, ArraySubType =
UnmanagedType.S truct)]
public Articulation[] articulations;

public EntityState(byt e[] rawData)
{
// Marshal the fixed data
int fixedSize = rawData.Length 22 ? 22 : rawData.Length;

IntPtr pData = Marshal.AllocHG lobal(fixedSize );
Marshal.Copy(ra wData, 0, pData, fixedSize);
Marshal.PtrToSt ructure(pData, this);
Marshal.FreeHGl obal(pData);

// Manually marshal values for each articulation
if (numArticulatio ns 0)
{
articulations = new Articulation[numArticulation s];
byte[] tempArray = new byte[16];
for (int i = 0; i < numArticulation s; i++)
{
Array.Copy(rawD ata, 22 + i * 16, tempArray, 0,
16);
unsafe
{
fixed (byte* pData2 = tempArray)
{
articulations[i] =

(Articulation)M arshal.PtrToStr ucture((IntPtr) pData2,
typeof(Articula tion));
}
}
}
}
}

public byte[] ToRaw()
{
byte[] byteArray = new byte[mySize];
IntPtr pointer = Marshal.AllocHG lobal(mySize);
Marshal.Structu reToPtr(this, pointer, false);
Marshal.Copy(po inter, byteArray, 0, mySize);
Marshal.FreeHGl obal(pointer);
return byteArray;
}
}

class Program
{
static void Main(string[] args)
{
// Simulate raw data read from hardware
int dataSize = 54;
byte[] rawData = new byte[dataSize];
rawData[0] = (byte)dataSize;
for (int i = 1; i < dataSize; i++)
{
rawData[i] = (byte)(i + 1);
}
rawData[21] = 2; // 2 articulations

// Convert raw data to class
EntityState entity = new EntityState(raw Data);

// Why is size of resulting structure smaller than 54?
if (Marshal.SizeOf (entity) != dataSize)
{
Console.WriteLi ne("Marshal says entity size is " +
Marshal.SizeOf( entity));
}

// Compare original data with data stored in class
byte[] rawData2 = entity.ToRaw();

if (rawData.Length != rawData2.Length )
{
Console.WriteLi ne("**** ERROR *****: Data not the same
length.");
return;
}
bool dataIdentical = true;

for (int i = 1; i < rawData.Length; i++)
{
if (rawData[i] != rawData2[i])
{
Console.WriteLi ne(i + ":\t" + rawData[i] + " != "
+ rawData2[i] + " *****");
dataIdentical = false;
}
else
{
Console.WriteLi ne(i + ":\t" + rawData[i] + " == "
+ rawData2[i]);
}
}
if (!dataIdentical )
{
Console.WriteLi ne("**** ERROR *****: Data not the
same.");
}
}
}
}

[Jeroen Mostert]

O.B. wrote:

It is actually variable sized. There is another attribute in the
EntityState class that specifies the number of articulations. I have
modified the test program (below) to include this attribute and how
I'm manually marshaling the data (see the ctor of EntityState). What
is odd is that the Marshal.SizeOf operation is returning a size
smaller than the actual array size returned by ToRaw(). Am I safe to
assume that the data returned by ToRaw is legit and Marshal's SizeOf
operation cannot handle dynamic arrays? Or is the SizeOf operation
correct and I am overwriting memory I shouldn't be?

Marshal.SizeOf will only return the amount of memory the marshaller thinks
is necessary (and also what it will use to marshal data back). The
marshaller can't handle variable-sized arrays, so it will always return the
size as if the array contained one element.

I have no idea why .ToRaw() works, actually. It shouldn't. :-) Apparently
the marshaller is clever enough to marshal the array fully, but not clever
enough to adjust its size measurements for it. You are not overwriting
memory that's not yours, so it shouldn't be a problem.

--
J.