473,326 Members | 2,111 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Memory Streams and Struct Pointers Revisited

_TR
I love C# and I've been porting a lot of my older C++ code, but there
is one app that I haven't quite figured out yet. I suppose I could
re-engineer the app from the ground up, but I've already accumulated a
lot of data files, so the format is set for now.

Basically, the data format consists of inlined Structs of different
types and lengths. I can decode the first byte to tell what type of
struct it is, then point a struct pointer to it to decode the
following bytes. The format is fairly easy to work with, and again I
already have megabytes of files in that format, so I don't want to
change it yet.

For speed, I read the entire file into RAM ahead of time rather than
paging it from disk.

Here's the question: How well would that approach translate to C#?
Someone claimed that there was no need for 'unsafe' code, given that
memory streams could be used to read the bytes into structs, but this
seems to require additional work (loading the structs as opposed to
simply pointing a *struct directly into the stream). Is there a
graceful way around this?

The fallback plan is to use VS2005's CLI/C++. I find C++'s syntax
tedious compared to C#, but the new CLI pointer mechanisms might be
preferable; I don't want the GC moving streams at awkward moments
(I'm trying to maintain good realtime response).

Any thoughts?

Nov 17 '05 #1
3 1831
Tough one :)

My guess is that you would have to redefine the types of those structs
in C# regardless of what implementation strategy you'll follow. Of
course you'll have to specify packing and other memory layout
attributes to make sure that they remain compatible with the memory
layout of the C++ structs.

I hope I have guessed correctly so far.

If that is indeed the case, then I would use P/Invoke to run the C++
code which unpacks the bytes into the C# structs. To me, It looks like
the easiest--abliet not the coolest--way to go.

Sorry I couldn't be of much help. Any other ideas anyone?

Nov 17 '05 #2
Tough one :)

My guess is that you would have to redefine the types of those structs
in C# regardless of what implementation strategy you'll follow. Of
course you'll have to specify packing and other memory layout
attributes to make sure that they remain compatible with the memory
layout of the C++ structs.

I hope I have guessed correctly so far.

If that is indeed the case, then I would use P/Invoke to run the C++
code which unpacks the bytes into the C# structs. To me, It looks like
the easiest--abliet not the coolest--way to go.

Sorry I couldn't be of much help. Any other ideas anyone?

Nov 17 '05 #3
_TR wrote:
For speed, I read the entire file into RAM ahead of time rather than
paging it from disk.
I don't really think that's likely to gain anything. You cannot
memory-map the bytes you read into CLR objects anyway.
Here's the question: How well would that approach translate to C#?
Write a parser for the structs. and a registry for them. Something along
the lines of:

public interface StructParser {
int Tag { get; }
object Parse(Stream s);
}
public class FooStructParser: StructParser {
const int Tag = 0xFFE4;
object Parse(Stream s) {
int x = Tools.ReadInt(s);
...
return new Foor(x);
}
}

public class FileReader {
IDictionary Parsers = new HashTable();
public void Register(StructParser p) {
Parsers[p.Tag] = p;
}
class Enumerator: IEnumrator {
Stream s;
FileReader r;
Object current;
public Enumerator(FileReader r, Stream s)
{ this.s = s; this.r = r; }
public object Current {
get {
if ( current == null )
throw new IndexOutOfBoundsException(); }
else
return current;
}
}
public bool MoveNext() {
try {
int x = Tools.ReadInt(s);
current = ((StructParser)Parsers[x]).Parse(s);
return true;
} catch ( EndOfStreamException ) {
current = null;
return false;
}
}
}
public Enumerator Objects(Stream s) { return new Enumerator(this,s); }
}

Note that the above can be made to work for composite and recusive
structs... :)
Someone claimed that there was no need for 'unsafe' code, given that
memory streams could be used to read the bytes into structs, but this
seems to require additional work (loading the structs as opposed to
simply pointing a *struct directly into the stream). Is there a
graceful way around this?
I don't think that's gonna be possible -- in C++ you could probably do
it if the data is POD without RTTI using placement-new for the objects,
but I don't think I would do it if I were you :)
The fallback plan is to use VS2005's CLI/C++. I find C++'s syntax
tedious compared to C#, but the new CLI pointer mechanisms might be
preferable; I don't want the GC moving streams at awkward moments
(I'm trying to maintain good realtime response).


You can lock data with "fixed", then the GC cannot move it.

Try coding a parser for a few of the structs and see if it's not really
the easy way out.

--
Helge Jensen
mailto:he**********@slog.dk
sip:he**********@slog.dk
-=> Sebastian cover-music: http://ungdomshus.nu <=-
Nov 17 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: Roman Hartmann | last post by:
hello, I do have a question regarding structs. I have a struct (profil) which has a pointer to another struct (point). The struct profil stores the coordinates of points. The problem is that I...
3
by: Christian F | last post by:
Hi, I'm a C-newbie and I would like to know if I am doing something wrong in the code below. It is working, but I'm afraid it might not be correct because I don't really understand everything of...
16
by: ben beroukhim | last post by:
I have huge number of legacy code which use standard files functions. I would like to pass a memory pointer rather than a FILE pointer. I am trying to use FILEs in the code to refer to memory...
15
by: puzzlecracker | last post by:
Got Confused on the interview with memory alligment questions... PLEASE HELP -- How much bytes of memory will structs below take on 32 bit machine? What about 64 bit machine? Why is it different?...
9
by: Alfonso Morra | last post by:
Hi, I am having some probs with copying memory blocks around (part of a messaging library) and I would like some confirmation to make sure that I'm going about things the right way. I have...
0
by: _TR | last post by:
I love C# and I've been porting a lot of my older C++ code, but there is one app that I haven't quite figured out yet. I suppose I could re-engineer the app from the ground up, but I've already...
8
by: nkrisraj | last post by:
Hi, I have a following structure: typedef struct { RateData rdr; int RateID; char RateBalance; } RateInfo;
16
by: Pedro Graca | last post by:
I have a file with different ways to write numbers ---- 8< (cut) -------- 0: zero, zilch,, nada, ,,,, empty , void, oh 1: one 7: seven 2: two, too ---- >8 -------------- ...
2
by: Mike | last post by:
Hi, I am new to C and having problems with the following program. Basically I am trying to read some files, loading data structures into memory for latter searching. I am trying to use structres...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.