473,289 Members | 2,106 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Random Access stream access in C#

Hi All,

I am a bit stuck with a project: Specifically, when making a database like
engine in 'the old days', I would have wrapped a record class with a stream
class, so I could have a file of records on disc, such that I could always
jump straight to the record number I wanted. Simple random file access.

Maybe they were fixed length records and I knew the n'th record was
(n*length of record) into the file. I could therefore jump straight to the
(n*length of record)th byte and read out the record, since I knew how long
each fixed length record was. Then by using a b-tree algorithm I could have
myself a mini-database engine.

The question is how do I implement the record management class in C#, using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes, how
do I jump straight to the record I want in a file of records on disc? (or in
memory, etc).

It may be that I have not thought this through enough, but I don't see an
easy way to do this.

If anybody could point me in the right direction I would be very grateful.

If I have described the problem badly give me a shout and I will do my best
to clarify,

Thanks,
Gary.
Apr 10 '06 #1
21 3733
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this these
days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
Hi All,

I am a bit stuck with a project: Specifically, when making a database like
engine in 'the old days', I would have wrapped a record class with a
stream
class, so I could have a file of records on disc, such that I could always
jump straight to the record number I wanted. Simple random file access.

Maybe they were fixed length records and I knew the n'th record was
(n*length of record) into the file. I could therefore jump straight to the
(n*length of record)th byte and read out the record, since I knew how long
each fixed length record was. Then by using a b-tree algorithm I could
have
myself a mini-database engine.

The question is how do I implement the record management class in C#,
using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use
pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes,
how
do I jump straight to the record I want in a file of records on disc? (or
in
memory, etc).

It may be that I have not thought this through enough, but I don't see an
easy way to do this.

If anybody could point me in the right direction I would be very grateful.

If I have described the problem badly give me a shout and I will do my
best
to clarify,

Thanks,
Gary.

Apr 10 '06 #2
Hi Vadym,

Many thanks for this. I like the index table idea. Sort of

1) I want the 268th record, so I ...
2) look up the 268th record in the index, which tell me where I can find ...
3) the 268th record in the 'proper file.

It is surprising that VB.Net has the FileGetObject Function et al, which I
think would do exactly what I want, but C# has not. (I am not a VB programmer
so I can't say for sure).

However, I appreciate the help.
regards,
Gary.

"Vadym Stetsyak" wrote:
Hello, Gary!

GB> The question is how do I implement the record management class in C#,
GB> using serialization, so that I can jump straight to a specific record?
GB> Since everything is wrapped up nicely in classes and objects. we don't
GB> use pointers much nowadays, nor know how 'big' classes/structs are in
GB> terms of bytes, how do I jump straight to the record I want in a file
GB> of records on disc? (or in memory, etc).

IMO in order to 'jump to record in a file' you'll have to design some record metainfo, that will describe records in the file ( memory ).
That is you will introduce some kind of structure to your storage file. Basically, it can look like the struct with record length and offset of the next record.

In the memory it will look like this

int recordLength
int nextRecordOffset
Record 1
int recordLength
int nextRecordOffset
Record 2

With this structure you have the ability to iterate records in file. Iteration is not the fastest way to get random record, for this purpose you can introduce index table, with fixed sizes,
each record of such index can contain the offset off appropriate record.

HTH
--
Regards, Vadym Stetsyak
www: http://vadmyst.blogspot

Apr 10 '06 #3
Hi Kevin,
I am an incurable 'must find out how things work' sort of person, and so I
just started doing this as a sort of hobby project when I have time at home.
My first interest was to write a B-Tree implementation and that sort of lead
me onto this.

By day I work as a programmer and you are right - I never would use this in
any production sense that I can think of. We use SQL Server virtually
exclusively, and I don't think I will be trying to rewrite that anytime soon
8-)

So the answer really is just as a hobby, voyage of discovery type thing -
and I just love writing code!

cheers,
Gary

"Kevin Spencer" wrote:
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this these
days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
Hi All,

I am a bit stuck with a project: Specifically, when making a database like
engine in 'the old days', I would have wrapped a record class with a
stream
class, so I could have a file of records on disc, such that I could always
jump straight to the record number I wanted. Simple random file access.

Maybe they were fixed length records and I knew the n'th record was
(n*length of record) into the file. I could therefore jump straight to the
(n*length of record)th byte and read out the record, since I knew how long
each fixed length record was. Then by using a b-tree algorithm I could
have
myself a mini-database engine.

The question is how do I implement the record management class in C#,
using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use
pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes,
how
do I jump straight to the record I want in a file of records on disc? (or
in
memory, etc).

It may be that I have not thought this through enough, but I don't see an
easy way to do this.

If anybody could point me in the right direction I would be very grateful.

If I have described the problem badly give me a shout and I will do my
best
to clarify,

Thanks,
Gary.


Apr 10 '06 #4
Good answer!

You can use a FileStream to do your "jumping around" in the file if you
like, and you still have access to pointers as well when you want/need them.

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:FD**********************************@microsof t.com...
Hi Kevin,
I am an incurable 'must find out how things work' sort of person, and so I
just started doing this as a sort of hobby project when I have time at
home.
My first interest was to write a B-Tree implementation and that sort of
lead
me onto this.

By day I work as a programmer and you are right - I never would use this
in
any production sense that I can think of. We use SQL Server virtually
exclusively, and I don't think I will be trying to rewrite that anytime
soon
8-)

So the answer really is just as a hobby, voyage of discovery type thing -
and I just love writing code!

cheers,
Gary

"Kevin Spencer" wrote:
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this
these
days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
> Hi All,
>
> I am a bit stuck with a project: Specifically, when making a database
> like
> engine in 'the old days', I would have wrapped a record class with a
> stream
> class, so I could have a file of records on disc, such that I could
> always
> jump straight to the record number I wanted. Simple random file access.
>
> Maybe they were fixed length records and I knew the n'th record was
> (n*length of record) into the file. I could therefore jump straight to
> the
> (n*length of record)th byte and read out the record, since I knew how
> long
> each fixed length record was. Then by using a b-tree algorithm I could
> have
> myself a mini-database engine.
>
> The question is how do I implement the record management class in C#,
> using
> serialization, so that I can jump straight to a specific record? Since
> everything is wrapped up nicely in classes and objects. we don't use
> pointers
> much nowadays, nor know how 'big' classes/structs are in terms of
> bytes,
> how
> do I jump straight to the record I want in a file of records on disc?
> (or
> in
> memory, etc).
>
> It may be that I have not thought this through enough, but I don't see
> an
> easy way to do this.
>
> If anybody could point me in the right direction I would be very
> grateful.
>
> If I have described the problem badly give me a shout and I will do my
> best
> to clarify,
>
> Thanks,
> Gary.


Apr 10 '06 #5
Gary,

Before you give up, wait until Jon gives an answer. He has given more
answers about this in this newsgroup. I can look it up as well, however I am
not sure if I have the right keywords in my mind, so wait on Jon as he is
active these days then I am almost sure that he will answer this.

It is (relative) simple to do as I have understand from Jon's messages. I
thought that it was this class and the corresponding writer class.

http://msdn.microsoft.com/library/de...classtopic.asp

Cor
Apr 11 '06 #6
Kevin,

In my opinion can there be plenty of reasons to do this. One of them is
speed in extreme situations the other one can be lack of memory (of course
than not with PC's or Servers).

(For the last it should be a very small database, I have used this in past.
However in normal use is the overhead in needed free memory (and the needed
updating/compress routines) very much. That was one of the reasons there
came the related databases).

A little extention to Gary message gives a fine description from a Relative
Database, the data was stored relatively to its starting pointer in an index
and its length. (And of course were the free areas one of the indexes)

Cor

"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> schreef in bericht
news:uT**************@TK2MSFTNGP02.phx.gbl...
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this
these days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
Hi All,

I am a bit stuck with a project: Specifically, when making a database
like
engine in 'the old days', I would have wrapped a record class with a
stream
class, so I could have a file of records on disc, such that I could
always
jump straight to the record number I wanted. Simple random file access.

Maybe they were fixed length records and I knew the n'th record was
(n*length of record) into the file. I could therefore jump straight to
the
(n*length of record)th byte and read out the record, since I knew how
long
each fixed length record was. Then by using a b-tree algorithm I could
have
myself a mini-database engine.

The question is how do I implement the record management class in C#,
using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use
pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes,
how
do I jump straight to the record I want in a file of records on disc? (or
in
memory, etc).

It may be that I have not thought this through enough, but I don't see an
easy way to do this.

If anybody could point me in the right direction I would be very
grateful.

If I have described the problem badly give me a shout and I will do my
best
to clarify,

Thanks,
Gary.


Apr 11 '06 #7
Hi Cor,
In my opinion can there be plenty of reasons to do this. One of them is
speed in extreme situations the other one can be lack of memory (of course
than not with PC's or Servers).
As I said, I was curious as to his reason. I didn't assume that he didn't
have a good reason. And he answered with an answer which I thought was a
very good reason.

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:up**************@TK2MSFTNGP02.phx.gbl... Kevin,

In my opinion can there be plenty of reasons to do this. One of them is
speed in extreme situations the other one can be lack of memory (of course
than not with PC's or Servers).

(For the last it should be a very small database, I have used this in
past. However in normal use is the overhead in needed free memory (and the
needed updating/compress routines) very much. That was one of the reasons
there came the related databases).

A little extention to Gary message gives a fine description from a
Relative Database, the data was stored relatively to its starting pointer
in an index and its length. (And of course were the free areas one of the
indexes)

Cor

"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> schreef in bericht
news:uT**************@TK2MSFTNGP02.phx.gbl...
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this
these days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
Hi All,

I am a bit stuck with a project: Specifically, when making a database
like
engine in 'the old days', I would have wrapped a record class with a
stream
class, so I could have a file of records on disc, such that I could
always
jump straight to the record number I wanted. Simple random file access.

Maybe they were fixed length records and I knew the n'th record was
(n*length of record) into the file. I could therefore jump straight to
the
(n*length of record)th byte and read out the record, since I knew how
long
each fixed length record was. Then by using a b-tree algorithm I could
have
myself a mini-database engine.

The question is how do I implement the record management class in C#,
using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use
pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes,
how
do I jump straight to the record I want in a file of records on disc?
(or in
memory, etc).

It may be that I have not thought this through enough, but I don't see
an
easy way to do this.

If anybody could point me in the right direction I would be very
grateful.

If I have described the problem badly give me a shout and I will do my
best
to clarify,

Thanks,
Gary.



Apr 11 '06 #8
Kevin,
In my opinion can there be plenty of reasons to do this. One of them is
speed in extreme situations the other one can be lack of memory (of
course than not with PC's or Servers).


As I said, I was curious as to his reason. I didn't assume that he didn't
have a good reason. And he answered with an answer which I thought was a
very good reason.

I like it forever to write about this by my so loved database type. They
were so extremely fast.

Forgive me.

Cor
Apr 11 '06 #9
Hi Cor,

I have writeen a database in the past as well, using C. That was about 11 or
12 years ago. It was certainly an educational experience. I learned a lot
about B-Tree indexing, relational database concepts, and so on. It helped me
quite a bit when I started using database products and tools.

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:uG**************@TK2MSFTNGP04.phx.gbl...
Kevin,
In my opinion can there be plenty of reasons to do this. One of them is
speed in extreme situations the other one can be lack of memory (of
course than not with PC's or Servers).


As I said, I was curious as to his reason. I didn't assume that he didn't
have a good reason. And he answered with an answer which I thought was a
very good reason.

I like it forever to write about this by my so loved database type. They
were so extremely fast.

Forgive me.

Cor

Apr 11 '06 #10
Hi Cor,

(Sorry for the delay in replying - hectic day at work!)

Thanks for tip - I will indeed monitor the thread to see if anybody else
wants to contribute. I don't know Jon but I am always apperciative of help.

Also thanks for the link. I will have another look at the binary reader
class, (I have not used it much), that your link points to.

If you can find any more about this I would appreciate it - but don't spend
too much time. As I said to Kevin, it is just a hobby/education/fun thing for
me; although I do appreciate any help of course.

regards,
Gary

"Cor Ligthert [MVP]" wrote:
Gary,

Before you give up, wait until Jon gives an answer. He has given more
answers about this in this newsgroup. I can look it up as well, however I am
not sure if I have the right keywords in my mind, so wait on Jon as he is
active these days then I am almost sure that he will answer this.

It is (relative) simple to do as I have understand from Jon's messages. I
thought that it was this class and the corresponding writer class.

http://msdn.microsoft.com/library/de...classtopic.asp

Cor

Apr 11 '06 #11
Kevin,

Ta for that - I will have a good look at the filestream class. We don't use
the IO classes much at work - its mostly data access/ADO type stuff - so I am
not too familiar with them. Thats the whole point of my little voyages of
discovery I guess.

Anyhow, thanks again,
regards,
Gary
(Sorry for the delay in replying - hectic day at work!)

"Kevin Spencer" wrote:
Good answer!

You can use a FileStream to do your "jumping around" in the file if you
like, and you still have access to pointers as well when you want/need them.

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:FD**********************************@microsof t.com...
Hi Kevin,
I am an incurable 'must find out how things work' sort of person, and so I
just started doing this as a sort of hobby project when I have time at
home.
My first interest was to write a B-Tree implementation and that sort of
lead
me onto this.

By day I work as a programmer and you are right - I never would use this
in
any production sense that I can think of. We use SQL Server virtually
exclusively, and I don't think I will be trying to rewrite that anytime
soon
8-)

So the answer really is just as a hobby, voyage of discovery type thing -
and I just love writing code!

cheers,
Gary

"Kevin Spencer" wrote:
What are your requirements? I'm curious as to why you want to write a
database engine. There are all sorts of ready-made solutions for this
these
days. What is the purpose of this?

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:5B**********************************@microsof t.com...
> Hi All,
>
> I am a bit stuck with a project: Specifically, when making a database
> like
> engine in 'the old days', I would have wrapped a record class with a
> stream
> class, so I could have a file of records on disc, such that I could
> always
> jump straight to the record number I wanted. Simple random file access.
>
> Maybe they were fixed length records and I knew the n'th record was
> (n*length of record) into the file. I could therefore jump straight to
> the
> (n*length of record)th byte and read out the record, since I knew how
> long
> each fixed length record was. Then by using a b-tree algorithm I could
> have
> myself a mini-database engine.
>
> The question is how do I implement the record management class in C#,
> using
> serialization, so that I can jump straight to a specific record? Since
> everything is wrapped up nicely in classes and objects. we don't use
> pointers
> much nowadays, nor know how 'big' classes/structs are in terms of
> bytes,
> how
> do I jump straight to the record I want in a file of records on disc?
> (or
> in
> memory, etc).
>
> It may be that I have not thought this through enough, but I don't see
> an
> easy way to do this.
>
> If anybody could point me in the right direction I would be very
> grateful.
>
> If I have described the problem badly give me a shout and I will do my
> best
> to clarify,
>
> Thanks,
> Gary.


Apr 11 '06 #12
Gary Bond <Ga***@community.nospam> wrote:

<snip>
The question is how do I implement the record management class in C#, using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes, how
do I jump straight to the record I want in a file of records on disc? (or in
memory, etc).


You would need to work out the format for the records yourself,
computing the size required for each of the fields etc. You could then
use FileStream (and possibly BinaryReader for convenience), setting the
Position property of the stream to seek from one record to another.

Now, that's not a lot of detail, but hopefully it's enough to get you
started - let me know if you need more help.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Apr 12 '06 #13
Hi Jon,

Thanks for taking the time to reply - I do appreciate it.

The amount of detail in your answer is fine - thats exactly how I would have
done it in an older language like C/Delphi, etc. Sort of ...

"I want the 67th record, so I multiply the record size by (67 -1), add on
any header size my file/stream probably contains, 'seek' to that location
and read a record. No worries"

I suppose in the world of the CTS and type safety everywhere, I was looking
for a way that did not involve playing around with the size of types, or
pointers, or unsafe code, (if I am going to use 'sizeof' for instance); a
more object oriented way if you will.

Does FileGetObject function et al in VB.Net fulfill this function? (I am not
too familiar with VB.Net, but a [very] quick look at the help file seemed to
say that it did).

Anyhow, if you have time, (and only if - this is only a hobby/fun project
for me), then I would be glad to hear your thoughts.

Thanks in any case for the help,
regards,
Gary.

"Jon Skeet [C# MVP]" wrote:
Gary Bond <Ga***@community.nospam> wrote:

<snip>
The question is how do I implement the record management class in C#, using
serialization, so that I can jump straight to a specific record? Since
everything is wrapped up nicely in classes and objects. we don't use pointers
much nowadays, nor know how 'big' classes/structs are in terms of bytes, how
do I jump straight to the record I want in a file of records on disc? (or in
memory, etc).


You would need to work out the format for the records yourself,
computing the size required for each of the fields etc. You could then
use FileStream (and possibly BinaryReader for convenience), setting the
Position property of the stream to seek from one record to another.

Now, that's not a lot of detail, but hopefully it's enough to get you
started - let me know if you need more help.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Apr 12 '06 #14
Gary Bond <Ga***@community.nospam> wrote:
Thanks for taking the time to reply - I do appreciate it.

The amount of detail in your answer is fine - thats exactly how I would have
done it in an older language like C/Delphi, etc. Sort of ...

"I want the 67th record, so I multiply the record size by (67 -1), add on
any header size my file/stream probably contains, 'seek' to that location
and read a record. No worries"
Yup.
I suppose in the world of the CTS and type safety everywhere, I was looking
for a way that did not involve playing around with the size of types, or
pointers, or unsafe code, (if I am going to use 'sizeof' for instance); a
more object oriented way if you will.
Well, you don't need unsafe code, and you don't need pointers in the C#
sense, although you'll definitely end up using references (as most
types are reference types).
Does FileGetObject function et al in VB.Net fulfill this function? (I am not
too familiar with VB.Net, but a [very] quick look at the help file seemed to
say that it did).

Anyhow, if you have time, (and only if - this is only a hobby/fun project
for me), then I would be glad to hear your thoughts.


Serialization (which I suspect is what FileGetObject does - I don't
know, I'm afraid) isn't really what you want for this kind of
application, I believe. It's got a lot of extra information such as
type info which you'd already know, and wouldn't be fixed length
either. For instance, if you have two records with strings of different
lengths, the serialized versions would be of different sizes.

The good thing about writing your own reading/writing of objects is:

1) It's very flexible.
2) You can guarantee to understand the file on inspection. No magic
involved.
3) It can be more efficient than a more generalised mechanism (in terms
of both space and time).

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Apr 12 '06 #15
If you want to use pointers (and it seems like you may), you will want to
use C#.

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Gary Bond" <Ga***@community.nospam> wrote in message
news:EC**********************************@microsof t.com...
Hi Jon,

Thanks for taking the time to reply - I do appreciate it.

The amount of detail in your answer is fine - thats exactly how I would
have
done it in an older language like C/Delphi, etc. Sort of ...

"I want the 67th record, so I multiply the record size by (67 -1), add on
any header size my file/stream probably contains, 'seek' to that location
and read a record. No worries"

I suppose in the world of the CTS and type safety everywhere, I was
looking
for a way that did not involve playing around with the size of types, or
pointers, or unsafe code, (if I am going to use 'sizeof' for instance); a
more object oriented way if you will.

Does FileGetObject function et al in VB.Net fulfill this function? (I am
not
too familiar with VB.Net, but a [very] quick look at the help file seemed
to
say that it did).

Anyhow, if you have time, (and only if - this is only a hobby/fun project
for me), then I would be glad to hear your thoughts.

Thanks in any case for the help,
regards,
Gary.

"Jon Skeet [C# MVP]" wrote:
Gary Bond <Ga***@community.nospam> wrote:

<snip>
> The question is how do I implement the record management class in C#,
> using
> serialization, so that I can jump straight to a specific record? Since
> everything is wrapped up nicely in classes and objects. we don't use
> pointers
> much nowadays, nor know how 'big' classes/structs are in terms of
> bytes, how
> do I jump straight to the record I want in a file of records on disc?
> (or in
> memory, etc).


You would need to work out the format for the records yourself,
computing the size required for each of the fields etc. You could then
use FileStream (and possibly BinaryReader for convenience), setting the
Position property of the stream to seek from one record to another.

Now, that's not a lot of detail, but hopefully it's enough to get you
started - let me know if you need more help.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Apr 12 '06 #16
Kevin Spencer <ke***@DIESPAMMERSDIEtakempis.com> wrote:
If you want to use pointers (and it seems like you may), you will want to
use C#.


I'm not sure why pointers would particularly be useful here. For most
of the time, when people talk about how they would have used pointers
in C/C++, references work perfectly well.

It's *possible* that pointers will be useful, but I doubt it.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Apr 13 '06 #17
> It's *possible* that pointers will be useful, but I doubt it.

I agree, Jon.

He started out by saying that he was doing this for self-education, which is
an admirable purpose, of course, and mentioned that he had previously used
pointers.

I agree that pointers have very limited use in .Net, and in fact, if I were
designing a database application "from scratch" I would not use the old
methods, for a number of reasons. In fact, I would probably use some form of
XML as the basis for something like this. XML can be compiled or pure text,
and with the speed of computing available today, the loss in terms of pure
speed would be more than made up for by the adherence to a standard that is
so extensible and cross-platform-compatible.

At any rate, it seemed like you had, for the most part, taken over the
topic, and I was merely making mention of the fact that, if he is accustomed
to using pointers, he might be better off to work with C#, which I prefer
for several reasons, possibly the least of which (but in my case absolutely
necessary at times) is pointer availability. I didn't want to go into gory
detail about it.

I still don't (I'm a bit snowed these days), so I'm happy to leave him in
your capable hands!

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Show me your certification without works,
and I'll show my certification
*by* my works.

"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om...
Kevin Spencer <ke***@DIESPAMMERSDIEtakempis.com> wrote:
If you want to use pointers (and it seems like you may), you will want to
use C#.


I'm not sure why pointers would particularly be useful here. For most
of the time, when people talk about how they would have used pointers
in C/C++, references work perfectly well.

It's *possible* that pointers will be useful, but I doubt it.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Apr 13 '06 #18
Hello, Jon!

JSC> I'm not sure why pointers would particularly be useful here. For most
JSC> of the time, when people talk about how they would have used pointers
JSC> in C/C++, references work perfectly well.

JSC> It's *possible* that pointers will be useful, but I doubt it.

Maybe, he ment, that pointers will be used to access data by address ( offset ) in the memory.

IMO if working with raw data, then pointers can become usefull.
--
Regards, Vadym Stetsyak
www: http://vadmyst.blogspot.com
Apr 13 '06 #19
Hi All,

First - thanks again for the help in my efforts to learn a bit.
I think I am getting somewhere, thanks to all your comments: For instance,
taking a 'duff test class' like this

[Serializable]
public class DuffClass
{
private string colour;
public string Colour
{
get { return colour; }
set { colour = value; }
}

public DuffClass(string startColour)
{
colour = startColour;
}
}

whose length could vary from one instantiation to the next, I can save it to
a file like this:

DuffClass dc = new DuffClass("This one is blue");
using (FileStream fs = new FileStream("Gaz.tst", FileMode.CreateNew))
{
using (BinaryWriter bw = new BinaryWriter(fs))
{
using (MemoryStream ms = new MemoryStream())
{
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, dc);
int StrmLength = (int)ms.Length;
// ms.Length gives size of serialized object here
// now save to the file
fs.Write(ms.ToArray(), 0, StrmLength);
}
}
}
Here I can find the length of bytes that represent the serialized object,
via putting it into a MemoryStream first, and I don't have to make
assumptions about the size of the underlying primitives/size of strings etc.
This was the bit that I could not 'get'. (Is there an easier way of doing
this?).

So, I can save lots of different size objects, into one storage file, and
get them back out again via the usual deserialize, since I know where they
start in the file, (assuming I keep track of their start positions via an
index of some description).

// open the file stream, seek to position 'xyz', then deserialize
using (FileStream fs = new FileStream("Gaz.tst", FileMode.Open))
{
fs.Seek(xyz, SeekOrigin.Begin);
IFormatter formatter = new BinaryFormatter();
DuffClass dc2 = (DuffClass)formatter.Deserialize(fs);
}

where 'xyz' is the start position of the object in the storage file.

I know this raises loads more questions than it answers, like

1) Variable length record managers are not easy to write, (what about
deleting records?, do I store the variable length records in lots of small,
fixed length, possibly non-contigous chunks or one big chunk?, what about a
'free-list' of available space in the file where deletions have occurred?,
etc)
2) I haven't even started on the B-Tree class yet, (which is where I started
all this)

but, at least your comments sort of broke my mental deadlock. So, what I am
going to do is:

1) See if I can really understand what is happening when you serialize
something, and maybe implement my own scheme to get 'stuff' into/out of a
stream, (ref Jon's comment on serialization above)
2) Actually write some code to to implement a variable record manager class
along these lines.

Could I come back with some code in the future, (might be a while), on this
topic for your comments?, or is there somewhere more appropriate if no?
In any case, thanks again all for giving me some of your time,
regards,
Gary.
Apr 13 '06 #20
Kevin Spencer <ke***@DIESPAMMERSDIEtakempis.com> wrote:

<snip>
At any rate, it seemed like you had, for the most part, taken over the
topic, and I was merely making mention of the fact that, if he is accustomed
to using pointers, he might be better off to work with C#, which I prefer
for several reasons, possibly the least of which (but in my case absolutely
necessary at times) is pointer availability. I didn't want to go into gory
detail about it.


Right. I think what I'm very keen on making sure of is that Gary
realises that most "normal" uses of pointers in C are achieved using
either indexes into arrays, or just normal references.

Hopefully between us we've got that point and your point that pointers
*are* available in C#, across now :)

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Apr 13 '06 #21
Gary Bond <Ga***@community.nospam> wrote:

<snip>
Here I can find the length of bytes that represent the serialized object,
via putting it into a MemoryStream first, and I don't have to make
assumptions about the size of the underlying primitives/size of strings etc.
This was the bit that I could not 'get'. (Is there an easier way of doing
this?).
I don't know of an easier way, no. It's fairly easy to encapsulate
though.
So, I can save lots of different size objects, into one storage file, and
get them back out again via the usual deserialize, since I know where they
start in the file, (assuming I keep track of their start positions via an
index of some description).
Yes - although that makes it a pain in terms of removing records, as
you noted later.
but, at least your comments sort of broke my mental deadlock. So, what I am
going to do is:

1) See if I can really understand what is happening when you serialize
something, and maybe implement my own scheme to get 'stuff' into/out of a
stream, (ref Jon's comment on serialization above)
Good plan. One of the things about serialization is that the type
information is embedded in the data - obviously that's somewhat
inefficient if you already know the type!
2) Actually write some code to to implement a variable record manager class
along these lines.
If you definitely need variable length records, that's fair enough.
Does your data definitely not work as fixed length records?
Could I come back with some code in the future, (might be a while), on this
topic for your comments?, or is there somewhere more appropriate if no?


Here is absolutely fine.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Apr 15 '06 #22

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Cybertof | last post by:
Hello, Is there a simple way to read a random access file that has been created with VB6 using direct writing to disk of Type....End Type structures ? I have not found this possibility in C#. ...
8
by: Danny Smith | last post by:
Hi, I need to read a file and be able to: 1. Find the current position in the stream 2. Have access to a handy ReadLine() method. Obviously the FileStream class supports random access, so...
1
by: Daniel | last post by:
streaming random filetypes from .aspx how to stream a file from inside a .aspx? e.g. so one could go href="./foo.aspx?file=bar.mp3" or href="./foo.aspx?foo.dat" etc. preferabley i would like...
5
by: TC | last post by:
I just noticed that the Stream class has a Seek method, which implies that a FileStream object can be used to manipulate a random-access file. Before now, I assumed that a Stream, as its name...
6
by: comp.lang.php | last post by:
/** * Generate the random security image * * @access public * @param $willUseFilePath (default false) boolean to determine if you will be using a file path * @param mixed $filePath (optional)...
4
by: tshad | last post by:
I am trying to set up an Image authorization where you type in the value that is in a picture to log on to our site. I found a program that is supposed to do it, but it doesn't seem to work. ...
3
by: Kza | last post by:
Hi, I am currently using xerces sax parser for c++, (I use DOM too, but I think SAX is more relevant here) for processing and displaying fairly large xml files. Usually I give xerces a filename,...
8
by: Daniel | last post by:
Hey guys Using Random(), how random is it, is it possible to be predicted? I have a requirement for a very good random number generator. I was doing something such as: Random randomSeed = new...
7
by: teh.sn1tch | last post by:
I created a random number generator for an application that uses a MersenneTwister class I found on the net. Basically I generate two random numbers from the MersenneTwister class and use each one...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.