using WriteFileEX from C#

[tom]

I'm trying to implement a class that writes blocks of data asynchronously to disk. Each block is an array of ulongs. I tried to use the standard beginwrite functions, but apparently they only support byte arrays. Due to the nature of my problem (huge amounts of data) it would be very cpu unfriendly to convert all the ulongs to bytes.

Therefore I switch to the native methods( WriteFileEx) but I got in trouble dealing with the overlapped structure and the callback routines. Does anybody know if there's some good c# example on how to deal with these.

Thanks for helping me out !

Tom

[John Wood]

Have you thought about putting the writing into a worker thread (create a
delegate and use the threadpool is easiest) ?

You can then use BinaryWriter to write out your ulongs, and it won't
interefere with the main thread.

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:B6**********************************@microsof t.com...

I'm trying to implement a class that writes blocks of data asynchronously to disk. Each block is an array of ulongs. I tried to use the standard
beginwrite functions, but apparently they only support byte arrays. Due to
the nature of my problem (huge amounts of data) it would be very cpu
unfriendly to convert all the ulongs to bytes.
Therefore I switch to the native methods( WriteFileEx) but I got in trouble dealing with the overlapped structure and the callback routines.
Does anybody know if there's some good c# example on how to deal with these.
Thanks for helping me out !

Tom

[AlexS]

Hi, Tom

You should consider using BinaryWriter with FileStream, which is opened in
asynchronous mode. This doesn't require explicit conversion coding in your
program.
Did you try this?

Generally speaking, your statement about "cpu unfriendliness" is not
substantiated by anything except your guess here, so I don't think this
style of reasoning is valid. Did you do any tests? Considering performance
gap between CPU and disk IO you might be surprised to see that whatever
conversions you do do not influence speed of IO.

HTH
Alex

"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:B6**********************************@microsof t.com...

I'm trying to implement a class that writes blocks of data asynchronously to disk. Each block is an array of ulongs. I tried to use the standard
beginwrite functions, but apparently they only support byte arrays. Due to
the nature of my problem (huge amounts of data) it would be very cpu
unfriendly to convert all the ulongs to bytes.
Therefore I switch to the native methods( WriteFileEx) but I got in trouble dealing with the overlapped structure and the callback routines.
Does anybody know if there's some good c# example on how to deal with these.
Thanks for helping me out !

Tom

[John Wood]

well an array of ulongs is just a loop writing out a ulong :)

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:6D**********************************@microsof t.com...

Hi John,

the reason I didn't use the Binary writer is that it only supports single ulong's, not arrays of ulong's (it only supports arrays of type byte or
char).
Working with the threadpool would indeed have been a solution. The reason why I decided not to go along this path is that the native methods allow
extra optimizations, like for example the one for sequential access.
Tom

"John Wood" wrote:

Have you thought about putting the writing into a worker thread (create a delegate and use the threadpool is easiest) ?

You can then use BinaryWriter to write out your ulongs, and it won't
interefere with the main thread.

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:B6**********************************@microsof t.com...

I'm trying to implement a class that writes blocks of data asynchronously

to disk. Each block is an array of ulongs. I tried to use the standard
beginwrite functions, but apparently they only support byte arrays. Due to the nature of my problem (huge amounts of data) it would be very cpu
unfriendly to convert all the ulongs to bytes.

Therefore I switch to the native methods( WriteFileEx) but I got in

trouble dealing with the overlapped structure and the callback routines.
Does anybody know if there's some good c# example on how to deal with

these.
Thanks for helping me out !

Tom

[John Wood]

You could also use BitConverter.GetBytes() on each long, in a loop, to copy
the longs into the byte array.

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:5F**********************************@microsof t.com...

You know... if you convert the ulong[] to a byte[] (ulong by ulong) you end up with somethign that is exactly the same as the original... at least
if you look how it's respresented in the memory... so if I just could assign
the address of the ulong[] to a byte[] my problems would be solved....
Most of the time I love managed code... but sometimes I really hate it ;-)

T

"John Wood" wrote:

well an array of ulongs is just a loop writing out a ulong :)

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:6D**********************************@microsof t.com...

Hi John,

the reason I didn't use the Binary writer is that it only supports single

ulong's, not arrays of ulong's (it only supports arrays of type byte or
char).

Working with the threadpool would indeed have been a solution. The

reason why I decided not to go along this path is that the native methods allow
extra optimizations, like for example the one for sequential access.

Tom

"John Wood" wrote:

> Have you thought about putting the writing into a worker thread

(create a

> delegate and use the threadpool is easiest) ?
>
> You can then use BinaryWriter to write out your ulongs, and it won't
> interefere with the main thread.
>
> --
> John Wood
> EMail: first name, dot, last name, at priorganize.com
> "to*@vandeplas.com" <to*************@discussions.microsoft.com>

wrote in > message news:B6**********************************@microsof t.com...
> > I'm trying to implement a class that writes blocks of data

asynchronously

> to disk. Each block is an array of ulongs. I tried to use the standard > beginwrite functions, but apparently they only support byte arrays. Due to

> the nature of my problem (huge amounts of data) it would be very cpu
> unfriendly to convert all the ulongs to bytes.
> >
> > Therefore I switch to the native methods( WriteFileEx) but I got

in > trouble dealing with the overlapped structure and the callback routines. > Does anybody know if there's some good c# example on how to deal

with these.

> >
> > Thanks for helping me out !
> >
> > Tom
>
>
>

[Ken Kolda]

Check out Buffer.BlockCopy()

Ken
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:5F**********************************@microsof t.com...

You know... if you convert the ulong[] to a byte[] (ulong by ulong) you end up with somethign that is exactly the same as the original... at least
if you look how it's respresented in the memory... so if I just could assign
the address of the ulong[] to a byte[] my problems would be solved....
Most of the time I love managed code... but sometimes I really hate it ;-)

T

"John Wood" wrote:

well an array of ulongs is just a loop writing out a ulong :)

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:6D**********************************@microsof t.com...

Hi John,

the reason I didn't use the Binary writer is that it only supports single

ulong's, not arrays of ulong's (it only supports arrays of type byte or
char).

Working with the threadpool would indeed have been a solution. The

reason why I decided not to go along this path is that the native methods allow
extra optimizations, like for example the one for sequential access.

Tom

"John Wood" wrote:

> Have you thought about putting the writing into a worker thread

(create a

> delegate and use the threadpool is easiest) ?
>
> You can then use BinaryWriter to write out your ulongs, and it won't
> interefere with the main thread.
>
> --
> John Wood
> EMail: first name, dot, last name, at priorganize.com
> "to*@vandeplas.com" <to*************@discussions.microsoft.com>

wrote in > message news:B6**********************************@microsof t.com...
> > I'm trying to implement a class that writes blocks of data

asynchronously

> to disk. Each block is an array of ulongs. I tried to use the standard > beginwrite functions, but apparently they only support byte arrays. Due to

> the nature of my problem (huge amounts of data) it would be very cpu
> unfriendly to convert all the ulongs to bytes.
> >
> > Therefore I switch to the native methods( WriteFileEx) but I got

in > trouble dealing with the overlapped structure and the callback routines. > Does anybody know if there's some good c# example on how to deal

with these.

> >
> > Thanks for helping me out !
> >
> > Tom
>
>
>

[tom]

Hi,

thanks for sending me this, probably I shouldn't say it either but this is a GREAT tip. It will most certainly end up in my hall off fame of best .net tips. Having a background in embedded C, you can not imagine how many times I missed the "misuse the memory" feature in .NET ;-)

Thanks, Tom

"Daniel Jin" wrote:

this advice is exactly what you shouldn't do, but I'm gonna tell you anyway. :)

this is a trick I discovered, you can use the strucklayout attribute to gain access to ulong array through byte array.

[StructLayout( LayoutKind.Explicit )]
public class OverlappedArray
{
[FieldOffset( 0 )] public byte[] ByteView;
[FieldOffset( 0 )] public ulong[] ULongView;
}

then you can do something like this

overlappedArray.ByteView = new byte[64];

for( int i = 0; i < OverlappedArray.ByteView.Length / 8; i++ )
overlappedArray.ULongView[i] = ulong.MaxValue;

then pass ByteView to anything that's expecting a byte array.

remember, be careful with the bound check. it will always to the size of the array you first initialized, so if you look at the Length through the ULongView, you'd still see 64, and you'd be corrupting memory. :) use this at your own risk.

"to*@vandeplas.com" wrote:

You know... if you convert the ulong[] to a byte[] (ulong by ulong) you end up with somethign that is exactly the same as the original... at least if you look how it's respresented in the memory... so if I just could assign the address of the ulong[] to a byte[] my problems would be solved....

Most of the time I love managed code... but sometimes I really hate it ;-)

T

"John Wood" wrote:

well an array of ulongs is just a loop writing out a ulong :)

--
John Wood
EMail: first name, dot, last name, at priorganize.com
"to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
message news:6D**********************************@microsof t.com...
> Hi John,
>
> the reason I didn't use the Binary writer is that it only supports single
ulong's, not arrays of ulong's (it only supports arrays of type byte or
char).
>
> Working with the threadpool would indeed have been a solution. The reason
why I decided not to go along this path is that the native methods allow
extra optimizations, like for example the one for sequential access.
>
> Tom
>
> "John Wood" wrote:
>
> > Have you thought about putting the writing into a worker thread (create
a
> > delegate and use the threadpool is easiest) ?
> >
> > You can then use BinaryWriter to write out your ulongs, and it won't
> > interefere with the main thread.
> >
> > --
> > John Wood
> > EMail: first name, dot, last name, at priorganize.com
> > "to*@vandeplas.com" <to*************@discussions.microsoft.com> wrote in
> > message news:B6**********************************@microsof t.com...
> > > I'm trying to implement a class that writes blocks of data
asynchronously
> > to disk. Each block is an array of ulongs. I tried to use the standard
> > beginwrite functions, but apparently they only support byte arrays. Due
to
> > the nature of my problem (huge amounts of data) it would be very cpu
> > unfriendly to convert all the ulongs to bytes.
> > >
> > > Therefore I switch to the native methods( WriteFileEx) but I got in
> > trouble dealing with the overlapped structure and the callback routines.
> > Does anybody know if there's some good c# example on how to deal with
these.
> > >
> > > Thanks for helping me out !
> > >
> > > Tom
> >
> >
> >