Help on sppeding up a 15,000 line batch process

[VM]

In my Windows app, I'm running a batch process that's composed of a FOR loop
that'll run 15,000 times (datatable row count), copy cthe data of each
row -3 fields- to a struct, and send the strct to an external method (using
DLLImport). The problem is that the processing speed is very inconsistent-
it may run very FAST during certain points but then it runs VERY slow at
other moments (especially when it's getting to the end of the table: after
12,000 records). I have an idea of the speed because I have a counter being
displayed in the form (with an Application.DoE vents()). I've eliminated the
DoEvents but it's still inconsistent.

This is a new version of an application that was previously made in MS
C/C++. The only differences between my C# app and the previous version
(besides the language it was written in) is that I had to recreate the
struct in C# (which the external method receives as parameter and modifies)
and that I use the DLLImport for the method. Would this be a reason why it's
so slow? Also, the memory usage (in Task Manager) is at 100% when running.

Thanks,
VM

[Gary Hunt]

Can I suggest you hook up a profiler to see where the problem lies...

Chances are it's a Interop problem but difficult to tell without the source
code etc...

There are some very good profilers out there and most have a 30 day trial
:-)

cheers,

g

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@tk2msf tngp13.phx.gbl. ..

In my Windows app, I'm running a batch process that's composed of a FOR loop that'll run 15,000 times (datatable row count), copy cthe data of each
row -3 fields- to a struct, and send the strct to an external method (using DLLImport). The problem is that the processing speed is very inconsistent- it may run very FAST during certain points but then it runs VERY slow at
other moments (especially when it's getting to the end of the table: after
12,000 records). I have an idea of the speed because I have a counter being displayed in the form (with an Application.DoE vents()). I've eliminated the DoEvents but it's still inconsistent.

This is a new version of an application that was previously made in MS
C/C++. The only differences between my C# app and the previous version
(besides the language it was written in) is that I had to recreate the
struct in C# (which the external method receives as parameter and modifies) and that I use the DLLImport for the method. Would this be a reason why it's so slow? Also, the memory usage (in Task Manager) is at 100% when running.

Thanks,
VM

[VM]

Thanks for your reply. Would interops naturally make the system slower or
would it be something with my code? In other words, can the speed be
improved or is it a dot net limitation?

VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#x******** *****@tk2msftng p13.phx.gbl...

Can I suggest you hook up a profiler to see where the problem lies...

Chances are it's a Interop problem but difficult to tell without the source code etc...

There are some very good profilers out there and most have a 30 day trial
:-)

cheers,

g

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@tk2msf tngp13.phx.gbl. ..

In my Windows app, I'm running a batch process that's composed of a FOR

loop

that'll run 15,000 times (datatable row count), copy cthe data of each
row -3 fields- to a struct, and send the strct to an external method

(using

DLLImport). The problem is that the processing speed is very

inconsistent-

it may run very FAST during certain points but then it runs VERY slow at
other moments (especially when it's getting to the end of the table: after 12,000 records). I have an idea of the speed because I have a counter

being

displayed in the form (with an Application.DoE vents()). I've eliminated

the

DoEvents but it's still inconsistent.

This is a new version of an application that was previously made in MS
C/C++. The only differences between my C# app and the previous version
(besides the language it was written in) is that I had to recreate the
struct in C# (which the external method receives as parameter and

modifies)

and that I use the DLLImport for the method. Would this be a reason why

it's

so slow? Also, the memory usage (in Task Manager) is at 100% when running.
Thanks,
VM

[Gary Hunt]

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++
but there are so many other factors involved. In my experience of calls via
Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may
be obvious (or probably won't be!)

cheers,

g
"VM" <vo******@yahoo .com> wrote in message
news:e6******** ******@TK2MSFTN GP09.phx.gbl...

Thanks for your reply. Would interops naturally make the system slower or
would it be something with my code? In other words, can the speed be
improved or is it a dot net limitation?

VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#x******** *****@tk2msftng p13.phx.gbl...

Can I suggest you hook up a profiler to see where the problem lies...

Chances are it's a Interop problem but difficult to tell without the

source

code etc...

There are some very good profilers out there and most have a 30 day trial

:-)

cheers,

g

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@tk2msf tngp13.phx.gbl. ..

In my Windows app, I'm running a batch process that's composed of a FOR

loop

that'll run 15,000 times (datatable row count), copy cthe data of each
row -3 fields- to a struct, and send the strct to an external method

(using

DLLImport). The problem is that the processing speed is very

inconsistent-

it may run very FAST during certain points but then it runs VERY slow

at other moments (especially when it's getting to the end of the table:

after 12,000 records). I have an idea of the speed because I have a counter

being

displayed in the form (with an Application.DoE vents()). I've eliminated the

DoEvents but it's still inconsistent.

This is a new version of an application that was previously made in MS
C/C++. The only differences between my C# app and the previous

version (besides the language it was written in) is that I had to recreate the
struct in C# (which the external method receives as parameter and

modifies)

and that I use the DLLImport for the method. Would this be a reason

why it's

so slow? Also, the memory usage (in Task Manager) is at 100% when

running.
Thanks,
VM

[VM]

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4 _W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(o bject sender, System.EventArg s e)
{
......
for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000
{
Application.DoE vents();
sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing() + "
rows ";
iRecCount++;
}
}

Thanks again,
VM
"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#R******** ******@TK2MSFTN GP12.phx.gbl...

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++ but there are so many other factors involved. In my experience of calls via Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may
be obvious (or probably won't be!)

cheers,

g
"VM" <vo******@yahoo .com> wrote in message
news:e6******** ******@TK2MSFTN GP09.phx.gbl...

Thanks for your reply. Would interops naturally make the system slower or

would it be something with my code? In other words, can the speed be
improved or is it a dot net limitation?

VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#x******** *****@tk2msftng p13.phx.gbl...

Can I suggest you hook up a profiler to see where the problem lies...

Chances are it's a Interop problem but difficult to tell without the source

code etc...

There are some very good profilers out there and most have a 30 day

trial :-)

cheers,

g

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> In my Windows app, I'm running a batch process that's composed of a FOR loop
> that'll run 15,000 times (datatable row count), copy cthe data of each > row -3 fields- to a struct, and send the strct to an external method
(using
> DLLImport). The problem is that the processing speed is very
inconsistent-
> it may run very FAST during certain points but then it runs VERY slow at

> other moments (especially when it's getting to the end of the table:

after

> 12,000 records). I have an idea of the speed because I have a

counter being
> displayed in the form (with an Application.DoE vents()). I've

eliminated the
> DoEvents but it's still inconsistent.
>
> This is a new version of an application that was previously made in MS > C/C++. The only differences between my C# app and the previous version > (besides the language it was written in) is that I had to recreate the > struct in C# (which the external method receives as parameter and
modifies)
> and that I use the DLLImport for the method. Would this be a reason why it's
> so slow? Also, the memory usage (in Task Manager) is at 100% when

running.

>
> Thanks,
> VM
>
>

[Gary Hunt]

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a
reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_Audit List.Rows[i];

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions
of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one - I
got beat by .Net as the code I was up against required __cdecl callbacks via
delegates which aren't supported without an IL tweak using ILDASM... :-(
"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4 _W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(o bject sender, System.EventArg s e)
{
.....
for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000
{
Application.DoE vents();
sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing() + " rows ";
iRecCount++;
}
}

Thanks again,
VM
"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#R******** ******@TK2MSFTN GP12.phx.gbl...

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++

but there are so many other factors involved. In my experience of calls

via

Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it may

be obvious (or probably won't be!)

cheers,

g
"VM" <vo******@yahoo .com> wrote in message
news:e6******** ******@TK2MSFTN GP09.phx.gbl...

Thanks for your reply. Would interops naturally make the system slower or would it be something with my code? In other words, can the speed be
improved or is it a dot net limitation?

VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#x******** *****@tk2msftng p13.phx.gbl...
> Can I suggest you hook up a profiler to see where the problem lies... >
> Chances are it's a Interop problem but difficult to tell without the
source
> code etc...
>
> There are some very good profilers out there and most have a 30 day

trial

> :-)
>
> cheers,
>
> g
>
> "VM" <vo******@yahoo .com> wrote in message
> news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> > In my Windows app, I'm running a batch process that's composed of a FOR

> loop
> > that'll run 15,000 times (datatable row count), copy cthe data of each > > row -3 fields- to a struct, and send the strct to an external

method > (using
> > DLLImport). The problem is that the processing speed is very
> inconsistent-
> > it may run very FAST during certain points but then it runs VERY slow

at

> > other moments (especially when it's getting to the end of the table: after
> > 12,000 records). I have an idea of the speed because I have a

counter > being
> > displayed in the form (with an Application.DoE vents()). I've

eliminated

> the
> > DoEvents but it's still inconsistent.
> >
> > This is a new version of an application that was previously made

in MS > > C/C++. The only differences between my C# app and the previous

version

> > (besides the language it was written in) is that I had to recreate the > > struct in C# (which the external method receives as parameter and
> modifies)
> > and that I use the DLLImport for the method. Would this be a

reason why

> it's
> > so slow? Also, the memory usage (in Task Manager) is at 100% when
running.
> >
> > Thanks,
> > VM
> >
> >
>
>

[VM]

Thanks for all your help. I just fixed all those details you told me about.
About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM
struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
....
[StructLayout(La youtKind.Sequen tial, CharSet=CharSet .Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:eF******** ******@TK2MSFTN GP11.phx.gbl...

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a
reference to the row and use that rather than indexing into the table each
time

e.g.
DataRow row = Datatable_Audit List.Rows[i];

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up
with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by
index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance -
rem out the interop call and see how long it takes - I'm guessing fractions of a second.

Which leads us to believe that it must be the marshalling that is causing
the problem.

Could you post the zip4Parm structure as that's probably where we make the
changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one - I got beat by .Net as the code I was up against required __cdecl callbacks via delegates which aren't supported without an IL tweak using ILDASM... :-(
"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is

the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it could

be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4 _W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(o bject sender, System.EventArg s e)
{
.....
for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000
{
Application.DoE vents();
sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing() + "

rows ";
iRecCount++;
}
}

Thanks again,
VM
"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#R******** ******@TK2MSFTN GP12.phx.gbl...

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw C++

but there are so many other factors involved. In my experience of

calls via

Interop I haven't noticed any particular slow down so the profiler wou

ld certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it

may be obvious (or probably won't be!)

cheers,

g
"VM" <vo******@yahoo .com> wrote in message
news:e6******** ******@TK2MSFTN GP09.phx.gbl...
> Thanks for your reply. Would interops naturally make the system slower or

> would it be something with my code? In other words, can the speed be
> improved or is it a dot net limitation?
>
> VM
>
> "Gary Hunt" <be*******@code quest.co.uk> wrote in message
> news:#x******** *****@tk2msftng p13.phx.gbl...
> > Can I suggest you hook up a profiler to see where the problem lies... > >
> > Chances are it's a Interop problem but difficult to tell without

the > source
> > code etc...
> >
> > There are some very good profilers out there and most have a 30 day trial
> > :-)
> >
> > cheers,
> >
> > g
> >
> > "VM" <vo******@yahoo .com> wrote in message
> > news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> > > In my Windows app, I'm running a batch process that's composed of a FOR
> > loop
> > > that'll run 15,000 times (datatable row count), copy cthe data

of

each

> > > row -3 fields- to a struct, and send the strct to an external method > > (using
> > > DLLImport). The problem is that the processing speed is very
> > inconsistent-
> > > it may run very FAST during certain points but then it runs VERY

slow

at
> > > other moments (especially when it's getting to the end of the table: > after
> > > 12,000 records). I have an idea of the speed because I have a

counter

> > being
> > > displayed in the form (with an Application.DoE vents()). I've
eliminated
> > the
> > > DoEvents but it's still inconsistent.
> > >
> > > This is a new version of an application that was previously made in

MS

> > > C/C++. The only differences between my C# app and the previous
version
> > > (besides the language it was written in) is that I had to

recreate the

> > > struct in C# (which the external method receives as parameter

and > > modifies)
> > > and that I use the DLLImport for the method. Would this be a

reason why
> > it's
> > > so slow? Also, the memory usage (in Task Manager) is at 100% when > running.
> > >
> > > Thanks,
> > > VM
> > >
> > >
> >
> >
>
>

[Gary Hunt]

Did setting ANSI vs Unicode make any differences?

g
"VM" <vo******@yahoo .com> wrote in message
news:OS******** *****@TK2MSFTNG P10.phx.gbl...

Thanks for all your help. I just fixed all those details you told me about. About the struct, here's a section of it. I'm not going to post the whole
thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any
marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the ZIP4_PARM struct and writes it to the struct in other empty fields when it finds is.
It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(La youtKind.Sequen tial, CharSet=CharSet .Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:eF******** ******@TK2MSFTN GP11.phx.gbl...

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a

reference to the row and use that rather than indexing into the table each time

e.g.
DataRow row = Datatable_Audit List.Rows[i];

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you end up with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather than by index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance - rem out the interop call and see how long it takes - I'm guessing fractions

of a second.

Which leads us to believe that it must be the marshalling that is causing the problem.

Could you post the zip4Parm structure as that's probably where we make the changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one - I

got beat by .Net as the code I was up against required __cdecl callbacks via

delegates which aren't supported without an IL tweak using ILDASM... :-(
"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..

The code's really short...and really easy. And this particular run takes over 30 minutes and the speed is very inconsistent. The only long code is the zip4Parm struct (I didn't post it) that contains over 60 fields

and they're all modified during the z4adrinq call. That's why I assumed it could

be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4 _W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(o bject sender, System.EventArg s e)
{
.....
for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000 {
Application.DoE vents();
sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing() +

"

rows ";
iRecCount++;
}
}

Thanks again,
VM
"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#R******** ******@TK2MSFTN GP12.phx.gbl...
> I guess that depends on the Interops in question!
>
> If you think it through then some Interop operations could certainly

be > tuned - for example - if you've got a 100MB of string data and

you're > passing it through Interop then the following would be true:
>
> 1. .Net stores it's string as Unicode
> 2. Calling an ANSI Interop function would require converting those > strings to ANSI to and from the Interop call
> 3. That's got to be slower than calling a UNICODE Interop function >
> I think the example above shows how Interop could be slower than the

raw C++
> but there are so many other factors involved. In my experience of calls via
> Interop I haven't noticed any particular slow down so the profiler wou ld

> certainly identify which bit of code is causing the problem.
>
> It could also be a GC issue - why don't you post the basic code and

it may

> be obvious (or probably won't be!)
>
> cheers,
>
> g
>
>
> "VM" <vo******@yahoo .com> wrote in message
> news:e6******** ******@TK2MSFTN GP09.phx.gbl...
> > Thanks for your reply. Would interops naturally make the system slower or
> > would it be something with my code? In other words, can the speed

be > > improved or is it a dot net limitation?
> >
> > VM
> >
> > "Gary Hunt" <be*******@code quest.co.uk> wrote in message
> > news:#x******** *****@tk2msftng p13.phx.gbl...
> > > Can I suggest you hook up a profiler to see where the problem

lies...

> > >
> > > Chances are it's a Interop problem but difficult to tell without

the > > source
> > > code etc...
> > >
> > > There are some very good profilers out there and most have a 30 day > trial
> > > :-)
> > >
> > > cheers,
> > >
> > > g
> > >
> > > "VM" <vo******@yahoo .com> wrote in message
> > > news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> > > > In my Windows app, I'm running a batch process that's composed of

a

> FOR
> > > loop
> > > > that'll run 15,000 times (datatable row count), copy cthe data

of each
> > > > row -3 fields- to a struct, and send the strct to an external

method

> > > (using
> > > > DLLImport). The problem is that the processing speed is very
> > > inconsistent-
> > > > it may run very FAST during certain points but then it runs VERY slow
> at
> > > > other moments (especially when it's getting to the end of the

table:

> > after
> > > > 12,000 records). I have an idea of the speed because I have a
counter
> > > being
> > > > displayed in the form (with an Application.DoE vents()). I've
> eliminated
> > > the
> > > > DoEvents but it's still inconsistent.
> > > >
> > > > This is a new version of an application that was previously made in

MS
> > > > C/C++. The only differences between my C# app and the

previous > version
> > > > (besides the language it was written in) is that I had to

recreate the
> > > > struct in C# (which the external method receives as parameter and > > > modifies)
> > > > and that I use the DLLImport for the method. Would this be a

reason

> why
> > > it's
> > > > so slow? Also, the memory usage (in Task Manager) is at 100% when > > running.
> > > >
> > > > Thanks,
> > > > VM
> > > >
> > > >
> > >
> > >
> >
> >
>
>

[Michael Giagnocavo [MVP]]

In addition to keep a reference to the row (instead of indexing each time),
you shouldn't do a string based column lookup each time either (["DEL"]).
The datatable can give you the column (or you can just use the index). Then
you can pass the column or index -- that should be faster too.

Also, you don't need to update the rowcount *every* time, do you? I'd
suggest doing this work on another thread, and then every second or so, have
your UI thread grab the completed rowcount and display it.

I think what you need to do is figure out where the slowdown is (obviously
:)). See if you can narrow it down to a particular set of rows. You
mentioned that near the end of the table, it goes slow. What happens if you
run it only on the last 5000 rows -- does it still go slow? A profiler
definately should tell you where the memory is going. If the memory grows,
and then hits a cieling, and things slow down, perhaps memory isn't being
freed somewhere (in unmanaged code perhaps)?

You *could* try a GC.Collect(2) each 5000 rows and see if that has any
effect (it'd only matter if the GC was getting messed up for some reason).

-mike
MVP

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..

The code's really short...and really easy. And this particular run takes
over 30 minutes and the speed is very inconsistent. The only long code is
the zip4Parm struct (I didn't post it) that contains over 60 fields and
they're all modified during the z4adrinq call. That's why I assumed it
could
be the DLL call and the amount of time it takes to fill the struct:

[DllImport("ZIP4 _W32.DLL")]
public static extern int z4adrinq([In,
Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
private void btn_run_Click(o bject sender, System.EventArg s e)
{
.....
for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000
{
Application.DoE vents();
sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
ZIP4_PARM zip4Parm = new ZIP4_PARM();
zip4Parm.iprurb = sFBU;
zip4Parm.iadl1 = sDEL;
zip4Parm.ictyi = sCTY;
int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+
fields are modified
statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing() +
"
rows ";
iRecCount++;
}
}

Thanks again,
VM
"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:#R******** ******@TK2MSFTN GP12.phx.gbl...

I guess that depends on the Interops in question!

If you think it through then some Interop operations could certainly be
tuned - for example - if you've got a 100MB of string data and you're
passing it through Interop then the following would be true:

1. .Net stores it's string as Unicode
2. Calling an ANSI Interop function would require converting those
strings to ANSI to and from the Interop call
3. That's got to be slower than calling a UNICODE Interop function

I think the example above shows how Interop could be slower than the raw

C++

but there are so many other factors involved. In my experience of calls

via

Interop I haven't noticed any particular slow down so the profiler would
certainly identify which bit of code is causing the problem.

It could also be a GC issue - why don't you post the basic code and it
may
be obvious (or probably won't be!)

cheers,

g
"VM" <vo******@yahoo .com> wrote in message
news:e6******** ******@TK2MSFTN GP09.phx.gbl...

> Thanks for your reply. Would interops naturally make the system slower or > would it be something with my code? In other words, can the speed be
> improved or is it a dot net limitation?
>
> VM
>
> "Gary Hunt" <be*******@code quest.co.uk> wrote in message
> news:#x******** *****@tk2msftng p13.phx.gbl...
> > Can I suggest you hook up a profiler to see where the problem lies...
> >
> > Chances are it's a Interop problem but difficult to tell without the
> source
> > code etc...
> >
> > There are some very good profilers out there and most have a 30 day

trial

> > :-)
> >
> > cheers,
> >
> > g
> >
> > "VM" <vo******@yahoo .com> wrote in message
> > news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> > > In my Windows app, I'm running a batch process that's composed of a

FOR

> > loop
> > > that'll run 15,000 times (datatable row count), copy cthe data of each > > > row -3 fields- to a struct, and send the strct to an external
> > > method
> > (using
> > > DLLImport). The problem is that the processing speed is very
> > inconsistent-
> > > it may run very FAST during certain points but then it runs VERY slow

at

> > > other moments (especially when it's getting to the end of the
> > > table:
> after
> > > 12,000 records). I have an idea of the speed because I have a

counter > > being
> > > displayed in the form (with an Application.DoE vents()). I've

eliminated

> > the
> > > DoEvents but it's still inconsistent.
> > >
> > > This is a new version of an application that was previously made in MS > > > C/C++. The only differences between my C# app and the previous

version

> > > (besides the language it was written in) is that I had to recreate the > > > struct in C# (which the external method receives as parameter and
> > modifies)
> > > and that I use the DLLImport for the method. Would this be a reason

why

> > it's
> > > so slow? Also, the memory usage (in Task Manager) is at 100% when
> running.
> > >
> > > Thanks,
> > > VM
> > >
> > >
> >
> >
>
>

[VM]

For 1,000 records, the difference was 66 sec. vs. 130 secs. which is pretty
good. I haven't tried it with the 15,000 records yet. But the old system did
the 15,000 records in just under 4 mins.

VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:en******** ******@TK2MSFTN GP12.phx.gbl...

Did setting ANSI vs Unicode make any differences?

g
"VM" <vo******@yahoo .com> wrote in message
news:OS******** *****@TK2MSFTNG P10.phx.gbl...

Thanks for all your help. I just fixed all those details you told me about.

About the struct, here's a section of it. I'm not going to post the whole

thing because the rest is the same as this. In the C version of the
application, the struct was in a header file so they didn't have to do any marshalling. Also, during the run I noticed that the loop may be in a
particular record for longer (maybe a second or more). Basically, the
extern method finds information based on the data I copied to the

ZIP4_PARM

struct and writes it to the struct in other empty fields when it finds is. It's similar to a database :

namespace ZipMaster
{
...
[StructLayout(La youtKind.Sequen tial, CharSet=CharSet .Ansi)]
public class ZIP4_PARM
{
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=4 )]
public string rsvd0;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl1;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string iadl2;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=51 )]
public string ictyi;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=3 )]
public string istai;
public string county;
public short respn;
public char retcc;
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=12 )]
public string adrkey;
public char auto_zone_ind;
public struct footer
{
public char a;
public char b;
public char c;
public char d;
public char e;
}
[MarshalAs( UnmanagedType.B yValTStr, SizeConst=6)]
public string rsvd3;
}
}

Thanks again,
VM

"Gary Hunt" <be*******@code quest.co.uk> wrote in message
news:eF******** ******@TK2MSFTN GP11.phx.gbl...

OK - a couple of good practices things to point out first

I know the compiler may pick these up but better with than without.

Accessing a row by index is probably not the fastest way so get yourself a reference to the row and use that rather than indexing into the table each time

e.g.
DataRow row = Datatable_Audit List.Rows[i];

Next one up - if you know that the row.Item[] is a string then cast is
rather than using ToString() - ToString has greater overhead so you

end

up with.

sFBU = (string)row["FBU"];

And back to the first one - why not use a foreach(...) loop rather

than

by index? or is there a specific requirement to go through like that?

All of the above are more than likely negligible in terms of performance - rem out the interop call and see how long it takes - I'm guessing fractions

of a second.

Which leads us to believe that it must be the marshalling that is causing the problem.

Could you post the zip4Parm structure as that's probably where we make the changes - as you're passing in a LPStruct then I'm assuming that the
DllImport statement doesn't need to specify Unicode or ANSI.

cheers,

g

PS Spent the weekend doing Interop so lets get to the bottom of this one -

I

got beat by .Net as the code I was up against required __cdecl

callbacks via

delegates which aren't supported without an IL tweak using ILDASM...

:-(

"VM" <vo******@yahoo .com> wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..
> The code's really short...and really easy. And this particular run

takes > over 30 minutes and the speed is very inconsistent. The only long code

is

> the zip4Parm struct (I didn't post it) that contains over 60 fields

and > they're all modified during the z4adrinq call. That's why I assumed it could
> be the DLL call and the amount of time it takes to fill the struct:
>
> [DllImport("ZIP4 _W32.DLL")]
> public static extern int z4adrinq([In,
> Out][MarshalAs(Unman agedType.LPStru ct)] ZIP4_PARM zip4_parm);
> private void btn_run_Click(o bject sender, System.EventArg s e)
> {
> .....
> for (int i=0;i<Datatable _AuditList.Rows .Count;i++) //count = 15,000 > {
> Application.DoE vents();
> sFBU = Datatable_Audit List.Rows[i]["FBU"].ToString();
> sDEL = Datatable_Audit List.Rows[i]["DEL"].ToString();
> sCTY = Datatable_Audit List.Rows[i]["CTY"].ToString();
> sZ4 = Datatable_Audit List.Rows[i]["Z4"].ToString();
> ZIP4_PARM zip4Parm = new ZIP4_PARM();
> zip4Parm.iprurb = sFBU;
> zip4Parm.iadl1 = sDEL;
> zip4Parm.ictyi = sCTY;
> int iError = z4adrinq(zip4Pa rm); //dll call - zip4Parm's 60+ > fields are modified
> statusBar.Panel s[0].Text = "Row Count = " + iRecCount.ToStr ing()

+

"
> rows ";
> iRecCount++;
> }
> }
>
> Thanks again,
> VM
>
>
> "Gary Hunt" <be*******@code quest.co.uk> wrote in message
> news:#R******** ******@TK2MSFTN GP12.phx.gbl...
> > I guess that depends on the Interops in question!
> >
> > If you think it through then some Interop operations could certainly be

> > tuned - for example - if you've got a 100MB of string data and you're > > passing it through Interop then the following would be true:
> >
> > 1. .Net stores it's string as Unicode
> > 2. Calling an ANSI Interop function would require converting those > > strings to ANSI to and from the Interop call
> > 3. That's got to be slower than calling a UNICODE Interop function > >
> > I think the example above shows how Interop could be slower than

the raw

> C++
> > but there are so many other factors involved. In my experience of

calls

> via
> > Interop I haven't noticed any particular slow down so the profiler wou

ld

> > certainly identify which bit of code is causing the problem.
> >
> > It could also be a GC issue - why don't you post the basic code

and it may
> > be obvious (or probably won't be!)
> >
> > cheers,
> >
> > g
> >
> >
> > "VM" <vo******@yahoo .com> wrote in message
> > news:e6******** ******@TK2MSFTN GP09.phx.gbl...
> > > Thanks for your reply. Would interops naturally make the system slower

> or
> > > would it be something with my code? In other words, can the

speed be > > > improved or is it a dot net limitation?
> > >
> > > VM
> > >
> > > "Gary Hunt" <be*******@code quest.co.uk> wrote in message
> > > news:#x******** *****@tk2msftng p13.phx.gbl...
> > > > Can I suggest you hook up a profiler to see where the problem
lies...
> > > >
> > > > Chances are it's a Interop problem but difficult to tell

without the

> > > source
> > > > code etc...
> > > >
> > > > There are some very good profilers out there and most have a

30 day

> > trial
> > > > :-)
> > > >
> > > > cheers,
> > > >
> > > > g
> > > >
> > > > "VM" <vo******@yahoo .com> wrote in message
> > > > news:%2******** ********@tk2msf tngp13.phx.gbl. ..
> > > > > In my Windows app, I'm running a batch process that's

composed of

a
> > FOR
> > > > loop
> > > > > that'll run 15,000 times (datatable row count), copy cthe

data of

> each
> > > > > row -3 fields- to a struct, and send the strct to an

external method
> > > > (using
> > > > > DLLImport). The problem is that the processing speed is very > > > > inconsistent-
> > > > > it may run very FAST during certain points but then it runs

VERY > slow
> > at
> > > > > other moments (especially when it's getting to the end of the table:
> > > after
> > > > > 12,000 records). I have an idea of the speed because I have a > counter
> > > > being
> > > > > displayed in the form (with an Application.DoE vents()). I've
> > eliminated
> > > > the
> > > > > DoEvents but it's still inconsistent.
> > > > >
> > > > > This is a new version of an application that was previously made in
> MS
> > > > > C/C++. The only differences between my C# app and the previous > > version
> > > > > (besides the language it was written in) is that I had to

recreate

> the
> > > > > struct in C# (which the external method receives as

parameter and

> > > > modifies)
> > > > > and that I use the DLLImport for the method. Would this be a
reason
> > why
> > > > it's
> > > > > so slow? Also, the memory usage (in Task Manager) is at 100%

when

> > > running.
> > > > >
> > > > > Thanks,
> > > > > VM
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>