Memory Usage in web service???

Hi all,
I have a web service that (for better or worse) returns rather large
xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
but this is a service that's used by few users (less than 10) so I'd prefer
not to rip it apart right now - my question is can someone explain the
following memory usage

first, my code looks like (pseudo coded)
DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
this
XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
MB
return oDoc; // executing this jumps w3wp to over 800 MB!!!


can someone please give me the reason why? Even if it copies the data to
stream it out, I wouldn't expect more than about 400-500MB of memory used???

Thanks
--
Dave Stienessen

re: Memory Usage in web service???

I am fairly certain it has to do with the DataSet being serialized, up until
then it is native code, but once you do that return it has to convert it to
XML.

You might want to look into compressing that some how, it would require
custom serializer-deserializer, but if the calls are going to be that large
it probably would be worth while to investigate that.

Good luck!

Jason Lind

"Dave Stienessen" wrote:
[color=blue]
> Hi all,
> I have a web service that (for better or worse) returns rather large
> xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
> but this is a service that's used by few users (less than 10) so I'd prefer
> not to rip it apart right now - my question is can someone explain the
> following memory usage
>
> first, my code looks like (pseudo coded)
> DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
> this
> XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
> MB
> return oDoc; // executing this jumps w3wp to over 800 MB!!!
>
>
> can someone please give me the reason why? Even if it copies the data to
> stream it out, I wouldn't expect more than about 400-500MB of memory used???
>
> Thanks
> --
> Dave Stienessen
>[/color]

re: Memory Usage in web service???

I am fairly certain it has to do with the DataSet being serialized, up until
then it is native code, but once you do that return it has to convert it to
XML.

You might want to look into compressing that some how, it would require
custom serializer-deserializer, but if the calls are going to be that large
it probably would be worth while to investigate that.

Good luck!

Jason Lind

"Dave Stienessen" wrote:
[color=blue]
> Hi all,
> I have a web service that (for better or worse) returns rather large
> xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
> but this is a service that's used by few users (less than 10) so I'd prefer
> not to rip it apart right now - my question is can someone explain the
> following memory usage
>
> first, my code looks like (pseudo coded)
> DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
> this
> XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
> MB
> return oDoc; // executing this jumps w3wp to over 800 MB!!!
>
>
> can someone please give me the reason why? Even if it copies the data to
> stream it out, I wouldn't expect more than about 400-500MB of memory used???
>
> Thanks
> --
> Dave Stienessen
>[/color]

re: Memory Usage in web service???

I'd agree, except I serialized the XmlDataDocument to a file myself and that
was 150MB - so if the serialized dataset (AKA XML) is 150MB and my initial
hit to do the work to get the dataset into memory was 250MB, thats ~400MB -
where's the other 400MB of RAM going?

--
Dave Stienessen



"Jason L Lind" wrote:
[color=blue]
> I am fairly certain it has to do with the DataSet being serialized, up until
> then it is native code, but once you do that return it has to convert it to
> XML.
>
> You might want to look into compressing that some how, it would require
> custom serializer-deserializer, but if the calls are going to be that large
> it probably would be worth while to investigate that.
>
> Good luck!
>
> Jason Lind
>
> "Dave Stienessen" wrote:
>[color=green]
> > Hi all,
> > I have a web service that (for better or worse) returns rather large
> > xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
> > but this is a service that's used by few users (less than 10) so I'd prefer
> > not to rip it apart right now - my question is can someone explain the
> > following memory usage
> >
> > first, my code looks like (pseudo coded)
> > DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
> > this
> > XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
> > MB
> > return oDoc; // executing this jumps w3wp to over 800 MB!!!
> >
> >
> > can someone please give me the reason why? Even if it copies the data to
> > stream it out, I wouldn't expect more than about 400-500MB of memory used???
> >
> > Thanks
> > --
> > Dave Stienessen
> >[/color][/color]

re: Memory Usage in web service???

I'd agree, except I serialized the XmlDataDocument to a file myself and that
was 150MB - so if the serialized dataset (AKA XML) is 150MB and my initial
hit to do the work to get the dataset into memory was 250MB, thats ~400MB -
where's the other 400MB of RAM going?

--
Dave Stienessen



"Jason L Lind" wrote:
[color=blue]
> I am fairly certain it has to do with the DataSet being serialized, up until
> then it is native code, but once you do that return it has to convert it to
> XML.
>
> You might want to look into compressing that some how, it would require
> custom serializer-deserializer, but if the calls are going to be that large
> it probably would be worth while to investigate that.
>
> Good luck!
>
> Jason Lind
>
> "Dave Stienessen" wrote:
>[color=green]
> > Hi all,
> > I have a web service that (for better or worse) returns rather large
> > xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
> > but this is a service that's used by few users (less than 10) so I'd prefer
> > not to rip it apart right now - my question is can someone explain the
> > following memory usage
> >
> > first, my code looks like (pseudo coded)
> > DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
> > this
> > XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
> > MB
> > return oDoc; // executing this jumps w3wp to over 800 MB!!!
> >
> >
> > can someone please give me the reason why? Even if it copies the data to
> > stream it out, I wouldn't expect more than about 400-500MB of memory used???
> >
> > Thanks
> > --
> > Dave Stienessen
> >[/color][/color]

re: Memory Usage in web service???

If u can divide the returns into 1M per, it will be faster than now.
In my project , the webservice with division returns has a good performace.
[color=blue]
> Hi all,
> I have a web service that (for better or worse) returns rather large
> xmlDataDocuments (up to 150MB) I know - this is huge for a web service call,
> but this is a service that's used by few users (less than 10) so I'd prefer
> not to rip it apart right now - my question is can someone explain the
> following memory usage
>
> first, my code looks like (pseudo coded)
> DataSet ds = getDataSet(); // memory usage of w3wp is at about 250MB after
> this
> XmlDataDocument oDoc = new XmlDataDocument(ds); // memory usage still at 250
> MB
> return oDoc; // executing this jumps w3wp to over 800 MB!!!
>
>
> can someone please give me the reason why? Even if it copies the data to
> stream it out, I wouldn't expect more than about 400-500MB of memory used???
>
> Thanks
> --
> Dave Stienessen
>[/color]