Severe memory problem with V8

[Jon Trickey]

We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running applications,
the db2syscs.exe process started gobbling up memory until it hit about
1.7 gigs (which seems to be the limit from what I've read) then it
started crapping out.. We have shrunk the buffer pools to much lower
that they used to be in 7, but db2syscs.exe still runs out of memory
after running for a few hours. The application and user loads are
exactly the same as before the migration. It's acting like a memory
leak, but it also seems to be related to the number of agents. When
the database first comes up, it uses about 300 megs of memory. After
all the users connect, it uses about 700 megs of memory. The memory
usage continues to climb until it starts to die when it hits about 1.7
gig. What changed in 8.1 to cause this? What can I do to fix it?
I'm getting tired of restarting db2 avery few hours.

Here is what I have tried so far to no avail-

db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
on to too much memory they didn't need any more.

NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
and max_connections because I never had to worry about running out of
memory before, and I figured it was more efficient to keep the extra
agents around

NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
limit the agents, and the memory, but still allow enought connections.
I also read that connection concentrator was supposed to kick in and
help this work, but instead, after about 100 people connected, people
could not connect anymore. No error was returned to the user, the
application would just hang when it tried to connect to the database.

We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
processors and 4 gigs of memory. The problems we are having are not a
system out of memory issue, its a db2 using more memory that it can
handle problem.

I'm out of ideas. We have opened a PMR with IBM support, but they
have not been any help yet.

Regards,
Jon

[Fan Ruo Xin]

"Jon Trickey" <jt****@safeautoNS.com> wrote in message
news:42********************************@4ax.com...

We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running applications,
the db2syscs.exe process started gobbling up memory until it hit about
1.7 gigs (which seems to be the limit from what I've read) then it
started crapping out.. We have shrunk the buffer pools to much lower
that they used to be in 7, but db2syscs.exe still runs out of memory
after running for a few hours. The application and user loads are
exactly the same as before the migration. It's acting like a memory
leak, but it also seems to be related to the number of agents. When
the database first comes up, it uses about 300 megs of memory. After
all the users connect, it uses about 700 megs of memory. The memory
usage continues to climb until it starts to die when it hits about 1.7
gig. What changed in 8.1 to cause this? What can I do to fix it?
I'm getting tired of restarting db2 avery few hours.
If I were you I would use the same mem configuration first. Compare the
memory usage. Then begin to use connection concentrator - concentrator will
change the way db2 use memory (shared or private).
What is the version of the db2 client?

Here is what I have tried so far to no avail-

db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
on to too much memory they didn't need any more.

NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
and max_connections because I never had to worry about running out of
memory before, and I figured it was more efficient to keep the extra
agents around

NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
limit the agents, and the memory, but still allow enought connections.
I also read that connection concentrator was supposed to kick in and
help this work, but instead, after about 100 people connected, people
could not connect anymore. No error was returned to the user, the
application would just hang when it tried to connect to the database.

We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
processors and 4 gigs of memory. The problems we are having are not a
system out of memory issue, its a db2 using more memory that it can
handle problem.

I'm out of ideas. We have opened a PMR with IBM support, but they
have not been any help yet.

Regards,
Jon

[JS]

"Fan Ruo Xin" <fa*****@sbcglobal.net> wrote in message news:<Fp****************@newssvr17.news.prodigy.co m>...

"Jon Trickey" <jt****@safeautoNS.com> wrote in message
news:42********************************@4ax.com...

We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running applications,
the db2syscs.exe process started gobbling up memory until it hit about
1.7 gigs (which seems to be the limit from what I've read) then it
started crapping out.. We have shrunk the buffer pools to much lower
that they used to be in 7, but db2syscs.exe still runs out of memory
after running for a few hours. The application and user loads are
exactly the same as before the migration. It's acting like a memory
leak, but it also seems to be related to the number of agents. When
the database first comes up, it uses about 300 megs of memory. After
all the users connect, it uses about 700 megs of memory. The memory
usage continues to climb until it starts to die when it hits about 1.7
gig. What changed in 8.1 to cause this? What can I do to fix it?
I'm getting tired of restarting db2 avery few hours.

If I were you I would use the same mem configuration first. Compare the
memory usage. Then begin to use connection concentrator - concentrator will
change the way db2 use memory (shared or private).
What is the version of the db2 client?

Here is what I have tried so far to no avail-

db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
on to too much memory they didn't need any more.

NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
and max_connections because I never had to worry about running out of
memory before, and I figured it was more efficient to keep the extra
agents around

NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
limit the agents, and the memory, but still allow enought connections.
I also read that connection concentrator was supposed to kick in and
help this work, but instead, after about 100 people connected, people
could not connect anymore. No error was returned to the user, the
application would just hang when it tried to connect to the database.

We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
processors and 4 gigs of memory. The problems we are having are not a
system out of memory issue, its a db2 using more memory that it can
handle problem.

I'm out of ideas. We have opened a PMR with IBM support, but they
have not been any help yet.

Regards,
Jon

Yes, we have seen the same memory gobbling issues, you can use db2mtrk
to see where all the memory is going. Make sure the connection
concentrator is turned on.

I would try setting MEMMAXFREE even lower, try 4096, this parameter
should shrink the agents memory to the size specified when the agent
is idle. Keep in mind that the preferred behaviour in v8 is for the
database manager to steal agents that are idle, not create new ones.

You may have to increase the addressable memory space for db2 by
modifying the boot.ini file. 1.7 gig is the limit, but it can be
increased to 3 gigs

[Phil Gunning]

If you have connnection concentrator enabled, then more shared memory
is used. Check the value of priv_mem_thresh and change it to reduce
the amount of memory retained per agent. Investigate the use of the
3GB switch which gives DB@ 3GB of virtual memory and reduces Windows
to 1GB on Advanced server. Many clients are using this to get around
the 1.75 GB limitation. I have a client who is on 8.1.5 from 7.2 and
they only had problems cause they changed the value of priv_mem_thresh
to cause more memory to be retained. Other settings at automatic
turned out ok....HTH Phil


fy***@hotmail.com (JS) wrote in message news:<e9**************************@posting.google. com>...

"Fan Ruo Xin" <fa*****@sbcglobal.net> wrote in message news:<Fp****************@newssvr17.news.prodigy.co m>...

"Jon Trickey" <jt****@safeautoNS.com> wrote in message
news:42********************************@4ax.com...

We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running applications,
the db2syscs.exe process started gobbling up memory until it hit about
1.7 gigs (which seems to be the limit from what I've read) then it
started crapping out.. We have shrunk the buffer pools to much lower
that they used to be in 7, but db2syscs.exe still runs out of memory
after running for a few hours. The application and user loads are
exactly the same as before the migration. It's acting like a memory
leak, but it also seems to be related to the number of agents. When
the database first comes up, it uses about 300 megs of memory. After
all the users connect, it uses about 700 megs of memory. The memory
usage continues to climb until it starts to die when it hits about 1.7
gig. What changed in 8.1 to cause this? What can I do to fix it?
I'm getting tired of restarting db2 avery few hours.

If I were you I would use the same mem configuration first. Compare the
memory usage. Then begin to use connection concentrator - concentrator will
change the way db2 use memory (shared or private).
What is the version of the db2 client?

Here is what I have tried so far to no avail-

db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
on to too much memory they didn't need any more.

NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
and max_connections because I never had to worry about running out of
memory before, and I figured it was more efficient to keep the extra
agents around

NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
limit the agents, and the memory, but still allow enought connections.
I also read that connection concentrator was supposed to kick in and
help this work, but instead, after about 100 people connected, people
could not connect anymore. No error was returned to the user, the
application would just hang when it tried to connect to the database.

We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
processors and 4 gigs of memory. The problems we are having are not a
system out of memory issue, its a db2 using more memory that it can
handle problem.

I'm out of ideas. We have opened a PMR with IBM support, but they
have not been any help yet.

Regards,
Jon

Yes, we have seen the same memory gobbling issues, you can use db2mtrk
to see where all the memory is going. Make sure the connection
concentrator is turned on.

I would try setting MEMMAXFREE even lower, try 4096, this parameter
should shrink the agents memory to the size specified when the agent
is idle. Keep in mind that the preferred behaviour in v8 is for the
database manager to steal agents that are idle, not create new ones.

You may have to increase the addressable memory space for db2 by
modifying the boot.ini file. 1.7 gig is the limit, but it can be
increased to 3 gigs

[Jon Trickey]

Thanks for your help. I had forgotten about the /3GB switch. I had
tried it a long time ago in previous versions of DB2 and could not get
it to work, but it's working now. I also lowered the applheapsz to
512 from 1024, and this helped quite a bit. We had set this higher
quite some time ago in a previous version because we were getting
messages in the diag log to increase it. I found a paper from IBM
that talks about the increased memory usage at the agent level and it
says that you should be able to significantly decrease applheapsz
because of the way V8 handles agent memory. I'm trying this set at
256 now on a test server to see if I can decrease it further. I wish
they would post this kind of info in the release notes or upgrade
documents.

About connection concentrator, I am having trouble finding much
documentation on it. All I can find is that setting max_connections >
numagents should enable it. Is there any way to tell if it is on?
Does the application have to do anything special for it? When we
increased max_connections > numagent we could not get very many more
applications than maxagents connected to the database.

Thanks again,
Jon

On 11 Aug 2004 18:59:23 -0700, pg******@gunningts.com (Phil Gunning)
wrote:

If you have connnection concentrator enabled, then more shared memory
is used. Check the value of priv_mem_thresh and change it to reduce
the amount of memory retained per agent. Investigate the use of the
3GB switch which gives DB@ 3GB of virtual memory and reduces Windows
to 1GB on Advanced server. Many clients are using this to get around
the 1.75 GB limitation. I have a client who is on 8.1.5 from 7.2 and
they only had problems cause they changed the value of priv_mem_thresh
to cause more memory to be retained. Other settings at automatic
turned out ok....HTH Phil


fy***@hotmail.com (JS) wrote in message news:<e9**************************@posting.google. com>...

"Fan Ruo Xin" <fa*****@sbcglobal.net> wrote in message news:<Fp****************@newssvr17.news.prodigy.co m>...

> "Jon Trickey" <jt****@safeautoNS.com> wrote in message
> news:42********************************@4ax.com...
> > We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
> > weekend, but we have a light user load then (about 200 users.) Today
> > when we had close to 600 users connecting and running applications,
> > the db2syscs.exe process started gobbling up memory until it hit about
> > 1.7 gigs (which seems to be the limit from what I've read) then it
> > started crapping out.. We have shrunk the buffer pools to much lower
> > that they used to be in 7, but db2syscs.exe still runs out of memory
> > after running for a few hours. The application and user loads are
> > exactly the same as before the migration. It's acting like a memory
> > leak, but it also seems to be related to the number of agents. When
> > the database first comes up, it uses about 300 megs of memory. After
> > all the users connect, it uses about 700 megs of memory. The memory
> > usage continues to climb until it starts to die when it hits about 1.7
> > gig. What changed in 8.1 to cause this? What can I do to fix it?
> > I'm getting tired of restarting db2 avery few hours.
>
> If I were you I would use the same mem configuration first. Compare the
> memory usage. Then begin to use connection concentrator - concentrator will
> change the way db2 use memory (shared or private).
> What is the version of the db2 client?
>
> >
> > Here is what I have tried so far to no avail-
> >
> > db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
> > on to too much memory they didn't need any more.
> >
> > NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
> > and max_connections because I never had to worry about running out of
> > memory before, and I figured it was more efficient to keep the extra
> > agents around
> >
> > NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
> > limit the agents, and the memory, but still allow enought connections.
> > I also read that connection concentrator was supposed to kick in and
> > help this work, but instead, after about 100 people connected, people
> > could not connect anymore. No error was returned to the user, the
> > application would just hang when it tried to connect to the database.
> >
> > We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
> > processors and 4 gigs of memory. The problems we are having are not a
> > system out of memory issue, its a db2 using more memory that it can
> > handle problem.
> >
> > I'm out of ideas. We have opened a PMR with IBM support, but they
> > have not been any help yet.
> >
> > Regards,
> > Jon

Yes, we have seen the same memory gobbling issues, you can use db2mtrk
to see where all the memory is going. Make sure the connection
concentrator is turned on.

I would try setting MEMMAXFREE even lower, try 4096, this parameter
should shrink the agents memory to the size specified when the agent
is idle. Keep in mind that the preferred behaviour in v8 is for the
database manager to steal agents that are idle, not create new ones.

You may have to increase the addressable memory space for db2 by
modifying the boot.ini file. 1.7 gig is the limit, but it can be
increased to 3 gigs

[Mark Yudkin]

You've already done what I was going to recommend:

We have opened a PMR with IBM support, but they have not been any help yet.
I recommend you keep bugging IBM support until they come up with a solution.
There isn't any better course of action for what seems to be a memory leak
(==> bug).

"Jon Trickey" <jt****@safeautoNS.com> wrote in message
news:42********************************@4ax.com... We migrated to 8.1 from 7.2 this weekend. Everything ran ok over the
weekend, but we have a light user load then (about 200 users.) Today
when we had close to 600 users connecting and running applications,
the db2syscs.exe process started gobbling up memory until it hit about
1.7 gigs (which seems to be the limit from what I've read) then it
started crapping out.. We have shrunk the buffer pools to much lower
that they used to be in 7, but db2syscs.exe still runs out of memory
after running for a few hours. The application and user loads are
exactly the same as before the migration. It's acting like a memory
leak, but it also seems to be related to the number of agents. When
the database first comes up, it uses about 300 megs of memory. After
all the users connect, it uses about 700 megs of memory. The memory
usage continues to climb until it starts to die when it hits about 1.7
gig. What changed in 8.1 to cause this? What can I do to fix it?
I'm getting tired of restarting db2 avery few hours.

Here is what I have tried so far to no avail-

db2set DB2MEMMAXFREE=262144 I thought maybe the agents were holding
on to too much memory they didn't need any more.

NUM_POOLAGENTS=100 This used to be set at 2000 along with maxagents
and max_connections because I never had to worry about running out of
memory before, and I figured it was more efficient to keep the extra
agents around

NUMAGENTS=100 , MAX_CONNECTIONS=2000 My hope was that this would
limit the agents, and the memory, but still allow enought connections.
I also read that connection concentrator was supposed to kick in and
help this work, but instead, after about 100 people connected, people
could not connect anymore. No error was returned to the user, the
application would just hang when it tried to connect to the database.

We are running WSE 8.1 fp6 on windows 2003 with 2 xeon 3.2 gig
processors and 4 gigs of memory. The problems we are having are not a
system out of memory issue, its a db2 using more memory that it can
handle problem.

I'm out of ideas. We have opened a PMR with IBM support, but they
have not been any help yet.

Regards,
Jon

[db2user]

test

[db2user]

Think you have INTRA-PARALLEL and /or concentrator enabled

Go to
http://www-106.ibm.com/developerwork...cle/dm-0406qi/
See <Application group shared memory> part and you will understand why db2
needs more memory than before.

Taking DB2 DB CFG default value, DB2 needs 160 Mb (Application group
shared memory) for set of 78 applications. If you have 600 applications
connected then 8 (=600/78) application group shared memory are needed ,
which represents (160 Mb * 8 ) of memory.

Reduce app_ctl_heap_sz and/or appgroup_mem_sz