Broker process for App Server consumes large amount of memory

[cj_brandt]

Red Hat Enterprise Linux Server release 5.1
Progress 10.2B03 - recently upgraded from 10.1C04
java version "1.5.0_11"

2 Linux servers host 25+ state-less app servers, a broker with fixed number of agents 5 - 15 depending on client size.
Was running on 10.1C04 for over a year - upgraded to 10.2B03 4 weeks ago.
No other changes made.

Since OE upgrade, during quiet periods of time with little or no app server traffic, one of the brokers will consume all available memory on the server in less than 5 minutes. This causes the server to become unresponsive and require a restart. This has happened 3 times in 4 weeks, once on one server and twice on another - the servers are basically identical.

Has anyone else seen anything like this ?

java process normally uses around 400mb of resident memory. Once problem occurs memory usage goes to 9gb , then 18gb, then 25gb, then our box is out of memory. That happens in 3 minutes so there isn't time to get a java dump.

 

[atopher1]

Had a similar issue when we upgraded to 10.2B:


http://communities.progress.com/pcom/message/82157


I've left the jvmArgs=-XX:+UseSerialGC flag in with no *noticeable* difference, however I later discovered that our main problem was not tidying up persistent procedures properly within appserver .p's


Chris

http://www.progresstalk.com/showthread.php?119464-Memory-Usage-After-Upgrade-to-10.2B

 

[TomBascom]

That sounds a bit like:

http://www.progresstalk.com/showthread.php?119464-Memory-Usage-After-Upgrade-to-10.2B

Which may have been addressed by:

"jvmargs=-Xmx1024m as the recomendation of the ProgressKB P153007 and there is a memory decrease"

 

[cj_brandt]

The previous article sounds to me like after the upgrade, when the components are brought up the memory usage is much higher on 10.2B. Our memory usage is basically the same on 10.2B as it was on 10.1C - until the java broker goes crazy grabbing memory and within a few minutes drains the box of memory. So 99.5% of the time the memory usage remains the same - its that other .5% that causes much pain.

We do set the -Xmx and -Xms parameters for the admin server.

I don't believe it is an issue with the code the app server agents are running. The agents don't see the large memory increase - just the broker process. We have been fortunate that the outages have happened while little or no activity was occurring with the agents - basically nobody was on the system. In the past though we have fixed some coding issues that caused the app server agents memory to increase.


Thanks for the suggestions so far, I also have a ticket open with PSC Tech Support. I will post the solution when / if we find it.

 

[cj_brandt]

We discovered that Progress has changed some of the code that deals with the app broker between 10.1C and 10.2B. The new code was triggering a bug in the JVM by Sun / Oracle Java. Sun / Oracle fixed the java bug in 1.5.0.16 and we will deploy 1.5.0.22.