Memory leaking on Windows Server 2008 x64

[techjohnny]

Our company is running OpenEdge 10.1B with Progress. I've found that after a while, the system's memory is all consumed by the cache and does not release properly, unless we shutdown the database and restart. I've installed Cacheset.exe and DynCache to try and elevate this problem, but it persists.

Any help would be great. This is causing our "App Servers" to malfunction and run out.

Thanks,

--tj

 

[Rob Fitzpatrick]

Our company is running OpenEdge 10.1B with Progress. I've found that after a while, the system's memory is all consumed by the cache

How do you measure this?

and does not release properly, unless we shutdown the database and restart. I've installed Cacheset.exe and DynCache to try and elevate this problem, but it persists.

Any help would be great. This is causing our "App Servers" to malfunction and run out.j

What Progress error messages do you see when the AppServers "malfunction"?

 

[techjohnny]

Our company is running OpenEdge 10.1B with Progress. I've found that after a while, the system's memory is all consumed by the cache and does not release properly, unless we shutdown the database and restart. I've installed Cacheset.exe and DynCache to try and elevate this problem, but it persists.

Any help would be great. This is causing our "App Servers" to malfunction and run out.

Thanks,

--tj

The message is "no more available app servers" and the memory is measured using Sysinternals Process Explorer

 

[TomBascom]

You are probably confusing cause and effect.

More likely clients are connecting to app servers and failing to release them.

What does "asbman" have to say about how many connections there are? Does it start with a small number and always increase until you run out?

 

[techjohnny]

All the App Servers report "Sending" they go haywire when memory is low.

You are probably confusing cause and effect.

More likely clients are connecting to app servers and failing to release them.

What does "asbman" have to say about how many connections there are? Does it start with a small number and always increase until you run out?

he

 

[TomBascom]

That doesn't really address the question...

Also - if there is a memory leak going on it is almost certainly a programming issue. The code probably isn't cleaning up after itself.

 

[Cringer]

I'm with Tom on this. Check your code first. All AppServers should be cleaned up after they are used. Check every call and make sure it is cleaned up. Be particularly aware of calls in loops. Each instance must clean itself up.

 

[Rob Fitzpatrick]

Which performance metric(s) in Process Explorer are you using to determine there is a memory leak?

 

[gareth.roberts]

There was a bug relating to online backups causing a memory leak on Windows 2008, if this is your problem upgrading to the latest version of Progress will solve it.

 

[cj_brandt]

you can see the memory usage for the app server processes using windows perfmon, is it slowly increasing ? What is the Operating Mode for the agents - State-Reset or State-less ?

 

[cj_brandt]

a side note - if you must be on 10.1B - try to get to at least 10.1B03 - SP 03 had some big performance improvements.

 

[techjohnny]

a side note - if you must be on 10.1B - try to get to at least 10.1B03 - SP 03 had some big performance improvements.

Yes, we are running 10.1B SP3 HF33.

I concur it's probably a process that's releasing the memory, very likely a customization. The application offers "Tracing Options" that allow enable tracing, but I'm not sure what to look for with our customization, except a process of elimination.

The Physical Memory: Available decreases, while the Physical Memory: System Cache increases. Eventually, the Physical Memory: Available falls to zero and stays. This is when the App Servers start acting up.

 

[Cringer]

It would be helpful if you answered all the questions people have posed.

 

[techjohnny]

you can see the memory usage for the app server processes using windows perfmon, is it slowly increasing ? What is the Operating Mode for the agents - State-Reset or State-less ?

Our "App Servers" are running "State-less." It's very difficult to determine which "customization" might be causing a memory leak, but it seems almost certain it's a customization. It doesn't happen immediately after the code is executed, but seems very gradual.

 

[cj_brandt]

If the time to start up a new agent isn't a problem, occasionally terminating idle state-less agents should help keep memory leaks to a minimum until the problem code can be identified.

With the correct logging, you should be able to identify what code is running in each app server agent by looking at the app server log and sorting based on the agent pids. Look at the agents that are using the most memory and see what programs they are calling more than the agents with the smaller memory usage. Left over persistent procedures are what normally cause our leaks.

Also the output from promon -> R&D -> 1 -> 18 "Client Database-Request Statement Cache" may help track down the code if you see a bump in memory usage.

 

[RealHeavyDude]

Most of the times I've seen issues with memory leaks on AppServer agents they were coming from bad code. The usual suspect would be handle-based dynamic objects that you create at runtime which are not removed properly when they are not needed any more.

Once I consulted a customer that had to restart his AppServer every 4 hours because of the AppServer agent's memory consuption grew up to the point where the whole machine froze. Their issue were not properly scoped handle-based dynamic objects. The way they used them this objects all ended up in the unnamed widget pool of the session where they remained until the session was shut down ( the AppServer stopped ). All they needed to do was to include one line of code at the top of each of the procedures that were using such dynamic objects ( which is good practice anyway IMHO ):

create widget-pool no-error.

This creates an unnamed widget pool scoped to the procedure into which all handle-based dynamic objects will go that are not explicitely scoped to a named widget pool - which, usage of named widget pools, I have rarely seen in code I had to deal with. It ensures that all such objects created by the procedure are removed when the procedure goes out of scope - is removed.

At least it's worth a try.

Heavy Regards, RealHeavyDude.

 

[TomBascom]

Yes, "create widget-pool" is chicken soup for dynamic programming. It cures lots of ills.

 

[techjohnny]

Yes, "create widget-pool" is chicken soup for dynamic programming. It cures lots of ills.

Okay, thank you for the response. Any of the suspect customizations have been created in .NET , I'm not sure "create" is valid. Here's a sample of code that's part of our application:

Private Sub QuoteDtl_AfterFieldChange(ByVal sender as object, byval args as datacolumnChangeEventArgs) Handles QuoteDtl_Column.ColumnChanging
Select Case args.Column.columnname
Case "PartNum"
Dim val as String = args.ProposedValue
Dim whereClause as String = "PartNum = '" + val + "'"
Dim recSelected as Boolean
Dim dsSearch as Dataset


If (val <> String.Empty) then
dsSearch = Epicor.mfg.ui.formfunctions.SearchFunctions.listLookup(Otrans, "QuoteDtlSearchAdapter", recSelected, false, whereclause)
If (recSelected = false) then
messagebox.show("Part HAS NOT been quoted")
'Throw new uiexception()
Else
messagebox.show("Part was Last Quoted on: " + dsSearch.Tables(0).Rows(dsSearch.Tables(0).Rows.Count -1 )("LastUpdate").ToString())
messagebox.show("Quote Number: " + dsSearch.Tables(0).Rows(dsSearch.Tables(0).Rows.Count - 1)("QuoteNum").ToString())
messagebox.show("For Customer: " + dsSearch.Tables(0).Rows(dsSearch.Tables(0).Rows.Count - 1)("CustomerCustId").ToString())
End IF
End If

End Select
End Sub
End Module

 

[TomBascom]

That is VB client-side code.

You claim to have an issue with the app-servers. App-servers are written in Progress 4GL. I expect that, in this case, the app-server call is: Epicor.mfg.ui.formfunctions.SearchFunctions.listLookup(Otrans, "QuoteDtlSearchAdapter", recSelected, false, whereclause).

Somewhere inside of there is the code that is (probably) doing some kind of dynamic 4gl programming. I would guess a dynamic query related to "QuoteDtlSearchAdapter".