AVM Memory leak problem

[D.Cook]

I have a persistent batch procedure that seems to grow in memory over time, and am having a hard time determining the cause. Generally I've always thought the AVM handled memory pretty well, and after reviewing the code I cannot find any glaring problems:

• Most logic is encapsulated in sub-procedures which have locally defined variables, so should be cleaned up when the sub-procedure completes.
• Temp-tables are maintained and cleaned appropriately.
• Memptrs are used, but I have ensured that they are unallocated (set-size(mptr) = 0) after every operation.
• I've tried changing the startup parameter -Bt (temp-table buffers) but it doesn't seem to have any effect.
• Some persistent procedures are maintained for performing their respective functions, but when the procedure instance is deleted, there is no memory freed up. Then when I run more functions, the memory continues to grow (I thought maybe it would re-use allocated memory)

I'm not experienced with memory management and don't really understand any of the memory metrics on Windows or UNIX..

So my question is, does anyone have any suggestions on where to look in the code, or how to maintain good memory usage? Aside from restarting the AVM periodically?

 

[Stefan]

The main cause of memory leaks is dynamic objects (queries, buffers, temp-tables), if you CREATE an OBJECT, you also must DELETE OBJECT it. If you CREATE it and then CREATE it again without deleting it - you have just leaked one.

All dynamic objects can be queried / monitored, look at:

- session:first-query
- session:first-buffer (from the buffer you can get to the table-handle)
- session:first-dataset

From each of these you can :next-sibling to the next one.

 

[Cringer]

How are you "deleting" the persistent procedures?

 

[Rob Fitzpatrick]

Have you looked at the client statistics startup parameters (-y, -yc, -yd, -yx)? They might give you more meaningful information than looking at OS metrics for your client process.

 

[rzr]

from the OE Debugging Manual:

Dynamic object monitoring:

You can monitor the creation and destruction of dynamic object instances in your application, using Diagnostics -> Monitor Dynamic Objects
This lets you track dynamic object instances your application has created and not yet destroyed. This can help you locate unused object instances that can result in memory leaks.

 

[D.Cook]

Great responses, thanks guys.

All dynamic objects can be queried / monitored, look at:
- session:first-query
- session:first-buffer (from the buffer you can get to the table-handle)
- session:first-dataset
From each of these you can :next-sibling to the next one.

Good idea, I've written a method to check for any orphaned objects, will post soon (EDIT: already done here apparently http://knowledgebase.progress.com/articles/Article/P124514)
And I discovered my memory leak seems to be due to dynamic buffer objects not being deleted. The dynamic query was deleted but it turns out you need to manually delete all the buffers associated with it. Difficult to do when you have a dynamic number of buffers associated with your query, and I think I've found a bug when deleting them. Investigations continue..

How are you "deleting" the persistent procedures?

Using DELETE PROCEDURE. Using the method above I can confirm they've been deleted.

Have you looked at the client statistics startup parameters (-y, -yc, -yd, -yx)?

Thanks haven't yet, looks useful though. will check these out when there's time.

from the OE Debugging Manual:
Diagnostics -> Monitor Dynamic Objects

Cheers I didn't realise there was a Debugger tool! Found it in the start menu, would love to know if there's a user guide or tutorial.

 

[D.Cook]

Also I found this rather extensive article about memory management on the Knowledge Base:
http://knowledgebase.progress.com/articles/Article/P98993

I keep forgetting to search it properly, seems to be pretty good info on there these days.

 

[D.Cook]

Re: AVM Memory leak problem (Properly deleting dynamic queries and dynamic buffers)

The cause of my leak was due to dynamic buffers attached to a dynamic query becoming orphaned and not deleted. Admittedly this is a very uncommon case: normally static buffers would be used, or the session life would be short enough that a few buffer objects floating around would make no real difference.

But just in case this helps someone, I've written a function to properly delete a procedure and any buffers associated with it.
Note that you cannot loop through a query's buffers and delete them because after the first one is deleted, the query is invalidated and buffer references are lost. So you need to collect all the references before deleting them. Thanks to Phillip Malone of Progress for providing a code sample.

/*Deletes a dynamic query object and any dynamic buffers associated with it.*/
function deleteDynamicQuery returns log (wp_query as widget-handle):
   def var wr_buffers as widget-handle extent no-undo.
   def var i as int no-undo.

   extent(wr_buffers) = wp_query:num-buffers. 
   do i = 1 to wp_query:num-buffers: 
      wr_buffers[i] = wp_query:get-buffer-handle(i). 
   end. 

   do i = 1 to extent(wr_buffers): 
      delete object wr_buffers[i]. 
   end. 

   delete object wp_query.
    
end function.

 

[RealHeavyDude]

Just stepping in:

Every dynamic handle based object that you create ends up in a widget pool. Per default it is the unnamed widget pool to the session unless you specify an unnamed widget pool scoped to the procedure. You can do this by simply adding a CREATE WIDGET-POOL to the beginning of each procedure that deals with dynamic handle based objects. That way it is ensured that they are removed when the procedure goes out of scope. Have a look in any window file that is created with the AppBuilder and you will see it if it hasn't been removed manually.

Of course you can create named widget pools too and deliberately work with them to achieve special behavior ...

Heavy Regards, RealHeavyDude.

 

[Cringer]

SO am I right thinking that if you create an unnamed widget pool in a procedure then anything created is scoped to that rather than the session pool, RHD?

 

[RealHeavyDude]

You are right.

Therefore I think it is good practice to include a CREATE WIDGET-POOL statement in any procedure that deals with handle-based dynamic objects (there is an equivalent for classes too).

Doing in so saved somebody because they had to restart their AppServers every 2 hours because of memory leaks. Including that statement into the include file that got included in all their AppServer procedures fixed the issue. They never had to restart their AppServers again.

Nevertheless, I strongly recommend anybody creating such object deliberately taking care of them ...

Heavy Regards, RealHeavyDude.

 

[Cringer]

Agreed it's best to take care of what you create, but that's really interesting to know. Thanks RHD.

 

[RealHeavyDude]

You're welcome. AFAIK, widget pool is another of this "stealth features" which are not widely known. Anybody ever wondered about the option "[ IN WIDGET-POOL widget-pool-name ]" that is available to the corresponding CREATE statements and what you can do with it?

Heavy Regards, RealHeavyDude.

 

[Cringer]

I've used named widget pools for dynamically created screen objects to save keeping track of all the handles. But that's it.

 

[D.Cook]

I've never quite gotten around to checking out widget-pools and exactly what they do, thanks for explaining RHD. This makes a lot of sense, I'll be creating an unnamed widget-pool for each relevant procedure now, just in case any future code contains a leak..

 

[RealHeavyDude]

One just should be aware that putting your dynamic handle based objects into a widget pool ( be it persistent, meaning scoped to the session - or just scoped to a block like a procedure ) will not prevent you from coding memory leaks. Very much so when you create them in a loop inside the widget pools' scope ...

Using an unnamed widget pool that is scoped to the procedure is a perfect solution for non-persistent procedure calls on the AppServer - but it might not solve your issue when it is scoped to the first procedure in your client session or a persistent procedure ...
Heavy Regards, RealHeavyDude.

 

[D.Cook]

Good point.
So if I was to create an unnamed widget-pool at the start of every non-persistent procedure, I should be pretty safe? I wouldn't think there'd be much added overhead.
Although that could be a problem if the objects need to be returned to the calling procedure (not in my case.. I think).

 

[RealHeavyDude]

Yes, you are save in the sense that every handle based dynamic object that you create will end up in that unnamed widget pool scoped to the procedure and that widget pool will be removed when the procedure ends, so that when the call comes back from the AppServer there are no such objects left in the memory of the AppServer agent.

There is one thing which might bite you were it really hurts: There are occasions where a dynamic handle based object is created implicitly and, although I found no documentation about that, out of my experience these always end up in the unnamed widget pool scoped to the session. For example when in the called procedure you have an output parameter for a static Temp-Table or ProDataSet and in the calling procedure you output a TABLE-HANDLE or DATASET-HANDLE, then the AVM will create these dynamic objects implicitly for you and you need to take care about their scope in your code unless you want to produce a memory leak. In a way that behavior does make sense to me but I would have loved not to have to find out that the rock hard way ...

Heavy Regards, RealHeavyDude.

 

[GregTomkins]

RHD, you are awesome, if this was Stack Overflow or Reddit, you would have, like, a million upvotes.

Just wondering: do you have any experience of the debugger, or other ways of showing the details of the memory used by a session? I don't believe Stefan's answer covers everything, although that is definitely good advice, which we follow. I am not sure how well the debugger would work with a headless AppServer?

 

[Rob Fitzpatrick]

From the Debugging and Toubleshooting manual:

If you want to debug an AppServer™ or WebSpeed® agent, you also must set the
appropriate debuggerEnabled property to 1 in the ubroker.properties file, in addition
to enabling debugging with the proDebugEnable command.

I haven't done this myself, but the above indicates it's possible to debug AppServer.

In terms of tracking memory use, look at the statistics client startup parameters. The LOG-MANAGER and log entry types may provide useful information for you as well (same manual).

Some info on debugging a remote process from the Windows debugger, shared with me by Mike Furgal at BravePoint:

When debugging a process on Windows on the same machine.
- proDebugEnable - this sets a flag in the registry stating that it's ok to use the debugger. This is done for security reasons.
- When you start the debugger, a small shared memory segment is created that holds the network port that the debugger uses to communicate to the process. The prowin32.exe process opens and listens on this port in a thread and the debugger talks to it on this port. The default port is 9999, but the shared memory is used to pass the available port. The process starts at 9999 to see if it's free then moves up or down from there to find a free one.

When debugging a process on a remote machine (Windows or Linux)
- need to run proDebugEnable on Linux to enable debugging
- need to run proDebugConfig on Linux, this creates that shared memory connection to find the port and prints out the port number
- In the Windows debugger you can specify the hostname and port.