Question How Do You Trace Locked Appserver Agents?

[MissBee]

Of course we usually don't need any quality checking tools, but for some reason I could use one now (it never goes same way as in the movies...)

So far I've just used the command "kill -USR1 PID" but it looks like we could utilize something more developed now.

So what kind of tools you use when trying to catch which program is locking appserver agents? I'd love to prevent us building one new internal tool again, so any idea is more than welcome

 

[TomBascom]

Perhaps you could say a bit about why you find the stack trace inadequate?

 

[MissBee]

It's ok to use the stack trace when the lock is ongoing but I guess the idea of our developers is to catch these error issues after the error is gone already. And idea was to gather the appserver logs and analyze them afterwards...

 

[TomBascom]

You could turn on all of the various -logentrytypes and set the -logginglevel to 4

The problem with that is that it will bury you in detail that you only rarely need.

Which is the fundamental problem with trying to capture data about problems that might happen. You don't know what you really need until something actually happens.

 

[Rob Fitzpatrick]

It would be helpful to know your OpenEdge version. Later versions have more troubleshooting options.

 

[MissBee]

It would be helpful to know your OpenEdge version. Later versions have more troubleshooting options.

OE11.5

 

[Rob Fitzpatrick]

A few ways to gain insight into what a client session is doing:

• Start it with -yx; it will write a list of callers and callees and their statistics to the file proc.mon in session:temp-directory.
• Start it with -logentrytypes 4gltrace; it will write messages to the client log for execution of UDFs, internal procedures, persistent procedures, invocations of RUN, PUBLISH, and SUBSCRIBE, etc.
• Enable client database request statement caching in the database for that session, via promon (R&D | 1 | 18 | 1). This causes the client to send its call stack data to the database each time it accesses the database. The data is stored in an array in the _Connect VST, where you can query it with your own code or with a utility like ProTop.
• Capture the client's call stack at a point in time by executing the proGetStack command (it takes the client session's process ID as a parameter) in a proenv window. This will produce a file called protrace.<PID> in session:temp-directory.

 

[MissBee]

Thank you Rob for your help !

 

[cj_brandt]

We have some automation around sending USR1 signals on Linux. It works better for us than using higher levels of logging.
We use lsof to find the pid's current working directory and then send the USR1 signal, grab the protrace file and send the important details in an email or alert.
Works great for the 3am page when the client wants to know what just happened.

We look for app server agents that have been processing for more than X number of minutes using asbman.
We look through the _Lock table for processes that are blocking each other - lots of info available on do's and don'ts for the _Lock table
We look through the activity VST's for processes that have more than X million number of reads.

This info works so well that we started sending the alerts to the application support teams so they don't even page the DBA for most issues anymore. They just check their email.

We have sent USR1 signals since 10.1C, so maybe 4 or 5 years, and we have never had a production process abnormally terminate, so it works great for us.