performance of a batch program

[vinod_home]

Hi,

We have a performance problem on a batch job.

The batch job runs in multiple streams (12 streams), all kicked off from a ksh script. They all process similar kind of data and are the same set of programs. We see out of 12 streams, one or two have really bad performance.

When all the streams seem to be processing 1000 records in 3 minutes, these bad streams process it in 8 minutes. What should I be looking at to figure out whats wrong both on the OS and database side.

The batch programs run on an AIX machine with 20 cpus and have enough memory to process. There are other processes running at the same time, but not so many to load up all the cpus. We use progress 9.1d07 and aix 5.2



Thanks in advance for the help,

 

[TomBascom]

It probably has to do with a mismatch in the distribution of data between the threads. Or possibly with lock contention.

ProTop is your friend:

http://www.greenfieldtech.com/articles/protop.shtml

use the User IO screen to monitor the batch jobs -- there will likely be a big difference between DB Access and/or OS Read values for the slower processes.

 

[vinod_home]

Thanks tom.

Its not the lock contention and neither mismatch in distribution. They build the list of records and update the streams they are going to run in a sequence before the process starts.

 

[TomBascom]

What people think they do and what their code actually does are not always the same thing.

Trust but verify.

You have a hard problem. You need hard data about what is really happening to resolve it. That means taking specific measurements of the runtime activities of these programs. You may very well discover that something isn't working the way that it is supposed to work. (Actually something isn't -- that's why you're posting the question in the first place right?)

Even if everything is going as planned at the logical level it may be that there are issues at the "physical" level. Perhaps, for instance, one otherwise innocent set of data just happens to be dramatically more scattered than the others. That would show up in appropriately gathered monitoring data and point you towards a solution.

 

[TomBascom]

One more thing that I almost forgot... you could also enable profiling on the batch jobs and see which lines of code are eating up all the time. There are some tools to help you with that on my web site too.

This will probably work best if you profile *all* of the batch jobs and compare them to see what is different between "good" and "bad".