Application freezes or slows down

[RP_wpb]

Hello:

**********
OE 10.2B05
Database on Solaris
Clients accessing application over Citrix
**********
Users of our progress application intermittently complain that the application freezes (no response at all) or responds slowly (eg:delayed movement b/n screens, delayed data population on the screens) .There were approx. 1500 - 2000 users using the app .
During the issue time,we observed that there are large number of (around 250+) Active transactions open for more than 5 minutes (through querying vst _trans) which is not a normal scenario in our app. This is one pattern we are observing whenever user reports the issue. After 10 to 15 min all the transactions clear out and also the application response comes back to normal.
So to start with would like to see if there are any pointers to indicate - if because of slowness (may be due to some contention or bottleneck)are the transactions taking more time or because of large number of transactions system is slow.
Are there any data points which we can collect during the issue time to see if there are any contention or bottleneck on the system. Few of the data points we are trying to collect info on Latch / bi growth / resource waits etc.

Thanks in advance for any feedback.

Best Regards,

 

[TomBascom]

That sounds a lot like checkpointing. What is their bi cluster size set to?

For quick and easy monitoring of important metrics grab ProTop During one of these periods of poor performance use the "Blocked Sessions" screen to see why things are performing poorly.

You might also want to consider bringing in an outside performance specialist

 

[Rob Fitzpatrick]

Also, you mentioned 10.2B05 but not which RDBMS license.

Can you do a showcfg and a proutil describe on the DB and provide the output?

 

[TomBascom]

With 1,500 to 2,000 users it must be Enterprise

 

[Rob Fitzpatrick]

Good point. I should know better than to post before my first coffee...

 

[RP_wpb]

Tom:
Cluster size of both DBs (we have 2 DB) are set to 64 MB each.
Below i've pasted the checkpoint history from Promon logged around issue time. Checkpoint '2054' stands out from rest with 'len' column showing 1733 sec.
Actually, I was going to write this earlier but we were reported with the issue again. This time, there were no large open txn and hardly any entries under blocked client session (observed from "Protop") . However checkpoint duration (start and the end) of one of the DB was going as high as 15 - 18 minutes.

05/11/12 Checkpoints
11:30:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
3520 11:27:24 157 0 30229 12189 3912 0 0 1.17 0.53
3519 11:24:00 193 204 31458 29941 3109 0 0 0.94 0.37
3518 11:20:47 184 193 28887 27666 2823 0 0 1.38 0.67
3517 11:17:45 172 182 32497 31612 1998 0 0 1.12 0.57
3516 11:14:29 186 196 32845 32193 2714 0 0 0.87 0.28
3515 11:10:03 253 266 22658 20215 7475 0 0 1.02 0.36
3514 11:05:41 249 262 24922 22055 6006 0 0 0.87 0.32
3513 11:01:24 243 257 21952 18805 7210 0 0 0.85 0.27
OpenEdge Monitor Session End.

05/11/12 Checkpoints
11:30:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
2054 11:27:52 129 0 25398 1797 3031 0 0 1.44 0.61
2053 11:19:13 494 519 29862 21316 13594 0 0 1.02 0.26
2052 11:07:29 673 704 19758 11321 16905 0 0 0.92 0.14
2051 10:55:32 675 717 18900 10038 14099 0 0 1.14 0.21
2050 10:43:37 679 715 25695 12960 16690 0 0 0.99 0.21
2049 10:32:39 625 658 31694 19629 19138 0 0 1.20 0.25
2048 10:23:42 489 537 28144 21518 12852 0 0 1.02 0.23
2047 10:16:27 413 435 29101 23820 9002 0 0 1.28 0.22
OpenEdge Monitor Session End.
======================================================================================================================

05/11/12 Checkpoints
11:45:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
3521 11:36:51 490 0 29303 16941 15175 0 0 0.86 0.23
3520 11:27:24 532 567 30229 24669 16807 0 0 1.17 0.53
3519 11:24:00 193 204 31458 29941 3109 0 0 0.94 0.37
3518 11:20:47 184 193 28887 27666 2823 0 0 1.38 0.67
3517 11:17:45 172 182 32497 31612 1998 0 0 1.12 0.57
3516 11:14:29 186 196 32845 32193 2714 0 0 0.87 0.28
3515 11:10:03 253 266 22658 20215 7475 0 0 1.02 0.36
3514 11:05:41 249 262 24922 22055 6006 0 0 0.87 0.32
OpenEdge Monitor Session End.

05/11/12 Checkpoints
11:45:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
2054 11:27:52 1029 0 25398 1797 24133 0 0 1.44 0.61
2053 11:19:13 494 519 29862 21316 13594 0 0 1.02 0.26
2052 11:07:29 673 704 19758 11321 16905 0 0 0.92 0.14
2051 10:55:32 675 717 18900 10038 14099 0 0 1.14 0.21
2050 10:43:37 679 715 25695 12960 16690 0 0 0.99 0.21
2049 10:32:39 625 658 31694 19629 19138 0 0 1.20 0.25
2048 10:23:42 489 537 28144 21518 12852 0 0 1.02 0.23
2047 10:16:27 413 435 29101 23820 9002 0 0 1.28 0.22
OpenEdge Monitor Session End.
======================================================================================================================

05/11/12 Checkpoints
12:00:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
3523 11:54:54 307 0 28249 23916 8823 0 0 0.93 0.29
3522 11:47:09 442 465 27155 21919 13415 0 0 0.92 0.33
3521 11:36:51 587 618 29303 21927 18645 0 0 0.86 0.23
3520 11:27:24 532 567 30229 24669 16807 0 0 1.17 0.53
3519 11:24:00 193 204 31458 29941 3109 0 0 0.94 0.37
3518 11:20:47 184 193 28887 27666 2823 0 0 1.38 0.67
3517 11:17:45 172 182 32497 31612 1998 0 0 1.12 0.57
3516 11:14:29 186 196 32845 32193 2714 0 0 0.87 0.28
OpenEdge Monitor Session End.

05/11/12 Checkpoints
12:00:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
2055 11:57:39 142 0 23995 1924 3474 0 0 1.01 0.18
2054 11:27:52 1733 1787 25398 7693 39875 0 0 1.44 0.61
2053 11:19:13 494 519 29862 21316 13594 0 0 1.02 0.26
2052 11:07:29 673 704 19758 11321 16905 0 0 0.92 0.14
2051 10:55:32 675 717 18900 10038 14099 0 0 1.14 0.21
2050 10:43:37 679 715 25695 12960 16690 0 0 0.99 0.21
2049 10:32:39 625 658 31694 19629 19138 0 0 1.20 0.25
2048 10:23:42 489 537 28144 21518 12852 0 0 1.02 0.23
OpenEdge Monitor Session End.
======================================================================================================================

05/11/12 Checkpoints
12:15:00
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
3527 12:14:44 16 0 32286 3146 430 0 0 1.19 0.49
3526 12:09:55 276 289 30974 29020 5735 0 0 1.70 0.99
3525 12:05:28 256 267 20945 19625 5731 0 0 1.59 1.04
3524 12:00:24 289 304 28542 26849 6165 0 0 1.24 0.54
3523 11:54:54 316 330 28249 24837 9718 0 0 0.93 0.29
3522 11:47:09 442 465 27155 21919 13415 0 0 0.92 0.33
3521 11:36:51 587 618 29303 21927 18645 0 0 0.86 0.23
3520 11:27:24 532 567 30229 24669 16807 0 0 1.17 0.53
OpenEdge Monitor Session End.

05/11/12 Checkpoints
12:15:00
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
2056 12:13:12 108 0 26003 1377 2957 0 0 0.98 0.23
2055 11:57:39 875 933 23995 14408 20061 0 0 1.01 0.18
2054 11:27:52 1733 1787 25398 7693 39875 0 0 1.44 0.61
2053 11:19:13 494 519 29862 21316 13594 0 0 1.02 0.26
2052 11:07:29 673 704 19758 11321 16905 0 0 0.92 0.14
2051 10:55:32 675 717 18900 10038 14099 0 0 1.14 0.21
2050 10:43:37 679 715 25695 12960 16690 0 0 0.99 0.21
2049 10:32:39 625 658 31694 19629 19138 0 0 1.20 0.25
OpenEdge Monitor Session End.
======================================================================================================================

05/11/12 Checkpoints
12:30:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
3531 12:29:11 50 0 29224 7776 846 0 0 0.93 0.37
3530 12:25:43 198 208 28118 27111 3085 0 0 0.90 0.31
3529 12:22:02 211 221 30480 29218 3480 0 0 1.06 0.46
3528 12:18:33 199 209 38297 37243 2377 0 0 1.02 0.46
3527 12:14:44 219 229 32286 30887 3921 0 0 1.19 0.49
3526 12:09:55 276 289 30974 29020 5735 0 0 1.70 0.99
3525 12:05:28 256 267 20945 19625 5731 0 0 1.59 1.04
3524 12:00:24 289 304 28542 26849 6165 0 0 1.24 0.54
OpenEdge Monitor Session End.

05/11/12 Checkpoints
12:30:01
Ckpt ------ Database Writes ------
No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time
2057 12:26:04 237 0 29166 4774 7349 0 0 1.17 0.30
2056 12:13:12 737 772 26003 12316 24250 0 0 0.98 0.23
2055 11:57:39 875 933 23995 14408 20061 0 0 1.01 0.18
2054 11:27:52 1733 1787 25398 7693 39875 0 0 1.44 0.61
2053 11:19:13 494 519 29862 21316 13594 0 0 1.02 0.26
2052 11:07:29 673 704 19758 11321 16905 0 0 0.92 0.14
2051 10:55:32 675 717 18900 10038 14099 0 0 1.14 0.21
2050 10:43:37 679 715 25695 12960 16690 0 0 0.99 0.21
OpenEdge Monitor Session End.
======================================================================================================================

 

[Rob Fitzpatrick]

This information is very difficult to read when you don't enclose it in CODE tags.

 

[TomBascom]

Longer checkpoint lengths are fine. Too short a length would be more worrisome.

When you did see blocked sessions what were they blocking on?

Do you have significant latch waits?

How many records per second are you reading?

 

[RP_wpb]

Sorry about that. I'll use it next time if i have to paste something like this.

 

[RP_wpb]

During blocked session, whatever we saw it was 'Rec' wait. They didnt last the refresh interval of 2 sec.
latch waits - Unfortunately we didnt monitor or record the latch wait (I am assuming this is the latch activity from 'Protop' and not 'Semaphore Latch waits' as in Promon)
Until this point starting from DB re-start on previous weekend, record read per sec is '33K' and '39k' respectively for each DB.

 

[TomBascom]

That sounds like the long term average record read rate from PROMON. It would be interesting to know what the interval rate is during a slowdown.

It would also be interesting to know what your startup parameters are. The simplest and most reliable way to share them is to extract the 70 or so lines of the .lg file from right after your most recent restart.

Client startup would also be interesting -- that's application dependent though. It might be command line params or a .pf file.

 

[RP_wpb]

Rob:
just so that I dont mess up again. Should I use code tag as below
Code:
paste content here
.........
.......
code:

 

[LarryD]

It's

[ code ]
paste content here
[ / code ]

removing the spaces in the code and / code tags.


It's not "messing up"... it's more to make it both easier to read as well as scrollable within the post. It especially is of use when posting actual source code or longish pastes.

 

[RP_wpb]

DB startup param and client start up below

[2012/05/20@04:45:28.039-0400] P-20325      T-1     I BROKER  0: (333)   Multi-user session begin.
[2012/05/20@04:45:28.064-0400] P-20325      T-1     I BROKER  0: (15321) Before Image Log Initialization at block 0  offset 0.
[2012/05/20@04:45:53.683-0400] P-20325      T-1     I BROKER  0: (452)   Login by progress on batch.
[2012/05/20@04:45:53.702-0400] P-20325      T-1     I BROKER  0: (5644)  Started for xyz_prod_odbc using tcp IPV4 address 0.0.0.0, pid 20325.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4234)  Progress OpenEdge Release 10.2B build 1519 SP05 on SunOS nosxprrsnp99 5.10 Generic_144488-17 sun4u.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4281)  Server started by progress on batch.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (6574)  Started using pid: 20325.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4235)  Physical Database Name (-db): /emc/xyz/prod/xyz/data/xyz.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4236)  Database Type (-dt): PROGRESS.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4237)  Force Access (-F): Not Enabled.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4238)  Direct I/O (-directio): Not Enabled.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (-----) LRU mechanism enabled.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4239)  Number of Database Buffers (-B): 2359296.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (-----) Number of Alternate Database Buffers (-B2): 262144.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (9422)  Maximum private buffers per user (-Bpmax): 64.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4240)  Excess Shared Memory Size (-Mxs): 1626.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (10013) The shared memory segment is locked in memory.
[2012/05/20@04:45:54.708-0400] P-20325      T-1     I BROKER  0: (4241)  Current Size of Lock Table (-L): 200000.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (13953) Maximum Area Number (-maxArea): 200.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4242)  Hash Table Entries (-hash): 740951.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4243)  Current Spin Lock Tries (-spin): 60000.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (6526)  Number of Semaphore Sets (-semsets): 60.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (13924) Maximum Shared Memory Segment Size (-shmsegsize) 32768 Mb.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4244)  Crash Recovery (-i): Enabled.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (6573)  Database Blocksize (-blocksize): 8192.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4245)  Delay of Before-Image Flush (-Mf): 3.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4247)  Before-Image File I/O (-r -R): Reliable.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4249)  Before-Image Truncate Interval (-G): 0.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4250)  Before-Image Cluster Size: 67108864.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4251)  Before-Image Block Size: 16384.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4252)  Number of Before-Image Buffers (-bibufs): 150.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (-----) Record free chain search depth factor 5 (-recspacesearchdepth)
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (9238)  BI File Threshold size (-bithold): 0.0   Bytes.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (6551)  BI File Threshold Stall (-bistall): Enabled.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4254)  After-Image Stall (-aistall): Enabled.
[2012/05/20@04:45:54.717-0400] P-20325      T-1     I BROKER  0: (4255)  After-Image Block Size: 16384.
[2012/05/20@04:45:54.718-0400] P-20325      T-1     I BROKER  0: (4256)  Number of After-Image Buffers (-aibufs): 150.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (8527)  Storage object cache size (-omsize): 2048
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4257)  Maximum Number of Clients Per Server (-Ma): 6.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4258)  Maximum Number of Servers (-Mn): 521.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4259)  Minimum Clients Per Server (-Mi): 3.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4260)  Maximum Number of Users (-n): 3601.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4261)  Host Name (-H): nosxprrsnp99.
[2012/05/20@04:45:54.719-0400] P-20325      T-1     I BROKER  0: (4262)  Service Name (-S): xyz_prod_odbc.
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (14268) TCP/IP Version (-ipver) : IPV4
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (4263)  Network Type (-N): tcp.
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (4264)  Character Set (-cpinternal): ISO8859-1.
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (4282)  Parameter File: /emc/xyz/prod/xyz/data/xyz.pf.
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (5648)  Minimum Port for Auto Servers (-minport): 2000.
[2012/05/20@04:45:54.722-0400] P-20325      T-1     I BROKER  0: (5649)  Maximum Port for Auto Servers (-maxport): 20000.
[2012/05/20@04:45:54.727-0400] P-20325      T-1     I BROKER  0: (10357) Pending client connection timeout (-PendConnTimeout): 10.
[2012/05/20@04:45:54.727-0400] P-20325      T-1     I BROKER  0: (8865)  This broker supports both 4GL and SQL server groups.
[2012/05/20@04:45:54.727-0400] P-20325      T-1     I BROKER  0: (9426)  Large database file access has been enabled.
[2012/05/20@04:45:54.727-0400] P-20325      T-1     I BROKER  0: (9336)  Created shared memory with segment_id: 27
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12813) Allowed index cursors (-c): 14404.
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12814) Group delay (-groupdelay): 10.
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12815) Lock table hash table size (-lkhash): 25621
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12816) Maxport (-maxport): 20000
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12817) Minport (-minport): 2000
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12818) Message Buffer Size (-Mm): 8192
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12821) Use muxlatches (-mux): 1
[2012/05/20@04:45:54.729-0400] P-20325      T-1     I BROKER  0: (12823) Semaphore Sets (-semsets): 60
[2012/05/20@04:45:54.730-0400] P-20325      T-1     I BROKER  0: (13870) Database Service Manager - IPC Queue Size (-pica) : 64.0  KBytes.
[2012/05/20@04:45:54.730-0400] P-20325      T-1     I BROKER  0: (13896) TXE Commit lock skip limit (-TXESkipLimit): 10000.
[2012/05/20@04:45:54.733-0400] P-20325      T-1     I BROKER  0: (15219) Encryption enabled: 0
[2012/05/20@04:45:54.733-0400] P-20325      T-1     I BROKER  0: (15218) Encryption cache size (-ecsize): 1000
[2012/05/20@04:45:54.738-0400] P-20325      T-1     I BROKER  0: (10471) Database connections have been enabled.
********************************************************************************************************************************************
[2012/05/20@04:45:48.330-0400] P-20368      T-1     I BROKER  0: (333)   Multi-user session begin.
[2012/05/20@04:45:48.341-0400] P-20368      T-1     I BROKER  0: (15321) Before Image Log Initialization at block 0  offset 0.
[2012/05/20@04:46:14.772-0400] P-20368      T-1     I BROKER  0: (452)   Login by progress on batch.
[2012/05/20@04:46:14.774-0400] P-20368      T-1     I BROKER  0: (5644)  Started for abc_prod_odbc using tcp IPV4 address 0.0.0.0, pid 20368.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4234)  Progress OpenEdge Release 10.2B build 1519 SP05 on SunOS nosxprrsnp99 5.10 Generic_144488-17 sun4u.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4281)  Server started by progress on batch.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (6574)  Started using pid: 20368.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4235)  Physical Database Name (-db): /emc/xyz/prod/abc/data/abc.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4236)  Database Type (-dt): PROGRESS.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4237)  Force Access (-F): Not Enabled.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4238)  Direct I/O (-directio): Not Enabled.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (-----) LRU mechanism enabled.
[2012/05/20@04:46:15.774-0400] P-20368      T-1     I BROKER  0: (4239)  Number of Database Buffers (-B): 2359296.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (-----) Number of Alternate Database Buffers (-B2): 786432.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (9422)  Maximum private buffers per user (-Bpmax): 64.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4240)  Excess Shared Memory Size (-Mxs): 1626.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (10013) The shared memory segment is locked in memory.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4241)  Current Size of Lock Table (-L): 200000.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (13953) Maximum Area Number (-maxArea): 200.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4242)  Hash Table Entries (-hash): 1037347.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4243)  Current Spin Lock Tries (-spin): 100000.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (6526)  Number of Semaphore Sets (-semsets): 60.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (13924) Maximum Shared Memory Segment Size (-shmsegsize) 32768 Mb.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4244)  Crash Recovery (-i): Enabled.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (6573)  Database Blocksize (-blocksize): 8192.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4245)  Delay of Before-Image Flush (-Mf): 3.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4247)  Before-Image File I/O (-r -R): Reliable.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4249)  Before-Image Truncate Interval (-G): 0.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4250)  Before-Image Cluster Size: 67108864.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4251)  Before-Image Block Size: 16384.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4252)  Number of Before-Image Buffers (-bibufs): 150.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (-----) Record free chain search depth factor 5 (-recspacesearchdepth)
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (9238)  BI File Threshold size (-bithold): 0.0   Bytes.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (6551)  BI File Threshold Stall (-bistall): Enabled.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4254)  After-Image Stall (-aistall): Enabled.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4255)  After-Image Block Size: 16384.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4256)  Number of After-Image Buffers (-aibufs): 150.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (8527)  Storage object cache size (-omsize): 2048
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4257)  Maximum Number of Clients Per Server (-Ma): 6.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4258)  Maximum Number of Servers (-Mn): 521.
[2012/05/20@04:46:15.775-0400] P-20368      T-1     I BROKER  0: (4259)  Minimum Clients Per Server (-Mi): 3.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4260)  Maximum Number of Users (-n): 3601.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4261)  Host Name (-H): nosxprrsnp99.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4262)  Service Name (-S): abc_prod_odbc.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (14268) TCP/IP Version (-ipver) : IPV4
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4263)  Network Type (-N): tcp.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4264)  Character Set (-cpinternal): ISO8859-1.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (4282)  Parameter File: /emc/xyz/prod/abc/data/abc.pf.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (5648)  Minimum Port for Auto Servers (-minport): 2000.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (5649)  Maximum Port for Auto Servers (-maxport): 20000.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (10357) Pending client connection timeout (-PendConnTimeout): 10.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (8865)  This broker supports both 4GL and SQL server groups.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (9426)  Large database file access has been enabled.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (9336)  Created shared memory with segment_id: 28
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12813) Allowed index cursors (-c): 14404.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12814) Group delay (-groupdelay): 10.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12815) Lock table hash table size (-lkhash): 25621
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12816) Maxport (-maxport): 20000
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12817) Minport (-minport): 2000
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12818) Message Buffer Size (-Mm): 8192
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12821) Use muxlatches (-mux): 1
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (12823) Semaphore Sets (-semsets): 60
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (13870) Database Service Manager - IPC Queue Size (-pica) : 64.0  KBytes.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (13896) TXE Commit lock skip limit (-TXESkipLimit): 10000.
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (15219) Encryption enabled: 0
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (15218) Encryption cache size (-ecsize): 1000
[2012/05/20@04:46:15.776-0400] P-20368      T-1     I BROKER  0: (10471) Database connections have been enabled.

Client start up parameter:

-v6colon        
-h 10           
-D 500          
-yy 1960        
-nb 255         
-s 128          
-TB 31          
-TM 32          
-T c:\temp      
-p start.p          
-q              
-inp 10000      
-Mm 8192        
-mmax 20000     
-Bt 2000        
-tmpbsize 8
-cpinternal 1252
-filterocxevents

 

[TomBascom]

That's an interesting -B value. Where did it come from?

I don't think you're doing yourself any favors with:

-Mi 3 -- use -Mi 1 and spread them out. Of course you have your broker type set to "both" so if you're actually mixing 4L *and* SQL access that might be problematic. But the solution to that is to setup a secondary broker. Not to continue with a bad idea.

-spin 60000 or 100000 -- both are probably too high.

10.2B05 is pretty new and that's good. But 10.2B06 is even better. -lruskips and the various -prefecth parameters may very well apply.

Are you using OE Replication?

How hard have you looked at your network as a possible source of performance problems? You have an awful lot of c/s connections -- the network is a prime candidate for slowdowns...

What do you have for a disk subsystem?

 

[RP_wpb]

-B was set up to have a large buffer pool. It was not arrived based on any specific process.

We will definitely review the Mi/Ma/Mn parameters. We dont require SQL access at this point of time.So we will start with only 4GL brokers.

-spin 60000 or 100000 -- both are probably too high.
This number was arrived based on 'trial and error' method on our nightly jobs. Any number below or higher than these numbers degraded the processing of our nightly process.
10.2B05 is pretty new and that's good. But 10.2B06 is even better. -lruskips and the various -prefecth parameters may very well apply.
we have applied 06 to our test environment. Hopefully we will move it to production sooner.

Are you using OE Replication?
We used it for a short period but later discontinued.

How hard have you looked at your network as a possible source of performance problems? You have an awful lot of c/s connections -- the network is a prime candidate for slowdowns...
Honestly, we have not looked at this. oflate we have started to collect data so that we understand the trend.

What do you have for a disk subsystem?
-Database on EMC SAN - RAID 10 and application on NAS.