abnormal shutdown of appserver

[rajeev.babu]

The production environment went down because of production appserver not responding. The appserver log file contains the below error: The root cause of this issue is not identified and the production went down 4 times in a month with the same error. Is there any way to identify the cause.

Found that proddb_AS was not running due to the error:

[13/10/08@23:58:26.648-0700] P-028036 T-3149438448 1 AS -- Connection failure for host 127.0.0.1 port 54648 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-008802 T-1091735024 1 AS -- Connection failure for host 127.0.0.1 port 56123 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-007280 T-4247584240 1 AS -- Connection failure for host 127.0.0.1 port 52470 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-024445 T-1125903856 1 AS -- Connection failure for host 127.0.0.1 port 53582 transport TCP. (9407)

[13/10/08@23:58:26.649-0700] P-004763 T-3820318192 1 AS -- (Procedure: 'nowInUTC mfsemstr.p' Line:93) Connection failure for host 127.0.0.1 port 49587 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-011774 T-3422432752 1 AS -- Connection failure for host 127.0.0.1 port 57429 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-014122 T-566033904 1 AS -- Connection failure for host 127.0.0.1 port 63236 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-016873 T-4116999664 1 AS -- Connection failure for host 127.0.0.1 port 57333 transport TCP. (9407)
[13/10/08@23:58:26.648-0700] P-009415 T-1874599408 1 AS -- Connection failure for host 127.0.0.1 port 56996 transport TCP. (9407)
[13/10/08@23:58:26.649-0700] P-013348 T-688512496 1 AS -- Connection failure for host 127.0.0.1 port 50575 transport TCP. (9407)
[13/10/08@23:58:26.649-0700] P-008318 T-3203321328 1 AS -- Connection failure for host 127.0.0.1 port 58067 transport TCP. (9407)
[13/10/08@23:58:26.649-0700] P-004763 T-3820318192 3 AS AS -- TRACE: Non-PERSISTENT Procedure END STOP. (8397)
[13/10/08@23:58:26.649-0700] P-004763 T-3820318192 1 AS -- ** Pipe to subprocess has been broken. (140)

[13/10/08@23:58:26.649-0700] P-004763 T-3820318192 1 AS -- Connection failure for host 127.0.0.1 port 49587 transport TCP. (9407)
[13/10/08@23:58:26.650-0700] P-024993 T-2593816048 1 AS -- Connection failure for host 127.0.0.1 port 53695 transport TCP. (9407)
[13/10/08@23:58:26.650-0700] P-015171 T-3777220080 1 AS -- Connection failure for host 127.0.0.1 port 56997 transport TCP. (9407)
[13/10/08@23:58:26.650-0700] P-011588 T-3185225200 1 AS -- Connection failure for host 127.0.0.1 port 62173 transport TCP. (9407)

[13/10/08@23:58:26.650-0700] P-024445 T-1125903856 2 AS AS Application Server Shutdown. (5476)

 

[Cringer]

What does Line 93 of mfsemstr.p do?

 

[rajeev.babu]

The program is in encrypted mode. I will verify and let you know. BTW the same appserver went down again on the same day

[13/10/09@07:52:35.895-0700] P-022174 T-4083514864 1 AS -- Connection failure for host 127.0.0.1 port 57544 transport TCP. (9407)
[13/10/09@07:52:35.898-0700] P-021042 T-2283425264 1 AS -- Connection failure for host 127.0.0.1 port 53050 transport TCP. (9407)
[13/10/09@07:52:35.899-0700] P-022174 T-4083514864 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.899-0700] P-002085 T-2237140464 1 AS -- Connection failure for host 127.0.0.1 port 61032 transport TCP. (9407)
[13/10/09@07:52:35.900-0700] P-026329 T-4168805872 1 AS -- Connection failure for host 127.0.0.1 port 54341 transport TCP. (9407)
[13/10/09@07:52:35.902-0700] P-021042 T-2283425264 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.904-0700] P-018364 T-2980343280 1 AS -- Connection failure for host 127.0.0.1 port 62261 transport TCP. (9407)
[13/10/09@07:52:35.904-0700] P-002914 T-2251783664 1 AS -- Connection failure for host 127.0.0.1 port 60108 transport TCP. (9407)
[13/10/09@07:52:35.904-0700] P-015185 T-1042386416 1 AS -- Connection failure for host 127.0.0.1 port 50080 transport TCP. (9407)
[13/10/09@07:52:35.904-0700] P-017337 T-3667357168 1 AS -- Connection failure for host 127.0.0.1 port 50312 transport TCP. (9407)
[13/10/09@07:52:35.904-0700] P-026329 T-4168805872 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.904-0700] P-002682 T-2256739824 1 AS -- Connection failure for host 127.0.0.1 port 50431 transport TCP. (9407)
[13/10/09@07:52:35.905-0700] P-005439 T-3417927152 1 AS -- Connection failure for host 127.0.0.1 port 51616 transport TCP. (9407)
[13/10/09@07:52:35.905-0700] P-027200 T-4065775088 1 AS -- Connection failure for host 127.0.0.1 port 53260 transport TCP. (9407)
[13/10/09@07:52:35.906-0700] P-002914 T-2251783664 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.906-0700] P-002085 T-2237140464 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.906-0700] P-017337 T-3667357168 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.907-0700] P-002682 T-2256739824 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.907-0700] P-018364 T-2980343280 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.907-0700] P-017931 T-2260782576 1 AS -- Connection failure for host 127.0.0.1 port 51964 transport TCP. (9407)
[13/10/09@07:52:35.907-0700] P-019782 T-2171715056 1 AS -- Connection failure for host 127.0.0.1 port 61707 transport TCP. (9407)
[13/10/09@07:52:35.907-0700] P-014544 T-55717360 1 AS -- Connection failure for host 127.0.0.1 port 62559 transport TCP. (9407)
[13/10/09@07:52:35.907-0700] P-027200 T-4065775088 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.907-0700] P-005326 T-58097136 1 AS -- (Procedure: 'isTimeout mfsemstr.p' Line:991) Connection failure for host 127.0.0.1 port 49772 transport TCP. (9407)
[13/10/09@07:52:35.908-0700] P-005326 T-58097136 3 AS AS -- TRACE: Non-PERSISTENT Procedure END STOP. (8397)
[13/10/09@07:52:35.908-0700] P-014544 T-55717360 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.908-0700] P-004583 T-2510261744 1 AS -- (Procedure: 'com/qad/shell/interface/GetGeneralizedCodes.p' Line:2181) Connection failure for host 127.0.0.1 port 51072 transport TCP. (9407)
[13/10/09@07:52:35.908-0700] P-005326 T-58097136 1 AS -- ** Pipe to subprocess has been broken. (140)
[13/10/09@07:52:35.908-0700] P-005326 T-58097136 1 AS -- Connection failure for host 127.0.0.1 port 49772 transport TCP. (9407)
[13/10/09@07:52:35.909-0700] P-015185 T-1042386416 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.909-0700] P-028920 T-345005552 1 AS -- Connection failure for host 127.0.0.1 port 53191 transport TCP. (9407)
[13/10/09@07:52:35.909-0700] P-005439 T-3417927152 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.909-0700] P-019782 T-2171715056 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.909-0700] P-004031 T-3647929840 1 AS -- Connection failure for host 127.0.0.1 port 51000 transport TCP. (9407)
[13/10/09@07:52:35.909-0700] P-004583 T-2510261744 1 AS -- (Procedure: 'com/qad/shell/authentication/Cleanup' Line:17) Connection failure for host 127.0.0.1 port 51072 transport TCP. (9407)
[13/10/09@07:52:35.910-0700] P-017931 T-2260782576 2 AS AS Application Server Shutdown. (5476)
[13/10/09@07:52:35.910-0700] P-004583 T-2510261744 3 AS AS -- TRACE: Non-PERSISTENT Procedure END STOP. (8397)
[13/10/09@07:52:35.910-0700] P-004583 T-2510261744 1 AS -- ** Pipe to subprocess has been broken. (140)
[13/10/09@07:52:35.910-0700] P-004583 T-2510261744 1 AS -- Connection failure for host 127.0.0.1 port 51072 transport TCP. (9407)
[13/10/09@07:52:35.910-0700] P-028920 T-345005552 2 AS AS Application Server Shutdown. (5476)

 

[Cringer]

Is there another application that is using the same port range as your appserver ports?

 

[rajeev.babu]

No Cringer. As per the standard procedure we dont/never use the same port for any other application. in this case
Port number = 18115
srvrMaxPort=28999
srvrMinPort=28000

 

[rajeev.babu]

Please note that we have this appserver and fin appserver trimming every 5 minutes to reduce the heavy load. Is that something going wrong, but again there is no issue with finappserver

 

[cj_brandt]

What does the database log show ?
Is one of the servers going down and that is why all the connection failures ? You can look at those with promon R&D -> 2 -> 2.

 

[Rob Fitzpatrick]

Just curious - if the AppServers are local to the DB, why are you connecting via TCP?

 

[RealHeavyDude]

It appears to me that your AppServer agents are trying to connect something ( a database? ) at runtime and that connection fails and is not properly error handled. If that's the case you might want to reconsider and connect the necessary database(s) at startup of the AppServer agents.

Heavy Regards, RealHeavyDude.

 

[rajeev.babu]

Thanks all for your reply. We are using AIA adapter also. Do we need to disable AIA adapter.

 

[rajeev.babu]

We have the financial appserver went down just like the production appserver with the following error

[13/10/25@01:55:18.180-0700] P-007874 T-3723103728 1 AS -- ** Pipe to subprocess has been broken. (140)
[13/10/25@01:55:18.180-0700] P-007874 T-3723103728 1 AS -- Connection failure for host 127.0.0.1 port 63799 transport TCP. (9407)
[13/10/25@01:55:18.180-0700] P-007874 T-3723103728 3 AS AS -- TRACE: shutdown Procedure 'program/shutdown.p' START (14244)

Yet to find the Root cause.

 

[jurriaan]

If interprocess communication breaks down, does the OS logs reveal anything (perhaps about processes being killed)?