D
dbeavon
Guest
I think I've asked this before but don't have a path forward yet. This is related to a fairly basic scenario that comes up in PASOE with client/server database connections. But the session manager seems unable to account for this issue and throws lots of unhandled errors. Here is the session manager log (abl_train_uec.2020-04-09.log) 20:46:49.080/37943896 [thd-9] ERROR com.progress.appserv.Session - LocalSession(4D305FC34905EAF3BE5BB1594EA2E8408AA6D4D9A13A.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 20:46:49.082/37943898 [thd-9] ERROR com.progress.appserv.Session - LocalSession(4D305FC34905EAF3BE5BB1594EA2E8408AA6D4D9A13A.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 20:56:19.089/38513905 [thd-4] ERROR com.progress.appserv.Session - LocalSession(DFF282303244F872EC661C2E92C5144A3AA206EB15A0.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 20:56:19.090/38513906 [thd-4] ERROR com.progress.appserv.Session - LocalSession(DFF282303244F872EC661C2E92C5144A3AA206EB15A0.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:05:32.086/39066902 [thd-7] ERROR com.progress.appserv.Session - LocalSession(9D7C250A6BEA4878C7363C3A653E27B1712944990B5D.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:05:32.086/39066902 [thd-7] ERROR com.progress.appserv.Session - LocalSession(9D7C250A6BEA4878C7363C3A653E27B1712944990B5D.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:05:32.088/39066904 [thd-9] ERROR com.progress.appserv.Session - LocalSession(D8EF567A80C14C841BFD923968589014040FFDEB9304.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:05:32.088/39066904 [thd-9] ERROR com.progress.appserv.Session - LocalSession(D8EF567A80C14C841BFD923968589014040FFDEB9304.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:05:45.735/39080551 [thd-6] ERROR com.progress.appserv.Session - LocalSession(D7BE5A9DF0052AD0B63F6CA93FA7C85A31BE8A67F13C.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:05:45.735/39080551 [thd-6] ERROR com.progress.appserv.Session - LocalSession(D7BE5A9DF0052AD0B63F6CA93FA7C85A31BE8A67F13C.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:05:45.737/39080553 [thd-8] ERROR com.progress.appserv.Session - LocalSession(77B4BF400CDBCB02337E7CEE33E610A06648BD87969A.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:05:45.738/39080554 [thd-8] ERROR com.progress.appserv.Session - LocalSession(77B4BF400CDBCB02337E7CEE33E610A06648BD87969A.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:05:54.299/39089115 [thd-5] ERROR com.progress.appserv.Session - LocalSession(84009B5AE0C1F1D9207CB1D949EB12C86613BB57BCD2.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:05:54.299/39089115 [thd-5] ERROR com.progress.appserv.Session - LocalSession(84009B5AE0C1F1D9207CB1D949EB12C86613BB57BCD2.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:06:01.582/39096398 [thd-4] ERROR com.progress.appserv.Session - LocalSession(9117478F0284697396831747451415D3B0DB586BE730.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:06:01.583/39096399 [thd-4] ERROR com.progress.appserv.Session - LocalSession(9117478F0284697396831747451415D3B0DB586BE730.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:06:04.741/39099557 [thd-9] ERROR com.progress.appserv.Session - LocalSession(1DF37236375E28A5895302865814944EBFDA0F7982CB.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:06:04.741/39099557 [thd-9] ERROR com.progress.appserv.Session - LocalSession(1DF37236375E28A5895302865814944EBFDA0F7982CB.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:06:04.741/39099557 [thd-1] ERROR com.progress.appserv.Session - LocalSession(33B3C329971F8E6FA0357D4F61324F50B9C62D08451F.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:06:04.741/39099557 [thd-1] ERROR com.progress.appserv.Session - LocalSession(33B3C329971F8E6FA0357D4F61324F50B9C62D08451F.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:06:04.743/39099559 [thd-5] ERROR com.progress.appserv.Session - LocalSession(7C16FBBE4C3FD470B3581E9E56483CA81FDEA7CA2FA5.oepas1) : error occurred while reading a message processAgentRsp() = java.io.EOFException:null. (18300) 21:06:04.743/39099559 [thd-5] ERROR com.progress.appserv.Session - LocalSession(7C16FBBE4C3FD470B3581E9E56483CA81FDEA7CA2FA5.oepas1) : an error occurred while reading response message = java.io.EOFException:null. (18296) 21:06:04.743/39099559 [thd-6] ERROR com.progress.appserv.Session - LocalSession(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : error occurred while sending a message WRITEDATA = java.net.SocketException: Software caused connection abort: socket write error:Software caused connection abort: socket write error. (18299) 21:06:04.744/39099560 [thd-6] ERROR c.p.appserv.PoolMgt.Connection - TcpAgentConnectionPool(bl9eolKsStacAJ-cR3F0hQ,P54W0MoTT125gb3a5jvhyw) : error in closing connection = java.net.SocketException: Software caused connection abort: socket write error:Software caused connection abort: socket write error (18267) 21:06:04.744/39099560 [thd-6] ERROR com.progress.appserv.Session - LocalSession(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : Error sending message UBRQ_WRITEDATA = java.net.SocketException: Software caused connection abort: socket write error:Software caused connection abort: socket write error. (18293) 21:06:04.744/39099560 [thd-6] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent. (18324) 21:06:04.744/39099560 [thd-6] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : network exception occurred processing request : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent : NetworkError[java.net.SocketException: Software caused connection abort: socket write error : Error sending message for (83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) = Software caused connection abort: socket write error]:Agent. (18318) 21:06:04.745/39099561 [thd-8] ERROR c.p.appserv.adapters.apsv.Request - APSV(33B3C329971F8E6FA0357D4F61324F50B9C62D08451F.oepas1) : An error occurred processing the POST request : Unexpected error : com.progress.appserv.broker.exception.BrokerException$InvalidSessionStateException: Session. (18320) 21:06:04.747/39099563 [thd-3] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.748/39099564 [thd-3] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.748/39099564 [thd-3] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.797/39099613 [thd-9] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.797/39099613 [thd-9] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.797/39099613 [thd-9] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.799/39099615 [thd-4] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.799/39099615 [thd-4] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.799/39099615 [thd-4] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.800/39099616 [thd-1] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.800/39099616 [thd-1] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.801/39099617 [thd-1] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.845/39099661 [thd-10] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.846/39099662 [thd-10] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.846/39099662 [thd-10] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.847/39099663 [thd-5] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.847/39099663 [thd-5] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.847/39099663 [thd-5] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.848/39099664 [thd-6] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! 21:06:04.848/39099664 [thd-6] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : An error occurred while writing message UBRQ_WRITEDATA = com.progress.appserv.broker.exception.BrokerException$NoAvailableConnectionsException: Agent. (18324) 21:06:04.848/39099664 [thd-6] ERROR c.p.appserv.adapters.apsv.Request - APSV(83F0506FC5DC69BC41E72BA8F3B5922F69152FA91485.oepas1) : No connections available to process request.. (18318) 21:06:04.888/39099704 [thd-8] ERROR com.progress.appserv.Session - processWriteData(vF3_2397R5a1pdkY2LC9QA,SESSION ) RQID : no connection found! Notice all the scary-looking unhandled exceptions. Even worse than these exceptions are the related issues with tomcat/HTTP sessions. None of them are closed after the failures. The APSV client is notified of the failure but the tomcat/HTTP sessions are left hanging around. The issue leaves a massive mess that needs to be cleaned up manually in the tomcat manager console. These exceptions come up during high utilization. If I look in the PASOE agent logs I find the problem: [20/04/09@20:46:48.134-0400] P-003800 T-002124 1 AS-14 -- The server or the system has no more resources. Please contact Progress Technical Support. (748) [20/04/09@20:46:48.134-0400] P-003800 T-002124 1 AS-14 MSAS Connection to networked db lumbertrack lost for AS-14. [20/04/09@20:46:48.135-0400] P-003800 T-002124 1 AS-14 -- ** Server rejected login. (700) [20/04/09@20:46:48.135-0400] P-003800 T-002124 1 AS-14 -- Error initializing the application server. (5479) [20/04/09@20:46:48.135-0400] P-003800 T-002124 1 AS-14 MSAS Unable to initialize session! [20/04/09@20:46:48.135-0400] P-003800 T-002124 1 AS-Aux-0 MSAS Could not get and/or initialize 'Stateless' session! Cannot process request. [20/04/09@20:46:48.135-0400] P-003800 T-002124 1 AS-Aux-0 MSAS Error handling request! Status=-1003 [20/04/09@20:46:48.136-0400] P-003800 T-003704 1 AS-ResourceMgr MSAS Spawning New Worker Thread. Number: 6 [20/04/09@20:46:48.136-0400] P-003800 T-002124 1 AS-Aux-0 MSAS Worker Thread exiting. Number: 4, Status: -14 [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-15 -- The server or the system has no more resources. Please contact Progress Technical Support. (748) [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-15 MSAS Connection to networked db lumbertrack lost for AS-15. [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-15 -- ** Server rejected login. (700) [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-15 -- Error initializing the application server. (5479) [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-15 MSAS Unable to initialize session! [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-Aux-0 MSAS Could not get and/or initialize 'Stateless' session! Cannot process request. [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-Aux-0 MSAS Error handling request! Status=-1003 [20/04/09@20:56:18.143-0400] P-003800 T-003704 1 AS-ResourceMgr MSAS Spawning New Worker Thread. Number: 4 [20/04/09@20:56:18.143-0400] P-003800 T-005836 1 AS-Aux-0 MSAS Worker Thread exiting. Number: 6, Status: -14 This is simply the error you get when the OE dbms stops accepting remote client/server connections. (You have attempted to connect to a database with too many users connected to it. Retry the connection later, or increase -n on the server.) Can someone tell me if this is expected behavior for PASOE? It seems like it should be a bit more resilient. I would think that a connectivity failure between the msagent's ABL session and the database shouldn't have such a severe resource consequences within the session manager (tomcat plugin). I suspect I can open a support ticket with Progress easily enough but I thought I would check if I missed something obvious in my configuration first. I can't imagine I'm the only one who has dealt with an ABL session that cannot connect to the database. There is a defect that is in the KB which points to a similar message, and similar connectivity issues dealing with PASOE and "external systems". It is vague, but I cannot imagine that by "external systems" they are referring to a connection with an OE database. That is where I'm seeing my issues. Progress KB - Opening a socket in PASOE crashes the agent Any tips would be appreciated.
Continue reading...
Continue reading...