HTTP client holding onto connections?

Tarby777

Member
Hi all,

In 11.6.4, I'm using the HTTP client to perform a number of operations in SharePoint via its REST API. The code is running on a Webspeed agent in PASOE. Monitoring the sessions using the Windows Resource Monitor, I see that a thread starts up and just stays in an active state and never dies, as opposed to other threads that show as greyed-out after they're done, and eventually quit. After a few calls, the agent is unable to handle any new requests. The problem of threads not dying is limited to those that contact SharePoint.

All my SharePoint code is contained in static methods in a single class, and I'm careful to DELETE OBJECT everything associated with the HTTP client in those methods when I'm done. My HTTP headers are set to "close" as opposed to "keep-alive" and I'm at a loss as to why the agent threads are hanging around.

I have run the PKB's leakCheck.p and the only new objects in memory after calling one of my methods are several instances of OpenEdge.NET.HTTP.BuilderRegistry - I understand from a Communities post by Peter Judge that this is expected, and OK.

I understand that 11.7.5 and v12 have several fixes for memory leaks in the HTTP client that have not been fixed in 11.6, but I'm not aware of any problem - fixed or not - around threads getting stuck in 11.6.

Here is a code sample:

Code:
CLASS Interface.IF012-SharePoint FINAL:

METHOD PUBLIC STATIC VOID UploadFileFromMemory (INPUT pmFile AS MEMPTR,
                                                    INPUT pcSharePointFileName AS CHARACTER,
                                                    INPUT pcSharePointFileType AS CHARACTER,
                                                    INPUT plApplySuffix AS LOGICAL,
                                                    INPUT pcSite AS CHARACTER,
                                                    INPUT pcSharePointFolder AS CHARACTER,
                                                    OUTPUT pcMessage AS CHARACTER):
       
        DEFINE VARIABLE oRequest        AS IHttpRequest NO-UNDO.
        DEFINE VARIABLE oRequestBody    AS OpenEdge.Core.Memptr NO-UNDO.
        DEFINE VARIABLE oResponse       AS IHttpResponse NO-UNDO.
       
        DEFINE VARIABLE cFile           AS CHARACTER NO-UNDO.
        DEFINE VARIABLE cTenantName     AS CHARACTER NO-UNDO.
        DEFINE VARIABLE cAccessToken    AS CHARACTER NO-UNDO.
        DEFINE VARIABLE cURI            AS CHARACTER NO-UNDO.
       
        /* Get the tenant name from Config */
        Interface.IF012-SharePoint:GetConfigValue(INPUT "{&SharePointConfigItem}",
                                                  INPUT "TenantName",
                                                  OUTPUT cTenantName).
                                                 
        IF cTenantName = "" THEN DO:
            pcMessage = "IF012-SharePoint:UploadFileFromMemory could not retrieve the tenant name from system config.".
            RETURN.
        END.
       
        /* Get the auth token from Config */
        Interface.IF012-SharePoint:GetAuthToken(OUTPUT cAccessToken,
                                                OUTPUT pcMessage).
                                                 
        IF pcMessage <> "OK" THEN
            RETURN.
                                                 
        IF GET-SIZE(pmfile) = 0 THEN DO:
            pcMessage = "IF012-SharePoint:UploadFileFromMemory could not read the local file from memory.".
            RETURN.
        END.
       
        /* If the user wants to use a suffix on the filename to avoid an existing file being overwritten,
           we have to find out what the current highest suffix for the filename is, and increment it by one. */
        IF plApplySuffix THEN
        DO:
            Interface.IF012-SharePoint:GetNextFreeFileName(INPUT pcSharePointFileName,
                                                           INPUT pcSharePointFileType,
                                                           INPUT pcSite,
                                                           INPUT pcSharePointFolder,
                                                           OUTPUT cFile,
                                                           OUTPUT pcMessage).
            IF pcMessage <> "OK" THEN RETURN.
        END.
       
        ELSE cFile = pcSharePointFileName + "." + pcSharePointFileType.
           
        oRequestBody = NEW OpenEdge.Core.Memptr(pmFile).
       
        CASE pcSite:
       
            WHEN "" then
                cURI = SUBSTITUTE("https://&1.sharepoint.com/_api/web/GetFolderByServerRelativeUrl('&2')/files/add(url='&3',overwrite=true)",
                    cTenantName,
                    pcSharePointFolder,
                    cFile).
         
            OTHERWISE
                cURI = SUBSTITUTE("https://&1.sharepoint.com/sites/&2/_api/web/GetFolderByServerRelativeUrl('&3')/files/add(url='&4',overwrite=true)",
                    cTenantName,
                    pcSite,
                    pcSharePointFolder,
                    cFile).
               
        END CASE.
       
        oRequest = RequestBuilder:Post(cURI, oRequestBody)
            :AcceptJson()
            :AddHeader("Connection",     "close")
            :AddHeader("Authorization",  cAccessToken)
            :HttpVersion("HTTP/1.1")
            :Request.

        oRequest:ContentType = "application/octet-stream".  
        oRequest:ContentLength = GET-SIZE(pmFile).
 
        oResponse = ClientBuilder:Build():Client:Execute(oRequest).

        IF oResponse:StatusCode = INTEGER(OpenEdge.Net.HTTP.StatusCodeEnum:OK) THEN
            pcMessage = "OK".
       
        ELSE
            pcMessage  = "IF012-SharePoint:UploadFileFromMemory was unable to upload a file to the SharePoint folder: " + cUri
                       + CHR(10) + CHR(13)
                       + CHR(10) + CHR(13)
                       + "Status code:   " + STRING(oResponse:StatusCode)
                       + CHR(10) + CHR(13)
                       + "Status reason: " + oResponse:StatusReason
                       + CHR(10) + CHR(13).
               
        FINALLY:
            DELETE OBJECT oRequest NO-ERROR.
            DELETE OBJECT oRequestBody NO-ERROR.
            DELETE OBJECT oResponse NO-ERROR.
        END FINALLY.
                 
    END METHOD.    

END CLASS.

Invoked like this:

Code:
USING Interface.IF012-SharePoint.

IF012-SharePoint:UploadFileFromMemory        
    (input mPhoto,
     input cfile,
     input "jpg",
     INPUT FALSE,
     input cSharepointSite,
     input cSharepointFolder,
     output cMessage).
 
  SET-SIZE(mPhoto) = 0.

Am I doing anything obviously wrong?

TIA
Tarby
 
Last edited:

Cecil

19+ years progress programming and still learning.
Code:
    define variable cSSLProtocols as character extent no-undo.
    define variable cSSLCiphers   as character extent no-undo.

extent(cSSLProtocols) = 2.
        extent(cSSLCiphers) = 10.
        
        // TLSv1.1 and TLSv1.2 are supported with OpenEdge 11.6 and later.  See article What version of SSL and/or TLS does Progress OpenEdge use ?
        // Supported ciphers and protocols at https://docs.progress.com/bundle/openedge-security-and-auditing/page/Supported-protocols-ciphers-and-certificates-for-OpenEdge-clients-and-servers.html
        assign
            cSSLProtocols[1] = 'TLSv1.2' 
            cSSLProtocols[2] = 'TLSv1.1'
            cSSLCiphers[1]   = 'AES128-SHA256'
            cSSLCiphers[2]   = 'DHE-RSA-AES128-SHA256'
            cSSLCiphers[3]   = 'AES128-GCM-SHA256'
            cSSLCiphers[4]   = 'DHE-RSA-AES128-GCM-SHA256'
            cSSLCiphers[5]   = 'ADH-AES128-SHA256'
            cSSLCiphers[6]   = 'ADH-AES128-GCM-SHA256'
            cSSLCiphers[7]   = 'ADH-AES256-SHA256'
            cSSLCiphers[8]   = 'AES256-SHA256'
            cSSLCiphers[9]   = 'DHE-RSA-AES256-SHA256'
            cSSLCiphers[10]  = 'AES128-SHA'.


Code:
        if temp-table ttFileAttachement:has-records then
            messageJSON:add("attachments", this-object:GetAttachmentsJSON() ).
        
        assign
            urlEndpoint = 'https://graph.microsoft.com/v1.0/me/messages'.
        
        /** PASS the URI string and have a URI class object returned **/
        uploadUrlObj = OpenEdge.Net.URI:Parse( urlEndpoint ).
        
        oLib = ClientLibraryBuilder:Build()
                                   :SetSSLProtocols(cSSLProtocols)
                                   :SetSSLCiphers(cSSLCiphers)
                                   :ServerNameIndicator( uploadUrlObj:Host )
                                   :Library.
                                  
        hc = ClientBuilder:Build()
                          :UsingLibrary(oLib)
                          :Client.
        
        req = RequestBuilder:POST(uploadUrlObj, messageJSON)
            :AddHeader('Authorization', 'bearer ' + string(OAuthAccessToken:AccessToken))
            :AcceptContentType('application/json')
            :Request.
 

Tarby777

Member
Hi Cecil. I've added your cipher/protocol code into my code, but I haven't had a chance to test it yet. They way they configured PASOE here, there's only a single instance shared between test and live, so I have to pick my moment otherwise I mess things up for the users. I'll update the thread when I have tested the changes. Out of interest, was your problem in SharePoint or something else?
 

Tarby777

Member
It's really not happy. I just ran with the cipher changes, and it crashed the _mproapsv process when I tried to connect to SharePoint. I tried it twice - same result both times. I see this in the log (I had to snip part of the stack trace to keep under the ProgressTalk message size limit) :

Code:
09:13:25.797/69055110 [4f80TVKeRSeLNRO8Z0jddQ-agent-watchdog] WARN  c.p.appserv.PoolMgt.AgentWatchdog - AgentWatchdog(4f80TVKeRSeLNRO8Z0jddQ) : agent zMPaSQgDRgKi31q4L5d15g PID= 9192 has terminated.
09:13:25.804/69055117 [catalina-exec-21] ERROR com.progress.appserv.Session - LocalSession(giCzYnDNQxmkR-KYXCOxUw) : error occurred while reading a message readMsg() = java.net.SocketException: Connection reset:Connection reset. (18300)
09:13:25.804/69055117 [catalina-exec-21] WARN  com.progress.appserv.Agent - TcpAgentConnectionPool(Scs8mFOcRAGSP7qExCxA_g) : removeAgentConnection() failed to remove connection : size= 0 max= 9
09:13:25.804/69055117 [catalina-exec-21] ERROR com.progress.appserv.Session - LocalSession(giCzYnDNQxmkR-KYXCOxUw) : an error occurred while reading response message = java.net.SocketException: Connection reset:Connection reset. (18296)
09:13:25.811/69055124 [catalina-exec-21] ERROR com.progress.O4gl.Trace - READPACKET IOException : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent (NetworkError[java.net.SocketException: Connection reset : Error reading message for (giCzYnDNQxmkR-KYXCOxUw) = Connection reset]:Agent)
com.progress.appserv.broker.exception.BrokerException$NetworkException: NetworkError[java.net.SocketException: Connection reset : Error reading message for (giCzYnDNQxmkR-KYXCOxUw) = Connection reset]:Agent
    at com.progress.appserv.broker.session.LocalSession.processAgentRsp(LocalSession.java:1235) ~[oebroker.11.6.4.jar:na]
    at com.progress.appserv.broker.session.LocalSession.readMsg(LocalSession.java:965) ~[oebroker.11.6.4.jar:na]
    at com.progress.appserv.clientrt.broker.BrokerSystemOEWeb.readMsg(BrokerSystemOEWeb.java:84) ~[oeclientrt.11.6.4.jar:na]
[snip]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.79]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.79]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at com.progress.appserv.services.security.OECORSFilter.doFilter(OECORSFilter.java:369) [oesecurity.11.6.4.jar:na]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.1.4.RELEASE.jar:3.1.4.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343) [spring-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260) [spring-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.79]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.79]
[snip]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.79]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
09:13:25.820/69055133 [catalina-exec-21] ERROR c.p.appserv.adapters.web.Request - Open4GLException executing POST request = com.progress.open4gl.Open4GLException: Communication layer message: General Error: READPACKET IOException : com.progress.appserv.broker.exception.BrokerException$NetworkException: Agent (NetworkError[java.net.SocketException: Connection reset : Error reading message for (giCzYnDNQxmkR-KYXCOxUw) = Connection reset]:Agent). (7175) : reason= null
09:13:41.629/69070942 [catalina-exec-1] ERROR com.progress.appserv.Agent - releaseABLSession(vRs7m_sAStG5fc4QOTTqTA) failed : invalid session count = 0 (18234)
 

Cecil

19+ years progress programming and still learning.
I think you need to raises a support ticket.

I'm connecting to MS Graph to create emails with attachments and I found that it would just hang for about 60 seconds and the return an error (can't remember the error). I was only after I stipulated to use ciphers in the connection it worked without error.

Are you able to reproduce the error outside of PASOE?
 

Tarby777

Member
> Are you able to reproduce the error outside of PASOE?

No, and that's the thing. My original problem is that after communicating with SharePoint from PASOE, Resource Monitor shows the thread that was instantiated for that task - which only takes a couple of seconds - just sitting there forever in active status... each new call to SP starts a new thread and eventually it clogs the system up. The communication with SharePoint is successful without the cipher/protocol code... I can read folders, upload files etc... it's just that PASOE doesn't clean up after itself when the job is done.

Adding the cipher/protocol code causes PASOE to go postal... the browser running the Webspeed code that calls SharePoint pops up a dialog about an unknown Appserver error, the _mproapsv process and all its threads die and a new _mproapsv process starts up.

If I run the same routines - with or without the cipher/protocol code - in Developer Studio, hooking up to the same SharePoint server, I see no error messages and no build up of threads / processes.
 
Top