How is -Ma determined...and is it trustworthy?

[doom1701]

This weekend was supposed to be the crowning jewel in my series of system upgrades over the past two months. I migrated our SX.Enterprise database (and character code and appserver code) to a shiny new VM running Redhat AS5 64bit. The new server is running OE 10.1C (64bit); old server is OE 10.1B.

The migration went fine--did a backup/restore (didn't want to touch the actual data on the old server just in case...good plan, unfortunately), a dump and load on the new server, upgraded our production terminal servers to 10.1C, checked all of the various ODBC stuff...everything went great. And there was no reason it shouldn't have--I had done the same thing three times previous and had various groups of users testing it beforehand. Then, this morning comes along, and I start getting calls about people being unable to connect. This was after we had about 120 connections (SX GUI users wind up with 2 or 3 database connections, so I figure we had about 50 users connected out of 140). Error says that there is no server available for the database.

My 4GL broker setup (I run separate brokers for 4GL and SQL) has a minimum number of clients per server value of 1, but no max. So I tried setting the max to a higher number (started with 20, and worked my way down from there), but the database wouldn't start. Then I realized that the .lg file for the database actually displays what max value it picks, so I compared old to new server. Both were picking an -Ma value of 3. While I'm surprised I never ran into a problem on the old system, I know that I was not reaching 3 per server on the new system.

I understand the overall function of the servers, but I'm at my limit right now. Ideally I'd like to be able to support 500+ connections (we're licensed for up to 1000 DB connections).

Below are the settings in the conmgr.properties file and a snippet from the .lg file showing the parms at startup. Both are from the old server; the only thing I've tried to change so far is blocksindatabasebuffers on the new server, but I even bumped that down to the original value.

 

[doom1701]

Having a heck of a time getting the section of the .lg file to post...I'll keep trying but if there's a value I can pull individually, don't hesitate to ask.

[configuration.nxt.defaultconfiguration]
    afterimageprocess=true
    afterimagestall=true
    asynchronouspagewriters=7
    beforeimagebuffers=30
    beforeimageprocess=true
    blocksindatabasebuffers=62500
    database=nxt
    directio=true
    displayname=defaultConfiguration
    locktableentries=50000
    maxservers=150
    maxusers=500
    otherargs=-lkrela
    semaphoresets=1
    servergroups=nxt.defaultconfiguration.defaultservergroup, nxt.defaultconfiguration.odbcservergroup
    spinlockretries=5000
    watchdogprocess=false

[database.nxt]
    autostart=false
    configurations=nxt.defaultconfiguration
    databasename=/db/nxt
    defaultconfiguration=nxt.defaultconfiguration
    displayname=nxt

[environment]

[servergroup.nxt.defaultconfiguration.defaultservergroup]
    configuration=nxt.defaultconfiguration
    displayname=defaultServerGroup
    minclientsperserver=1
    numberofservers=100
    port=nxt
    type=4gl

[servergroup.nxt.defaultconfiguration.odbcservergroup]
    configuration=nxt.defaultconfiguration
    displayname=odbcServerGroup
    maxclientsperserver=1
    numberofservers=50
    port=7150
    type=sql

 

[doom1701]

One more tidbit. I found some monitoring code someone shared here and fit it to my setup so I could fire off multiple connections to the database easily. With the settings above, I'm topping out at 78 remote connections to the database. I can open a connection on the database server directly, though. I can also connect in other ways to the database server (I've been firing off an ssh connection after I seem to peg the database and I get in OK).

If I add a SQL connection to the mix, I can still get 78 remote 4GL connections along with my SQL connection.

And here's something else (just found this): my min and max port numbers are the defaults (I think 1025-2000?). On our live server, if I do a netstat -nap |grep _mprosrv, all of the server processes are sitting on a port somewhere in that range. On the new server, 26 of the 100 are in that range; the rest are all above port 40,000. And looking further at the netstat results, all of the servers in the 1025-2000 port range have three connections each, and the rest have none.

Edit: It's definitely the port selection. I watched the .lg file as the connections first started, and it said it was setting up servers on ports below 2000 (lots in the 1900 range). But looking at the list, there are no servers in the 1900 port range. And even after a reboot, it's still 26 servers below 2000, and the rest are above. I'm considering changing the number of 4GL servers to 25 and just letting it set a higher maximum user count...but that won't answer why so many servers are going onto oddball ports. There's nothing else on this server it could be running into.

 

[TomBascom]

I would shift -minport & -maxport to some unused range with lots of open ports like -minport 50000 -maxport 51000 (you will have to check your local /etc/services to see how open that range really is...)

I would also go the other direction with -Ma & -Mn -- more servers, fewer clients per server. If you are trying to connect 500 users try -Mn 200 -Ma 5.

 

[doom1701]

Wow, compared to the services file in RH4, the file in RH5 is freakin' massive. I wouldn't be surprised if there were only 26 open ports in the default range. Unfortunately, setting a high range doesn't work; the 50000 - 51000 range is actually wide open in the services file, but if I put that into the config file and start the database, I get

ERROR: Unable to locate database <nxt>

If I keep the parameters (so I know that I put them in right) but I set it back to around the default port range, the database starts up fine. I may just check into cleaning up the services file--nothing is actually using those ports, but there's very few free ports in the lower ranges of the existing file.

I agree on the servers vs users thing--a consultant (that doesn't work for Infor, so I trust him...too bad Infor won't let him talk to me anymore) explained that to me when we setup the secondary SQL broker. Hopefully I can make this work while keeping that same setup.

 

[TomBascom]

It is unclear to me how and where you're setting minport & maxport but it shouldn't cause the problem that you describe.

BTW, I know I'm a broken record but... you, or your management, need to remind Infor who the customer is.

Not all Infor customers fall for that bunk.

 

[doom1701]

Well, like I've said before--the consultant is an Infor MA, so if they piss off Infor, they lose their bread and butter. Infor isn't telling me to not talk to the consultant...the consultant is telling me he can't talk to me.

I'm setting it in conmgr.properties (the only way I know how) using the long names for the parameters. Can I use the -shortname parms in that file as well?

Strange; the admin server log keeps saying that 50000 and 51000 are not valid values for those parameters; they need to be between 1025 and 65535. I guess I missed something in math class...

Anyway, it did let me expand the port range up to 1025-3000; looks like it is scanning for unregistered ports in the services file, as I can now get quite a few more connections. Looks like I've got 1 server that's on an oddball port number. I'll play with it more tomorrow--I'm done with this for the day.

 

[TomBascom]

There are other consultants in the world who don't have that problem.

You can use the command line parameters via the "otherargs" property. I usually just specify a .pf file so that I can add other args without editing the properties file. IOW:

otherargs=-pf myargs.pf

 

[doom1701]

I pulled the out of the box RH 5 services file into Excel this morning and totaled the number of unused ports. Between 1025 and 4000 there are only 200 open ports. Between 1025 and 2000 there are only 41 (and it's possible that there are a couple of things already on the server using some unregistered ports, which would be why I could only get 26 active servers). Expanding the port range seems to work fine. I guess the only thing that leaves me scratching my head is why Progress would suddenly start to pick ports way outside of the assigned range when it ran out without any kind of message in the log. When you actually max out all of your servers, the brokers reject the connection and log it; when there are open servers but they are outside of the assigned port range, the client times out trying to connect to them. Might actually be a client issue...but some mention would have been nice.

Oh well, not going to lose sleep over it. Thanks for pointing me in the right direction, Tom!