Question Server or system has no more resources

[jmac13]

Hi All,

Our Client is using 10.1c and recently they had issue with not being able to get on the system. They reported a message of Server out of Rescources message.

When I've looked in the DB log at the time they reported it I've got the below from one of the 5 databases we connect to.

The Brooker is complaining about No servers available. At first I thought this was to do with hitting the user limit. But then I would of thought they would of got "you have attempted to connect to database with too many users" error.

also the user said all sessions were showing as used but nobody was logged on (on promon). So just wondered what ppl thought? Could the -l message indicate that sessions are getting locked up?

3 of the dbs have number of servers set to 20, Number of clients per server 8 and they is also a separate server for sql connections for each one. 2 Other Db's have 20 and 8 set but no SQL. I dont believe we have sql connections to them and the below log is from a DB that has 4gl server and SQL server.

Thanks in advance

[2014/02/04@09:29:29.294+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:30:11.638+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:30:23.044+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:31:15.654+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:31:20.060+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:31:28.466+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:31:48.904+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:32:17.451+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:32:44.904+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:32:59.388+0000] P-4128       T-4132  I BROKER  0: (1154)  No servers are available.  Try again later.
[2014/02/04@09:33:02.216+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 200 to 210.
[2014/02/04@09:33:02.216+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:02.341+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 210 to 220.
[2014/02/04@09:33:02.341+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:02.513+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 220 to 230.
[2014/02/04@09:33:02.513+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:02.669+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 230 to 240.
[2014/02/04@09:33:02.669+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:02.810+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 240 to 250.
[2014/02/04@09:33:02.810+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:02.966+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 250 to 260.
[2014/02/04@09:33:02.966+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:03.060+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 260 to 270.
[2014/02/04@09:33:03.076+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:03.201+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 270 to 280.
[2014/02/04@09:33:03.201+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.
[2014/02/04@09:33:03.388+0000] P-3656       T-3420  I SRV    18: (5408)  WARNING: -l exceeded. Automatically increasing from 280 to 290.
[2014/02/04@09:33:03.388+0000] P-3656       T-3420  I SRV    18: (5512)  Previous message sent on behalf of user 128, server pid 3656, broker pid 4128.

 

[RealHeavyDude]

I've stumbled across the same issue some 2 years ago. I've never really got an explanation from Tech Support - but here it goes:

It turned out that the user count on individual remote servers was not reset when the users logged off so that the broker thought they were still occupied when they are not. You can check it in promon R&D / 1 / 3 to see the discrepancy.

I changed the port range with -minport and -maxport to a different range and the problem was gone. In the end it is just speculation, but at that time some port scanning by sercurity guys to identify vulnerable ports on our Solaris boxes was going on. I am inclined to believe that the port scanning caused the problem - in the end it appears to me that under specific circumstances the broker / remote server relation can get confused and I consider that a bug, not matter what Tech Support said and that they were not able to reproduce the problem.

Heavy Regards, RealHeavyDude.

 

[jmac13]

Thanks RHD. Strange its happen on another server (same customer) not so long ago.. so its bit of a head scratcher

 

[TomBascom]

Something similar can also happen on some releases with online backups taking up a user slot and not releasing it. SFAIK that's fixed on all *current* releases but if you're behind in your patching you might be seeing something along those lines.

The -l warning is just normal .lg file noise. Nothing to get excited about but you might want to set it high enough that you don't get that message anymore.

 

[Rob Fitzpatrick]

Something similar can also happen on some releases with online backups taking up a user slot and not releasing it. SFAIK that's fixed on all *current* releases but if you're behind in your patching you might be seeing something along those lines.

The -l warning is just normal .lg file noise. Nothing to get excited about but you might want to set it high enough that you don't get that message anymore.

My recollection is that that issue was introduced in 10.2B01. The functional issue (_connect slots used and not freed) was fixed in SP02. The "cosmetic" issue (_license and .lic file numbers are incorrect) was fixed in SP05. In a 10.1C environment there is some other issue. I'm inclined to think its may be related to minport/maxport, or some environmental issue like a port scanner, as RHD said.

I suggest you bounce it off Progress TS.

 

[jmac13]

My Boss is trying to pin this on me turning on the watchdog process I'm no one thinks this could be the cause? I personal said it wasn't that but I just want to check

 

[RealHeavyDude]

You boss is living by his CMA strategy - isn't he? ( CMA - cover my a.. )

The watch dog is a utility process for the database which is responsible for cleaning up shared memory connections. It has absolutely nothing to do with remote servers or clients.

Heavy Regards, RealHeavyDude.

 

[TomBascom]

As RHD says, WDOG is innocent. It takes exactly one connection.

 

[jmac13]

Hi Guys,

I've had this happen AGAIN.. sounds like ghost sessions are getting left behind taking the slots.. unfortunately ever time they've reported it, it has been after the fact so I've not been able to look at the dbs and all I can see is the No servers are available. Try again later. message in one of the db logs. Any Ideas what else I should be looking for? Could Watchdog clean up memory for one of these sessions but its still left behind?

thanks all

 

[Rob Fitzpatrick]

Which service pack is installed?
In this multi-DB environment is it always the same DB that throws the error?

Again, watchdog cleans up after dead self-service clients. It ignores remote clients, i.e. the clients that connect to servers and that will cause you to see (1154) errors.

You said your DBs are set with -Mpb 20 and -Ma 8. If you simply had more than 160 4GL clients attempting to connect then you wouldn't see 1154 errors; you'd see 748. So something else is going on. This error can also be caused by improper startup parameters. Can you please post your database startup parameters from the DB where the error occurs? If this is a multi-broker environment then we need a separate list of params for each broker. I am wondering if the value(s) for -Mpb are wrong in relation to -Mn.

 

[jmac13]

Hi Rob,

for this info am just looking at promon R&D 1 > 12 ?

thanks

 

[Rob Fitzpatrick]

How do you start your DBs? Are there shell scripts with .pf files or are they managed DBs configured in Progress Explorer?

Promon R&D 1 12 (basically _Startup) leaves out a lot of information so I don't find it too useful for issues like this. In particular I am looking for the per-broker parameters related to client/server, e.g. -n (DB-wide), -Mn (DB-wide), -Mpb (per-broker), -Ma (per-broker), -Mi (per-broker), -minport (per-broker), -maxport (per-broker).

Please answer my previous questions.

 

[jmac13]

Start DBs via Progress explorer/open edge explore depending on server. In this case its Progress explorer

 

[jmac13]

DB SFDC (one that had the error)

DB Wide
Maximum Users -n 160

Maximum Servers -Mn 24

4gl Broker
Maximum Servers Per Broker -Mpb 20

Maximum Clients per Server -Ma 8

Minimum Clients per Server -Mi 1

-minport 5000 -maxport 3000

I've also got a SQL brooker setup
Maximum Servers Per Broker -Mpb 4

Maximum Clients per Server -Ma 2

Minimum Clients per Server -Mi 1

-minport 5000 -maxport 3000


I've attached the screenshots of where i got this info from

thanks

 

[Cringer]

Are you sure about those min/maxport settings? They look wrong to me!

Edit: Just seen you mixed them up, so that's ok!

 

[Rob Fitzpatrick]

OK, first problem: your -Mn is too low. This will prevent you from spawning enough servers. You should set -Mn to the sum of -Mpb for all brokers plus the number of secondary brokers.

20 + 4 + 1 = 25. So -Mn should be 25, not 24.

Second problem: -n is too low.
4GL: 20 servers * 8 clients per server = 160 connections max.
SQL: 4 servers * 2 clients per server = 8 connections max.
Plus helper processes, utilities, self-service clients, etc. etc.

Set -n to 200.

Third suggestion: do not reuse port ranges across brokers or databases. Make every range unique and non-default. Yes, it's more work but it pays dividends when you're troubleshooting. Your ranges overlap and they are in the default range for Windows. I strongly suggest you change them. And if other DBs on the same box are configured the same, change those as well.

And you still haven't answered all my questions.

 

[jmac13]

Thanks Rob

to answer the other question.. I've had it happen to another site not so long ago and it was another DB..

Yes I agree some of the numbers aren't quite right.. But they been like this for ages.. I'll change them..
if I do change the port range whats the next step if this happens again? any suggestions what they should be I'm guessing above the default windows range?

 

[Rob Fitzpatrick]

Keep your server ports out of the default range for Progress, out of the low range (<1025) and out of the dynamically-assigned range (>32768). Other than that, pick a range you like and keep your services file free of crap and you should be fine.

When a port is defined in the OS services file, even if it isn't in use, a Progress 4GL broker won't assign a server to it (SQL behaves differently). So make sure you know which ranges are available in the services file before you start. But you only need one port per server so it isn't a lot.

I like to keep my ranges non-contiguous so that the minport/maxport for a given broker for a given DB has room to grow. That way if a client's user base grows and we want to add more servers we can just expand -maxport for the appropriates broker(s) and not have to reconfigure everything from scratch. A little planning saves a lot of future work and headaches.

If you make the changes that I suggested and the problem happens again then that means that the problems I identified above weren't your only problems. Also, you can minimize potential problems by ensuring your client is on the latest SP. And preferably, when possible, upgrade them from 10.1C.