Server or system has no more resources

[dlangschied]

Hi!
I am having an issue where I cannot login to one of my databases using a remote connection. It is only one of a dozen databases, so it does not appear to be server related (Linux RHEL). I can connect via ODBC and SELF, so it does not appear to be the database (OE10.1). Any ideas?


Thanks!
David

 

[Rob Fitzpatrick]

The specific error number would be helpful. It sounds like a 748 error.

In this case you can no longer connect new remote 4GL clients, as you have hit a constraint, for example:

• you already have your max connections (specified by -n); it may be set too low;
• you have already spawned all available servers and those servers are each at their maximum clients;
• your existing servers are maxed out, and you have _Servers slots available, but the broker can't spawn another server as the ports in the range from -minport to -maxport are not free;
• your total servers (-Mn) is too low, given the number of secondary brokers and your individual brokers' settings for max servers per broker (-Mpb);
• you have remote 4GL clients that are disconnecting from the database but are not being cleaned up from _Connect for some reason;
• you have found a Progress bug that results in this symptom.

Connect a self-service client and run promon or ProTop and get the lay of the land. Also check your startup parameters (the ones that are client-related) for your brokers and see if they are correct.

 

[dlangschied]

It is a748 error. What makes this strange is that the only connection that does not work. There is only one client that connects to the database in this way and the number of users connecting is most times only one. There are only 50 users total connecting to the database, as well.

 

[Rob Fitzpatrick]

At the time when you are getting this error remotely, look at the DB in promon.

 

[dlangschied]

What am I suppose to be looking for?

 

[Rob Fitzpatrick]

Look at Servers by Broker (promon R&D | 1 | 17), assuming your OE release is late enough that you have that option; 10.1C maybe? Have all your allocated servers been spawned? How many connections are reported for each?

Have you verified that the startup parameters related to remote client connections are correct? Posting an extract of your DB log from your latest startup (first 70 lines or so) would be helpful.

 

[dlangschied]

Here is the output from promon. It shows alot of logins and no current users. Since I am unable to login this makes sense. As far as the startu, I will provide that, but, as you can see, there are four brokers all with 50 max users. The max users for the database is 200.

[FONT=r_ansi][FONT=r_ansi]04/03/12 Status: Servers By Broker
08:28:34
Sv Pend. Cur. Max. Port
No Pid Type Protocol Logins Users Users Users Num
0 8070 Login TCP 0 0 0 50 12005
1 17751 Auto TCP 403 0 0 50 1049
2 18487 Auto TCP 85 0 0 50 1050
3 15234 Auto TCP 10 0 0 50 1072
4 4678 Auto TCP 2 0 0 50 1099

Thanks for helping!
[/FONT][/FONT]

 

[dlangschied]

Also, from activity... 50 users logged in none remote.

[FONT=r_ansi][FONT=r_ansi]4 Servers, 50 Users (50 Local, 0 Remote, 0 Batch),4 Apws
[/FONT][/FONT]

 

[Rob Fitzpatrick]

It helps to use CODE tags to make this info a little more legible:

04/03/12 Status: Servers By Broker
08:28:34
Sv                             Pend. Cur.  Max.  Port
No Pid   Type  Protocol Logins Users Users Users Num
0  8070  Login TCP      0      0     0     50    12005
1  17751 Auto  TCP      403    0     0     50    1049
2  18487 Auto  TCP      85     0     0     50    1050
3  15234 Auto  TCP      10     0     0     50    1072
4  4678  Auto  TCP      2      0     0     50    1099

I would still like to see the database log, but based on what you have provided so far here's what I see:

• You run one 4GL broker
• -Mn is 4
• -Ma is 50 (!)
• -n is 200
• you have 50 self-service users
• your first server has dealt with the majority of remote logins, and your first two servers have dealt with nearly all logins, so
  • either you have very few remote users connecting concurrently, in which case -Ma 50 makes no sense, or
  • your -Mi has been set too high (it should be left at the default of 1 so connections will round-robin between servers)
• with up to 200 remote users (50 * 4) and 50 self-service users, -n is too low

It looks like you can do with some tuning. All that said I don't yet see why you're getting 748 errors, unless the data shown from promon was not collected at the time you were experiencing the errors.

 

[TomBascom]

I generally agree the -Mi = 1 is preferred.

However in single-broker situations where 4GL and SQL connections are mixed it can be a problem. A server can only be a 4GL server or a SQL server so if all -Mn servers are active, all serving 4GL clients there will not be any SQL servers available to connect to (or vice versa).

The best solution, of course, is to start distinct brokers for 4GL and SQL connections. But if you cannot, or will not, do that then you can throw performance under the bus and set -Ma and -Mi to very large values thereby ensuring that a server stays open for the other sorts of connections.

 

[dlangschied]

Ok,
I am not oppose to making changes. I am not having any issues with SQL user connections, but I have had on my plate to separate them on another port. I do this only because I have had issues in the past where SQL does not always function after a restart of the database.

I am a little confused about the inferrence that the first server is handling the remote logins (Rob's comment). My reality is that I cannot make any remote connections. If I try to make a remote connection, these numbers do not change. I guess the one thing that I wish to stress is that I cannot even make a single remote client connection via 4GL. SQL works just fine self works fine. The only thing that I even use remote client for is to connect to my production databases to compile my developed code. My work around is to compile my code using my test database (no schema differences). I just want to figure out what went wrong and to fix it. I also don't want my databases to become unavailable to self or SQL.

Thanks for your help! You guys are invaluable!
David

Below are the server numbers from startup:
[FONT=r_ansi][FONT=r_ansi][2012/03/25@06:40:14.459-0400] P-8070 T-0 I BROKER 0: (4257) Maximum
Number of Clients Per Server (-Ma): 50.
[2012/03/25@06:40:14.459-0400] P-8070 T-0 I BROKER 0: (4258) Maximum
Number of Servers (-Mn): 5.
[2012/03/25@06:40:14.459-0400] P-8070 T-0 I BROKER 0: (4259) Minimum
Clients Per Server (-Mi): 1.
[2012/03/25@06:40:14.459-0400] P-8070 T-0 I BROKER 0: (4260) Maximum
Number of Users (-n): 201.
[/FONT][/FONT]

 

[TomBascom]

Rob is saying that because the first server has had 403 logins. The 2nd server has had 85, the 3rd 10 and the 4th has only had 2.

 

[Rob Fitzpatrick]

I think we're getting there. It would really help to have all your parameters.

Look at your process list (ps -ef | grep dbname) and see what processes are running against the database. I can't tell from this promon output what kind of servers these are, but with -Mn 5 and only 4 servers running, you aren't yet at your max server count. So it may be that the broker is trying to spawn a server when a remote user connects, and is unable to. If that is the case you will see a message to that effect in the database log. Check the log at the time when the remote user tries to connect.

Also, just to confirm, you are on 10.1C and not 10.2B, correct? There is a bug in probkup online in pre-SP02 10.2B that can cause 748 errors.

As Tom said, it is best to segregate 4GL from SQL, having different brokers and ports for each. It will make your life simpler.

And it looks like you are not specifying -minport and -maxport (again, I'm guessing); I recommend that you do. Lack of available TCP ports can prevent servers from spawning. And in the case of 4GL servers specifically, the port doesn't even have to be in use by another process. If a port is defined in /etc/services the broker won't use it for a 4GL server.

 

[dlangschied]

Well, I feel as though I am not stating my case efficiently. Please bear with me (though I don't deserve it).

1. This database and all other databases (5) have been operational for about 3 years as OE 10.1C SP04. The Server is RHEL 3. The applicaiton is QAD eB2.1.
2. They all, right or wrong, have been started as server type BOTH for the duration
3. They all, right or wrong, have been started as -n 200, -Ma 50, -Mn 5, -Mi 1 for the duration
4. After adding a new data extent to three of my databases 10 days ago, one of the three cannot be logged into using my_db -ld my_db_ld -S my_db_sv -H my_host. This connection is performed from our software control application, Roundtable, in order to compile code from our application server using databases on our database server.
5. I have no issues with users who connect directly using a telnet session.
6. I have no issues with ODBC connections from my barcode server.
7. I have no issues with ODBC connections from our corporate offices for our report tool QlikView.

I feel as though a restart of the database is in order, but I want to know what is wrong, so that I can be certain that a restart will fix it. I am willing to change some things, such as startup parameters to fix whatever is wrong. I cannot at this time separate the SQL and 4GL. I have some issues at our corporate office that prevents me from changing port numbers easily.

Thanks!
David Langschied

 

[Rob Fitzpatrick]

The priority right now is to address the 748 errors. Without knowing more, I think a restart would just cloud the issue.

Please take a look at your process list and tell me what broker and server (_mprosrv and _sqlsrv2) processes are running against this database specifically. Your broker and your 4GL servers are _mprosrv, and the startup parameters will tell you which is which (e.g. your broker will have params like -n, -Mn, -Ma, etc., and servers will have -m1). SQL servers are _sqlsrv2 processes.

Also, try to connect a remote 4GL client (which you have said will fail) and see what is in the database log at that timestamp.

 

[dlangschied]

Please take a look at your process list and tell me what broker and server (_mprosrv and _sqlsrv2) processes are running against this database specifically.

[FONT=r_ansi][FONT=r_ansi]root 8070 1 0 Mar25 ? 00:02:37 /pkg1/appl/dlc/101c/bin/_mprosrv /pkg1/db/admprod_01.db -pf /pkg1/appl/qad/pfs/admprod_01.pf
[/FONT][/FONT]
admprod_01
[FONT=r_ansi][FONT=r_ansi]-B 80000
-L 60000
-bibufs 135
-aibufs 135
-S admprod_01
-N TCP
-spin 3000
-aistall
-n 200
-ServerType Both
[/FONT][/FONT]

[FONT=r_ansi][FONT=r_ansi]root 17751 1 1 Mar25 ? 03:08:44 /pkg1/appl/dlc/101c/bin/_sqlsrv2 -D juniper.serverid=1 -D juniper.dbfile="/pkg1/db/admprod_01" -D juniper.config ="admprod_01.virtualconfig" -D juniper.service="admprod_01.virtualconfig.virtual" -D juniper.propertyfile="" -D juniper.cplog="ISO8859-1" -D juniper.cpcase="Bas
root 18487 1 0 Mar25 ? 00:10:26 /pkg1/appl/dlc/101c/bin/_sqlsrv2 -D juniper.serverid=2 -D juniper.dbfile="/pkg1/db/admprod_01" -D juniper.config ="admprod_01.virtualconfig" -D juniper.service="admprod_01.virtualconfig.virtual" -D juniper.propertyfile="" -D juniper.cplog="ISO8859-1" -D juniper.cpcase="Bas
root 15234 1 0 Mar26 ? 00:00:59 /pkg1/appl/dlc/101c/bin/_sqlsrv2 -D juniper.serverid=3 -D juniper.dbfile="/pkg1/db/admprod_01" -D juniper.config ="admprod_01.virtualconfig" -D juniper.service="admprod_01.virtualconfig.virtual " -D juniper.propertyfile="" -D juniper.cplog="ISO8859-1" -D juniper.cpcase="Bas
root 4678 1 0 Mar27 ? 00:00:29 /pkg1/appl/dlc/101c/bin/_sqlsrv2 -D juniper.serverid=4 -D juniper.dbfile="/pkg1/db/admprod_01" -D juniper.config ="admprod_01.virtualconfig" -D juniper.service="admprod_01.virtualconfig.virtual" -D juniper.propertyfile="" -D juniper.cplog="ISO8859-1" -D juniper.cpcase="Bas\

try to connect a remote 4GL client (which you have said will fail) and see what is in the database log at that timestamp.

# mpro admprod_01 -ld qadadm -S admprod_01 -H troap006lnx
[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi][2012/04/03@16:09:33.975-0400] P-8070 T-0 I BROKER 0: (1154) No servers are available. Try again later.

[FONT=r_ansi][FONT=r_ansi]
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol]R,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] Error [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol],,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,T
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] The server or the system has no more resources. Please contact Progress [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] Technical Support. (748) [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol],,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].[/FONT][/FONT][FONT=r_ansi][FONT=r_ansi] <OK> [/FONT][/FONT][FONT=r_symbol][FONT=r_symbol].
[/FONT][/FONT][FONT=r_symbol][FONT=r_symbol]F,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,G
[/FONT][/FONT][/FONT][/FONT]

 

[TomBascom]

You have spawned 4 SQL servers. That's it. You're out of servers to spawn.

There are no 4GL servers for 4GL users to connect to.

This is the problem with using server type "both". It means that you can start both. But once one has been started it is whatever it is.

It may be that you have never seen this before just because you don't usually start very many remote connections.

You should be able to convince yourself that this is the case by starting a secondary broker on a new port and logging in to it via a remote client.

 

[TomBascom]

I believe my previous answer is correct.

Not to muddy the water or anything but to address the "all I did was add an extent" issue:

Adding extents increases the number of file handles needed. This /could/ potentially impact remote connections because they need file handles for the TCP/IP sockets too. I suppose that there /might/ be some potential for you to have added just enough extents to be right at the threshold for a self-service connection to work but where a remote connection might need just a couple too many. But you would need to wotrk pretty hard at it to have that many files open -- the "ulimit -Hn" on my system is 8192.

 

[Rob Fitzpatrick]

So as it stands you can't run a remote 4GL client because there are no 4GL servers running; the broker must spawn one. All four existing servers are SQL and you can't spawn a fifth (4GL) server for some reason.

As you can see from the info on your existing servers:

04/03/12 Status: Servers By Broker
08:28:34
Sv                             Pend. Cur.  Max.  Port
No Pid   Type  Protocol Logins Users Users Users Num
0  8070  Login TCP      0      0     0     50    12005
1  17751 Auto  TCP      403    0     0     50    1049
2  18487 Auto  TCP      85     0     0     50    1050
3  15234 Auto  TCP      10     0     0     50    1072
4  4678  Auto  TCP      2      0     0     50    1099

There is a fair bit of spread between the SQL servers' assigned port numbers. This is not unexpected on a database server running many DBs, and possibly other application services.

Since you are not specifying the -minport and -maxport broker startup parameters, Progress is using its default Unix range of 1025 to 2000. As I said earlier, when the broker wants to spawn a 4GL server it has to find a port that (a) is not in use and (b) is not defined in /etc/services. If you are running many databases, effectively all re-using the same port range, it's a crap shoot.

Since you are open to the possibility of changing your startup parameters and restarting the DB, I recommend that you first look at /etc/services and find a continguous range of unassigned ports that you can use for this database's servers. Hopefully you have a well-ordered services file with entries kept in numeric order, but there is no guarantee of that. At any rate, find a range you can use. I like to add a comment to the file at this point specifying the ports I'm using, their use, the date, etc., so that no one else gets the idea that this range is available. Note that you don't actually define the ports in the file. You're just "reserving" them with a comment.

You don't need a large range; just enough ports for the number of servers you will want to spawn. Then add "-minport <low port number> -maxport <high port number>" to your broker startup parameters, bounce the database, and connect your SQL clients to spawn those servers. Then check promon to see that you are running 4 or fewer SQL servers. Finally, try to connect a 4GL client and see what happens.

One other thing: before your restart it would be helpful to see the output of the "Status: Servers" screen in promon (R&D | 1 | 3).

 

[dlangschied]

[FONT=r_ansi][FONT=r_ansi]I have the output requested below. I have an idea though. Can I stop one of the SQL brokers and then start my 4GL? I do not have alot of SQL connections that are concurrent. There would only be 5 maximum. So if I stop one SQL broker and initiate a 4GL session would it be able to start it then?

Also, I tried to launch another service and it is telling me that I have reached the maiximum number of users. Is it counting the number of max users for the open brokers ((50 * 4) + 1) to get to 201? If that is the case, then I have an intriguing question: If I start more servers (and increase users, to say 251), but I do not start a 4GL broker for a few days. Will I still have the possiblity that I end up with 5 SQL brokers at some point and still cannot launch my 4GL? We only compile against our production database set every other week or so. Am I all wet here?

04/04/12 Status: Servers
07:12:57
Sv Pend. Cur. Max. Port
No Pid Type Protocol Logins Users Users Users Num
0 8070 Login TCP 0 0 0 50 12005
1 17751 Auto TCP 454 0 0 50 1049
2 18487 Auto TCP 94 0 0 50 1050
3 15234 Auto TCP 12 0 0 50 1072
4 4678 Auto TCP 2 0 0 50 1099

I am sorry to keep buggin you, but having not seen this situation arise, it has me intrigued. And, of course, now that it has happened, I have to explain to higher ups "why".
[/FONT][/FONT]