BROKER: Could not spawn a server. (1890)

#1
HI All,

Today we received below application connectivity error.
Broker : Could not spawn a server. (1890).

For 5 mins users are able to connect to application after few mins they facing the same.

Can i know reason during which situation we will face this error.Please provide your inputs.

I checked in Knowledge base we need to increase -Mn parameter values.
But we have good enough -Mn parameter value and is set to 35.

And also i cross checked in database log file i not found any error.

I checked -in Promon > R&D > servers.

09/11/18 Status: Servers

11:38:40 Sv Pend. Cur. Max. Port

No Pid Type Protocol Logins Users Users Users Num

31 572330 Auto tcp 10 0 4 15 13425

32 431622 Auto tcp 10 0 5 15 13430

33 242904 Auto tcp 7 0 5 15 13431

34 362192 Auto tcp 7 0 5 15 13436

35 204204 Auto tcp 8 0 5 15 13440


Progress version : 9.1e
OS : AIX 5.3
 

Attachments

Rob Fitzpatrick

ProgressTalk.com Sponsor
#2
Can i know reason during which situation we will face this error.Please provide your inputs.
Client receives error message (1890) on attempted connection to a running database

A connection broker is attempting to spawn a server in response to a connection request from a client. It can't do that because it has run out of eligible unused ports with the range specified for -minport and -maxport.

Please post the complete startup parameters of your primary database broker and any secondary connection brokers.

I checked in Knowledge base we need to increase -Mn parameter values.
This is not necessarily true.
 
#3
-B 70000 # Number of (8K) Blocks in database buffer
-bibufs 200 # Number of before-image buffers
-aibufs 300 # Number of after-image buffers
#-Mi 2 # Min processes on a client server
-Ma 15 # Max number of REMOTE clients per db server
-Mn 35 # Max number of REMOTE client servers
-Mxs 32768 # Shared memory overflow size (override)
-spin 15000 # Number of spin lock retries
-L 2048000 # Lock Table entries
-minport 13100 # Min Port Number
-maxport 14999 # Max Port Number
-spin 1000 # Spin lock tries
-bibufs 100 # Number of before-image buffers
-aibufs 50 # Number of after-image buffers
-semsets 256 # Dan Foreman's recommendation

database log file


11:50:22 AIW 42: Started. (2518)
11:50:23 BROKER 0: Progress OpenEdge Release 9.1E on AIX. (4234)
11:50:23 BROKER 0: Server started by pgresdba on batch. (4281)
11:50:23 BROKER 0: Started using pid: -28712. (6574)
11:50:23 BROKER 0: Physical Database Name (-db): /progdata/prd/slcdb1/prdslc. (4235)
11:50:23 BROKER 0: Database Type (-dt): PROGRESS. (4236)
11:50:23 BROKER 0: Force Access (-F): Not Enabled. (4237)
11:50:23 BROKER 0: Direct I/O (-directio): Not Enabled. (4238)
11:50:23 BROKER 0: Number of Database Buffers (-B): 70000. (4239)
11:50:23 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
11:50:23 BROKER 0: Excess Shared Memory Size (-Mxs): 33554432. (4240)
11:50:23 BROKER 0: The shared memory segment is not locked in memory. (10014)
11:50:23 BROKER 0: Current Size of Lock Table (-L): 2048000. (4241)
11:50:23 BROKER 0: Hash Table Entries (-hash): 18289. (4242)
11:50:23 BROKER 0: Current Spin Lock Tries (-spin): 15000. (4243)
11:50:23 BROKER 0: Number of Semaphore Sets (-semsets): 256. (6526)
11:50:23 BROKER 0: Crash Recovery (-i): Enabled. (4244)
11:50:23 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
11:50:23 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
11:50:23 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
11:50:23 BROKER 0: Before-Image Truncate Interval (-G): 60. (4249)
11:50:23 BROKER 0: Before-Image Cluster Size: 4194304. (4250)
11:50:23 BROKER 0: Before-Image Block Size: 8192. (4251)
11:50:23 BROKER 0: Number of Before-Image Buffers (-bibufs): 200. (4252)
11:50:23 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
11:50:23 BROKER 0: BI File Threshold Stall (-bistall): Disabled. (6552)
11:50:23 BROKER 0: After-Image Stall (-aistall): Not Enabled. (4254)
11:50:23 BROKER 0: After-Image Block Size: 8192. (4255)
11:50:23 BROKER 0: Number of After-Image Buffers (-aibufs): 300. (4256)
11:50:23 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
11:50:23 BROKER 0: Maximum Number of Clients Per Server (-Ma): 15. (4257)
11:50:23 BROKER 0: Maximum Number of Servers (-Mn): 36. (4258)
11:50:23 BROKER 0: Minimum Clients Per Server (-Mi): 5. (4259)
11:50:23 BROKER 0: Maximum Number of Users (-n): 1051. (4260)
11:50:23 BROKER 0: Host Name (-H): p5lp5. (4261)
11:50:23 BROKER 0: Service Name (-S): prdslc. (4262)
11:50:23 BROKER 0: Network Type (-N): tcp. (4263)
11:50:23 BROKER 0: Character Set (-cpinternal): iso8859-1. (4264)
11:50:23 BROKER 0: Parameter File: /prd/dba/etc/pf/db_prdslc.pf. (4282)
11:50:23 BROKER 0: Minimum Port for Auto Servers (-minport): 13100. (5648)
11:50:23 BROKER 0: Maximum Port for Auto Servers (-maxport): 14999. (5649)
11:50:23 BROKER 0: This broker supports both 4GL and SQL server groups. (8865)
11:50:23 BROKER 0: Large database file access has been enabled. (9426)
 
#4
proserve /progdata/prd/slcdb1/prdslc -m3 -Mi 3 -Ma 5 -Mn 4 -Mpb 4 -S prdslcoi -ServerType SQL
proserve /progdata/prd/slcbard1/prdslcb -m3 -Mi 3 -Ma 5 -Mn 4 -Mpb 4 -S prdslcboi -ServerType SQL
 
#5
Progress 9.1e (and AIX 5.3) is ancient, obsolete and unsupported. You should upgrade immediately. It is irresponsible to continue to run these releases. You data is at risk.
 
#6
You show -m3 broker startup for 2 different databases. Which one matches your .lg extract? (Presumably prdslc but it is good to check...)

Are there any other brokers starting? Like maybe a 4gl broker? You did apparently start a "both" broker with the db itself.

-Mn should not be on the command line with -m3. Your overall -Mn is 36 (the .lg file says that). If the "both" broker is using more than 32 of those then you could run out before the SQL broker can use 4.
 
#7
HI Tom,

we have only SQL broker for connecting SQL connections.

prdslc.pf file


-B 70000 # Number of (8K) Blocks in database buffer
-bibufs 200 # Number of before-image buffers
-aibufs 300 # Number of after-image buffers
#-Mi 2 # Min processes on a client server
-Ma 15 # Max number of REMOTE clients per db server
-Mn 35 # Max number of REMOTE client servers
-Mxs 32768 # Shared memory overflow size (override)
-spin 15000 # Number of spin lock retries
-L 2048000 # Lock Table entries
-minport 13100 # Min Port Number
-maxport 14999 # Max Port Number
-spin 1000 # Spin lock tries
-bibufs 100 # Number of before-image buffers
-aibufs 50 # Number of after-image buffers
-semsets 256 # Dan Foreman's recommendation

Promon info..

Code:
09/11/18        Status: Startup Parameters
14:57:28

Maximum clients:                   1051
Maximum servers:                   35
Maximum clients per server:        0
Lock table size:                   2048000 entries
Database buffers:                  70000 (560000 kb)
APW queue check time:              100 milliseconds
APW scan time:                     1 seconds
APW buffers to scan:               116
APW max writes / scan:             25
Spinlock tries before timeout:     15000
Before-image buffers:              200 (1600 kb)
Before-image file name:            -
After-image file name:             -
After-image buffers:               300 (2400 kb)


And also we not found any pending users in R&D> servers

09/11/18        Status: Servers
14:57:56

Sv                                     Pend.   Cur.   Max.   Port
No    Pid  Type       Protocol Logins  Users  Users  Users    Num

  0 102360 Login      tcp         346      0      0     15  13017
  1 497924 Auto       tcp          14      0      5     15  13446
  2 105162 Auto       tcp           6      0      5     15  13447
  3 464340 Auto       tcp           5      0      5     15  13448
  4 418184 Login      TCP           0      0      0      5 -10535
  5 707766 Auto       tcp          10      0      5     15  13449
  6  14228 Auto       tcp          10      0      5     15  13450
  7  61066 Auto       tcp           7      0      5     15  13451
  8 486788 Auto       tcp           6      0      5     15  13452
  9 456470 Auto       TCP           8      0      0      5   1025
10 567540 Auto       tcp           6      0      5     15  13453
11 565514 Auto       tcp          12      0      5     15  13454
12 505310 Auto       tcp          11      0      5     15  13455
13 450196 Auto       tcp           7      0      5     15  13456
14 687526 Auto       tcp           7      0      5     15  13457
15 233926 Auto       tcp          13      0      5     15  13458
16 499892 Auto       tcp          10      0      5     15  13460
17 509182 Auto       tcp           9      0      5     15  13461
18 358322 Auto       tcp           9      0      5     15  13464
19 131562 Auto       tcp          19      0      5     15  13465
20  58274 Auto       tcp           7      0      5     15  13466
21 567038 Auto       tcp          12      0      5     15  13467
22 673642 Auto       tcp          11      0      5     15  13468
23 409578 Auto       tcp           7      0      5     15  13470
24 404054 Auto       tcp          11      0      5     15  13471
25 650668 Auto       tcp           8      0      5     15  13472
26   5642 Auto       tcp           7      0      5     15  13474
27 383292 Auto       tcp           9      0      5     15  13477
28 266354 Auto       tcp          19      0      5     15  13478
29 298332 Auto       tcp          40      0      5     15  13479
30 267870 Auto       tcp          18      0      5     15  13481
31 441898 Auto       tcp          14      0      5     15  13482
32 548208 Auto       tcp          18      0      5     15  13483
33 496718 Auto       tcp           7      0      1     15  13485
34      0 Inactive                 0      0      0      0      0
35      0 Inactive                 0      0      0      0      0
 
Last edited by a moderator:

Rob Fitzpatrick

ProgressTalk.com Sponsor
#10
11:50:23 BROKER 0: Minimum Clients Per Server (-Mi): 5. (4259)

34 0 Inactive 0 0 0 0 0
35 0 Inactive 0 0 0 0 0

11:50:23 BROKER 0: Minimum Port for Auto Servers (-minport): 13100. (5648)
11:50:23 BROKER 0: Maximum Port for Auto Servers (-maxport): 14999. (5649)

When all servers of a broker have reached their minimum user count (-Mi) and a client requests a connection, and the broker has not yet spawned its maximum number of servers, it will attempt to spawn another one.

These are your server port numbers in use, according to promon:
Code:
1025

13446
13447
13448
13449
13450
13451
13452
13453
13454
13455
13456
13457
13458

13460
13461

13464
13465
13466
13467
13468

13470
13471
13472

13474

13477
13478
13479

13481
13482
13483

13485
The gaps indicate non-sequential port numbers.

Clearly:
  • You haven't specified -minport/-maxport on the SQL broker, as there is a port 1025 in use, I assume by a SQL server.
  • Either there are multiple database brokers running on this machine with overlapping -minport/-maxport ranges (likely) or there is a lot of cruft in /etc/services within the established -minport/-maxport ranges.
I cannot stress this enough: change your port configuration. For all brokers on all databases. With planning and proper configuration based on user requirements and system resources, this problem is completely avoidable.

Re-using a very large -minport/-maxport range across multiple brokers and databases on a single server is really easy and doesn't require any planning and "just works". Until it doesn't. At that point, unraveling exactly why you got this error is a huge pain. So don't do it.

Do some planning. If a 4GL broker will spawn 40 servers then it needs no more than 40 ports in its -minport/-maxport range. Specifically, select ports that are not defined, and never will be, in /etc/services. Make a habit of adding that range as a comment in /etc/services, keeping it in numerical order. That way, someone coming along after the fact to look at the file for an available port or port range for a broker or servers won't re-use that range. Add a comment that tells people the DB name and broker type. That way you know what to remove from the file when it comes time to decommission the database.

Use a separate range for the servers of every broker in every database, every time. If you maintain that discipline, this problem will become a thing of the past. It may seem like more work but it isn't difficult and in the long run it will save you time.
 
Top