Question New Server Spec

Cringer

ProgressTalk.com Moderator
Staff member
Been asked to provide a wishlist for a new production server. It's proving quite tricky actually as there are a number of issues that are affecting the performance of the actual DB that would be solved just be upgrading the OS and Progress to 64bit.
Current box is a 24 Core X7460 @ 2.66GHz. 16GB RAM (most of which is useless)
We currently serve a number of databases on here, although the plan is to move all except production DBs and schema holders onto other boxes as part of the migration.
We're currently running 11.2.1 (32b) on Win 2003 R2 (32b), but the plan is to move to 11.4 (64b) on 2008 R2 (64b).
The main production DB is 350GB with around 150-200 physical users and a bunch of AppServers and batch processes.
We currently have a lot of performance issues that could be tuned out but we're restricted by the 32b executables and OS.
I realise that it's hard for folks to comment without seeing the application itself, but if anyone has any pointers I'd really appreciate it. I believe the current production box cost ~£50k so I'm assuming the budget will be similar.
Apparently we're using http://www.nviron.co.uk/ to supply the server. The worry for me is that all the questions they are asking us are SQL oriented. :mad:
So here is your perfect opportunity to be opinionated... :)
Please!
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
Been asked to provide a wishlist for a new production server. It's proving quite tricky actually as there are a number of issues that are affecting the performance of the actual DB that would be solved just be upgrading the OS and Progress to 64bit.
Sounds like a good thing.
Do you know what your current application bottlenecks are?
Which ones do you anticipate will not be resolved just by moving to a new OS and to 64-bit OpenEdge?
What are your availability requirements for this application environment from the business?
Current box is a 24 Core X7460 @ 2.66GHz. 16GB RAM (most of which is useless)
We currently serve a number of databases on here, although the plan is to move all except production DBs and schema holders onto other boxes as part of the migration.
That's good. Production should always be segregated from other workloads.
Why would you have schema-holder DBs on production? Sounds like something you would have on a build server for running compiles.
You don't need or want a ton of cores on a Progress DB server. What you need is a lot of RAM, for shared memory, and lots of fast enterprise storage (think direct-attached SSDs, RAID 10). You need to talk about what you are doing today and what you plan to do on the new box with storage.
We're currently running 11.2.1 (32b) on Win 2003 R2 (32b), but the plan is to move to 11.4 (64b) on 2008 R2 (64b).
Do not provision a new box with a five-year-old OS that will be at end of mainstream support in January of next year. What would IT do then, pay for extended support? Run an unsupported OS? Upgrade the OS on a server to a newer version? All unappealing and unnecessary choices, IMO.
The main production DB is 350GB with around 150-200 physical users and a bunch of AppServers and batch processes.
We currently have a lot of performance issues that could be tuned out but we're restricted by the 32b executables and OS.
Do the AppServers run on the DB server today?
Will they run on the DB server in the new configuration?
Where exactly do the 32-bit dependencies exist in your application?
Is it in the client app(s) that your users run, in the code the AppServers run, in the batch processes (I doubt it), or in some combination of the above?
Where do your clients' applications run? Are they remote, running on their PCs?
What is the state of your network? That's important in a client/server architecture.
Obviously you are moving to a 64-bit OS. You should try very hard to move to a 64-bit OE back end, even if it means having to make architectural changes to your application.
I realise that it's hard for folks to comment without seeing the application itself, but if anyone has any pointers I'd really appreciate it. I believe the current production box cost ~£50k so I'm assuming the budget will be similar.
That's a fair bit of money. But I assume it's for two boxes, not one. This is production after all. So you need a box that is equivalent to (or preferably, identical to) production for DR.
Apparently we're using http://www.nviron.co.uk/ to supply the server. The worry for me is that all the questions they are asking us are SQL oriented. :mad:!
I wouldn't expect them to know or even have heard of Progress. They need to know it's a database workload and you need lots of RAM and lots of very fast storage.
 

Cringer

ProgressTalk.com Moderator
Staff member
Sounds like a good thing.
Do you know what your current application bottlenecks are?
Which ones do you anticipate will not be resolved just by moving to a new OS and to 64-bit OpenEdge?
What are your availability requirements for this application environment from the business?

The biggest bottleneck we have at the moment is Checkpointing and Buffer Flushes. Waiting for scheduled downtime to increase the BI Cluster Size. We'll be able to re-monitor then to establish what other bottlenecks there are specifically, but not having a massive -B is hurting us, and not wanting to implement much at all for -B2 is as well. I've also scheduled in to move the Schema and one small frequently accessed table to -B2 at the next downtime. Will then reassess bottlenecks.
The biggest I know of is storage. We don't have enough concurrent storage to do a DB dump and load so maintenance (we need a structure change desperately) is hard work. Against the storage section of the requirements we have "lots"! :)
Availability-wise we have a commitment for 24/7 availability with planned outages for maintenance. I'm doing the High Availability workshop at the PUG next week so that I can learn how to achieve this requirement.

That's good. Production should always be segregated from other workloads.
Why would you have schema-holder DBs on production? Sounds like something you would have on a build server for running compiles.
You don't need or want a ton of cores on a Progress DB server. What you need is a lot of RAM, for shared memory, and lots of fast enterprise storage (think direct-attached SSDs, RAID 10). You need to talk about what you are doing today and what you plan to do on the new box with storage.

The schema holders are for SQL DataServer. Don't understand that side of things but we push data to Access Dimensions Accounts and apparently we need the schema holders for that. My guess is that a build server would be useful for that too.
Re cores - would you say 24 cores is sufficient, too much?

Do not provision a new box with a five-year-old OS that will be at end of mainstream support in January of next year. What would IT do then, pay for extended support? Run an unsupported OS? Upgrade the OS on a server to a newer version? All unappealing and unnecessary choices, IMO.

I'm not up on OS versions. All our servers (bar production) are on 2008 R2. What OS would you recommend?

Do the AppServers run on the DB server today?
Will they run on the DB server in the new configuration?
Where exactly do the 32-bit dependencies exist in your application?
Is it in the client app(s) that your users run, in the code the AppServers run, in the batch processes (I doubt it), or in some combination of the above?
Where do your clients' applications run? Are they remote, running on their PCs?
What is the state of your network? That's important in a client/server architecture.
Obviously you are moving to a 64-bit OS. You should try very hard to move to a 64-bit OE back end, even if it means having to make architectural changes to your application.

Yes the AppServers run on the DB server. They will likely still run on the DB Server in the new configuration, although if there's good reasons to separate them off then I can ask the question.
The main 32bit dependencies are on the client machines with all manner of ocx etc. Clients will remain 32bit for the foreseeable future. As I understand it a number of the AppServers send emails using Redemption. I gather you can get 64bit Redemption executables, but even so it shouldn't be a hard job to pass back the information to the client to send the emails rather than the AppServer doing it. The current situation is lazy coding IMO.

That's a fair bit of money. But I assume it's for two boxes, not one. This is production after all. So you need a box that is equivalent to (or preferably, identical to) production for DR.

I wouldn't expect them to know or even have heard of Progress. They need to know it's a database workload and you need lots of RAM and lots of very fast storage.

I think the current production box is going to be recommissioned as the DR box. Current DR box is the previous production server minus a few bits that broke in an office move. As you can tell we don't exactly run the most professional of systems. I want to change that.

Thanks for your input! :)
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
The schema holders are for SQL DataServer.
Sorry, I misunderstood. I didn't realize you were running DataServers. Yes, you need those schema-holder DBs.
Re cores - would you say 24 cores is sufficient, too much?
It's more than sufficient, I think. Is it too much? I don't know. I'll defer to others' more informed opinions.
What OS would you recommend?
I would recommend the most modern OS that the back end of your application will support. If you can run on Server 2012 R2, go for it.
The main 32bit dependencies are on the client machines with all manner of ocx etc.
So nothing should stop you from moving your RDBMS, AppServer, batch, and compile licenses (i.e. all your server-side licenses) to 64-bit. The Windows remote clients don't need to move to 64-bit as they connect via TCP. They don't know or care which platform (i.e. 32/64) the DB is on.
The qualification "main dependencies" suggests there is some room for doubt. Obviously, you're going to test all this before you paint yourself into a corner.
I think the current production box is going to be recommissioned as the DR box.
That's a common practice. And in my opinion, it's a bad one.
As you can tell we don't exactly run the most professional of systems. I want to change that.
Then start now. Do it right: spec out a production system and buy two. You will run your production workload on your DR box at some point, even if it is only during your annual DR test. If it can't handle the load then it shouldn't be accepted by the business as the DR box. Remember, this is the box you're moving away from because your applications are so hampered. And if it was purchased in an era when 32-bit server OSes were common, I very much doubt it deserves any place in your production/DR environment.
 

Cringer

ProgressTalk.com Moderator
Staff member
Thanks Rob, also for the DataServer clarification - not an area I know much about tbh. Really helpful advice all round.
 

TheMadDBA

Active Member
Agree about the OS choice, the fast (hopefully local) disks and 64 bit for the database server. A few more things to add...

Make sure the new box or boxes are not NUMA boxes. Get it in writing before you buy anything. They are the devil when it comes to shared memory databases.

As far the 24 cores.. that really depends on what your requirements are. I know of shops where 24 cores would be overkill and some where it wouldn't be close to enough. Trend your current CPU usage and go from there.

One thing to worry about is that performance monitor in older (and maybe current) versions of Windows doesn't track disk IO by default.. you have to turn it on. Make sure that is turned on and trend your disk IO to get sizing requirements for the new hardware. Same for memory.

Trending of your application is critical for sizing the new hardware, everything else is just guessing really.

Finally :) - don't be surprised that a lot of tuning can still happen with 32 bit. The easy answer always seems to be to throw more memory at the problem, but it is quite possible you have network and coding issues that could be making up most of your problems.
 

Cringer

ProgressTalk.com Moderator
Staff member
Thanks Mad. Network issues are possible, although less likely. Coding issues are a definite yes. Trust me I've been banging on about that for a long time now. Trouble is, general consensus is that if the system is running slow it must be a DBA problem not a code problem. I've decided that the best thing to do is to tune up the DB as much as we possibly can, within reason, and then show that performance is still suffering and push it back at the development manager with plenty of evidence. The trouble is, we have a history of letting hackers loose on the code with no controls. It seems to be a common issue with in-house systems as there's no motivation to do it right.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
I have found that the *stat VSTs have been very valuable in exposing bad coding practices.

Get the developer to describe the business process behind the code they've written. Usually they provide the rope to hang themselves. It's something like:
dev: "Here we query table foo using index bar, bracketing on this subset of records."
me (providing stats): "No, your query predicate doesn't match the index components. You're doing a table scan on 50M records. Twice. Here's the proof."
dev:"..."

When this conversation takes place in front of the appropriate audience, there can be incentive provided to improve the code.

But in the absence of good development practices, and someone in management who can tell good code from bad, this is bound to be a tougher battle.
 

TheMadDBA

Active Member
Heh.. been there done that before.

Even with bad code there are some network/appserver settings you can tweak before you move to 64 bit. Like the network message sizes and making sure you have an appropriate number of appservers in the pool and they have enough memory to avoid disk writes for temp-tables,etc.

Tracking the table/index usage can help you identify application issues that even the managers will not be able to ignore... like massive number of reads using suspect indexes or small tables being read at an excessive rate.
That will also help you identify candidates for -B2.

Good luck and welcome to the wonderful world of being a DBA where everything is your fault and you rarely get credit for anything good :D
 

TomBascom

Curmudgeon
I think the current production box is going to be recommissioned as the DR box.

That's a common practice. And in my opinion, it's a bad one.

Rob is being nice.

It is not merely a bad idea. It is a dreadful idea. Such a DR box is less than useless -- it is actively harmful. It misleads the business into thinking that they have a viable strategy when they are, in fact, demonstrating quite clearly that 1) they have no strategy and 2) they do not care about data integrity or performance. Run away. Run very far away.

If "they" feel compelled to somehow reuse the old server as something other than the boat anchor that it surely is then an alternative use could potentially be as a dev or test box. But never production.
 

TomBascom

Curmudgeon
M = N * C

Where M = "mips", N = number of cores, C = clock speed.

For M "mips" of CPU power being applied to a database you are always better off with fewer, faster cores.

Avoid NUMA like the plague. Once you eliminate or minimize IO as a problem CPU bottlenecks will raise their heads. NUMA is the RAID 5 of CPU architectures. Sales weenies love it and will foist it off on your management whether it makes sense for you or not.

NUMA is very nice for certain workloads. Web servers and Exchange servers spring to mind. Databases do not. It works well when there is NO need to coordinate concurrent access to large hunks of shared memory. It is very, very, very bad when you need to have lots of concurrent access to shared memory. I.e. databases.

The problem is that the CPU caches have to coordinate. And it is *very* expensive to leave the CPU die and go across the bus to do that. NUMA architectures deliberately segment the bus and make it even more expensive than it already was. When you toss in operating systems (and virtualization regimes) that institutionalize default settings which deliberately horizontally spread workloads among as many cores as possible you have a database performance train-wreck.

The NUMA nightmare has infested big iron for a long time. HP Superdomes were the first really bad ones and then there were the many SUN flavors ("niagra" and "cool threads" being a couple examples). Uncle Larry is still trying to fix SUN's gear to work reasonably well with his flagship software. IBM does it too but they at least are aware of the downsides and provide guidance to avoid it. Wintel has been less susceptible to it because so many servers are really just dressed up desktops but that is starting to change for the worse.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
I was thinking the same. Although with batch and AppServer clients running on the box, you'd want to look through the code for any obvious platform dependencies, e.g. hard-coded or database-resident file system paths, assumptions of "\" as a path separator, registry access, use of Win32 API functions, some OS-COMMAND statements, etc. etc. None of these are showstoppers but they may require some work and testing to ensure a seamless transition.

Of course, you also need someone who knows something about the OS. Even the best OS, in the hands of an ignorant sysadmin, will ultimately lead to problems. But please don't read that as an argument against switching a DB server from Windows to Linux. :)
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
Re: NUMA (sorry, slightly off-topic)

I'll confess my ignorance. I believe I understand the theory of what NUMA is and why it carries a performance penalty. But having looked into it a little bit it is not clear to me how to go out in the market and choose hardware to avoid it. Are certain processor architectures known to be NUMA in all cases? Are there others that are guaranteed not to be NUMA?

Or is it dependent not on CPU architecture but on the system builder/mainboard design, e.g. how many physical CPUs there are and what interconnect is used to connect them to system memory?

Also: OS support. I know Windows has been NUMA-aware for a few releases now. I know less about the internals of other OSes. Are there any modern OSes that are not NUMA-aware?
 

TheMadDBA

Active Member
lol... nevermind that. I just reread your question.

Only real way is to pin down the vendor on the specifics. Although it does seem much more common in Intel based machines and HP.
 

Cringer

ProgressTalk.com Moderator
Staff member
lol... nevermind that. I just reread your question.

Only real way is to pin down the vendor on the specifics. Although it does seem much more common in Intel based machines and HP.
I think we're going to be going with a Dell as the vendor is Nviron and they're Dell partners.
 

Cringer

ProgressTalk.com Moderator
Staff member
lol he might take the 'dummy' reference rather than the 'Linux' reference... :/
 

TomBascom

Curmudgeon
If there are lots of cores it is probably NUMA.

If the CPUs come on discrete boards it is probably NUMA.

If the amount of memory is related to the number of CPUs it is probably NUMA.
 
Top