Hardware requirements

[Blakey43]

Hi,

We are running an application based on Progress 9.10E SP4
Its sitting on a Windows 2003 R2 64bit server. This Dell server has a mirrored pair of disks for the OS, and a RAID 10 array of 8 disks for the Progress installation and databases. There's 8GB RAM and a pair of Xeon E5410s running at 2.33 GHz. It performs adequately apart from when its doing an online backup, and then users complain.
We are planning an upgrade of database and application, and moving to new hardware at the same time. There will be no change in the number of users, or core functionality, it's just an upgrade. We're planning to go to Progress 10.2B, SP5, on a VMware virtual machine. We'll possibly go with an iSCSI SAN solution, using a Dell R720 host.

We've done a Dell DPACK analysis of the current system and got figures from a 7 day period for iops, processor use, etc. Based on these figures the idea is to specify the hardware to supply the requirements. Trouble is, we can't analyse the new system, because it's not in yet.
My question is, in general, will the hardware requirements of the new system be similar to the old? Is my analysis of the old system's requirements valid for the new system? Does Progress v10 hammer the hardware any more than v9?

Thanks in advance,
Steve

 

[Cringer]

Personally I would seriously consider going to 11.1 whilst you're upgrading anyway. Yes the upgrade will probably take longer with 2 conversions, but you'll be running the latest version and be able to take advantage of all the new features.

I'll let others advise you on hardware as that's not my bag.

 

[Blakey43]

Hi Cringer,
I would certainly upgrade to the latest version if I could. Unfortunately that's outside of my control, and the application provider is working with 10.2.
Cheers
Steve

 

[Cringer]

Do you get shipped compiled code, or do you have the opportunity to compile your own? If so, there's nothing tying you to 10.2. It would certainly be worth putting pressure on your vendor to move on ASAP. Also, have you consulted your vendor for hardware advice?

 

[Rob Fitzpatrick]

If you are going to use 10.2B, I would strongly suggest SP06 or SP07, both for the bug fixes and for the performance improvements made possible by new startup parameters, both for index rebuild and for general application performance.

Part of the problem with VM vendor tools that analyze your current system to spec out a VM is that it's a chicken-and-egg problem to some extent. If you are currently hardware-constrained e.g. low RAM, then you may have an under-optimized database where the parameters are intentionally set lower than is optimal, e.g. -B, -aibufs, -bibufs, -Mn/-Mpb, -Ma, etc. Thus the vendor's tool tells you that you don't need much RAM because you aren't using much. Tools are nice, but they don't tell the whole story. They don't tell you about the resources you would use if you had the chance.

You mentioned a RAID 10 array for your database. Ideally you should have more than one array of physical disks for your database. Your after-image extents should always be kept separate from your data storage areas. If they are together on the same disks, a failure of the array means you lose both your data and your data recovery mechanism. If you are moving to a SAN, put your BI file on a separate LUN as well.

You mentioned application impact when you back up your database. Does that mean you have users working around the clock or does it mean that you're doing online backups during the day? And when you do this, are you writing the backup file to the same array as the database?

 

[Blakey43]

Do you get shipped compiled code, or do you have the opportunity to compile your own? If so, there's nothing tying you to 10.2. It would certainly be worth putting pressure on your vendor to move on ASAP. Also, have you consulted your vendor for hardware advice?

Hi,
I think we have the code on site, but we don't have the skills to compile it. I will ask the app vendor directly why we're not using v11, but yes, I have asked for support and advice re hardware, and am following that direction in that he has advised VMware and a SAN.
Thanks
Steve

 

[Blakey43]

If you are going to use 10.2B, I would strongly suggest SP06 or SP07, both for the bug fixes and for the performance improvements made possible by new startup parameters, both for index rebuild and for general application performance.

Part of the problem with VM vendor tools that analyze your current system to spec out a VM is that it's a chicken-and-egg problem to some extent. If you are currently hardware-constrained e.g. low RAM, then you may have an under-optimized database where the parameters are intentionally set lower than is optimal, e.g. -B, -aibufs, -bibufs, -Mn/-Mpb, -Ma, etc. Thus the vendor's tool tells you that you don't need much RAM because you aren't using much. Tools are nice, but they don't tell the whole story. They don't tell you about the resources you would use if you had the chance.

You mentioned a RAID 10 array for your database. Ideally you should have more than one array of physical disks for your database. Your after-image extents should always be kept separate from your data storage areas. If they are together on the same disks, a failure of the array means you lose both your data and your data recovery mechanism. If you are moving to a SAN, put your BI file on a separate LUN as well.

You mentioned application impact when you back up your database. Does that mean you have users working around the clock or does it mean that you're doing online backups during the day? And when you do this, are you writing the backup file to the same array as the database?

Hi,
I will ask about SP 6 & 7.
I understand what you're saying about resources, but I thought what you were going to say is that the vendor tool would be biased towards an analysis which recommended more powerful hardware! The main thing this analysis has shown is that the hardware iops capability is way below what's needed. The iops requirement currently is massive and so the hardware solution has been designed to provide large iops - around 8000. What I'm concerned about is that on v10, iops requirement could be even higher that this!
I realise that another approach to this problem would be to reduce the iops requirement. I get what you're saying about other arrays, and yes, the workers are here 24/7 and the online backup is a full one at 1am and a differential at 1pm. It does go to the same array as the database. But most of the iops are read events, so if it went elsewhere (say to a network drive) the iops would still be high.
Should we have AI files, data and BI files on three separate LUNs?
Thanks
Steve

 

[Rob Fitzpatrick]

Hi,
I will ask about SP 6 & 7.
I understand what you're saying about resources, but I thought what you were going to say is that the vendor tool would be biased towards an analysis which recommended more powerful hardware!

I'm not telling you whether the results are good or bad. I'm just pointing out that they are a programmatic analysis done with an algorithm you can't inspect that has no knowledge of OpenEdge databases. Take them with a few grains of salt.

The main thing this analysis has shown is that the hardware iops capability is way below what's needed. The iops requirement currently is massive and so the hardware solution has been designed to provide large iops - around 8000. What I'm concerned about is that on v10, iops requirement could be even higher that this!

Why do you think you will have higher I/O requirements in v10? It has a more efficient storage engine than v9 and if you move your DB to Type II storage, as you should, then physical I/O for the same logical I/O should decrease, given the more efficient layout of the data (asocial blocks and clusters).

I realise that another approach to this problem would be to reduce the iops requirement. I get what you're saying about other arrays, and yes, the workers are here 24/7 and the online backup is a full one at 1am and a differential at 1pm.

If you are doing daily full backups and you are using after-imaging, why are you doing a daily differential backup? That seems unnecessary.

It does go to the same array as the database. But most of the iops are read events, so if it went elsewhere (say to a network drive) the iops would still be high.

What I am saying is that your backups should be more efficient if you are reading from one array and writing to another, rather than doing reads and writes in the same place and cutting your I/O bandwidth in half. Write your backup locally, then when it's done you can copy it off-server and off-site.

Should we have AI files, data and BI files on three separate LUNs?

If you can, sure.

 

[Blakey43]

i'm not telling you whether the results are good or bad. I'm just pointing out that they are a programmatic analysis done with an algorithm you can't inspect that has no knowledge of openedge databases. Take them with a few grains of salt.

OK, I understand. But they did show that we have a high IOPs requirement, during backup times, and that did fit in with user experience, leading to the conclusion that the hardware isn't up to the job. That may not be the correct conclusion, because with careful examination and reconfiguration, performance could be improved on the same hardware, right?.

why do you think you will have higher i/o requirements in v10? It has a more efficient storage engine than v9 and if you move your db to type ii storage, as you should, then physical i/o for the same logical i/o should decrease, given the more efficient layout of the data (asocial blocks and clusters).

Well, that's exactly what I wanted to hear! I didn't think we would have higher i/o requirements, I was in the dark about such things.

if you are doing daily full backups and you are using after-imaging, why are you doing a daily differential backup? That seems unnecessary.

The application vendor has recommended it, and we'd need their support in a recovery event.

what i am saying is that your backups should be more efficient if you are reading from one array and writing to another, rather than doing reads and writes in the same place and cutting your i/o bandwidth in half. Write your backup locally, then when it's done you can copy it off-server and off-site.

I'll aim to set up the SAN with this in mind.

Thanks for your help.

 

[Rob Fitzpatrick]

The application vendor has recommended it, and we'd need their support in a recovery event.

Ask them for a detailed explanation of why you need to add the I/O overhead of incremental backups to your system when you already have daily backups and after-imaging. The assurance of "their support" may be somewhat reassuring to you but you'll be much better off if you can rely on yourself. In other words, don't wait for a disaster to come up with a recovery plan. It should already be documented and tested.

Once that's done, what does the plan say you're going to do with the incremental backups? I don't think you need them. If you have a good full backup and every AI file that has been written since that backup, then you can recover up to the last transaction completed in the last AI extent.

Do you restore your backups elsewhere? Do you roll forward your AI extents onto the restored DB? Do you have scripts for managing your AI files (transfer, roll forward, archiving, etc.)?

 

[Blakey43]

Ask them for a detailed explanation of why you need to add the I/O overhead of incremental backups to your system when you already have daily backups and after-imaging. The assurance of "their support" may be somewhat reassuring to you but you'll be much better off if you can rely on yourself. In other words, don't wait for a disaster to come up with a recovery plan. It should already be documented and tested.

I will ask the question. We are also using Fathom replication on this system, so disaster recovery would involve failing over to the target server.

Once that's done, what does the plan say you're going to do with the incremental backups? I don't think you need them. If you have a good full backup and every AI file that has been written since that backup, then you can recover up to the last transaction completed in the last AI extent.

Do you restore your backups elsewhere? Do you roll forward your AI extents onto the restored DB? Do you have scripts for managing your AI files (transfer, roll forward, archiving, etc.)?

We do restore the backups to another server occasionally which is then used for test purposes. AI files are archived off every 8 hours to a different server, but sorry to say I have had a look at the program that runs every 8 hours but can't fully understand what it's doing. All I know is that AI files appear on the other server every eight hours. We also run the same program manually every month just before we reboot the server.
In terms of recovering and restoring, I'm much more comfortable with MS SQL! I'm going to have to update my "skills" for v10 and if we go to v11 I should say probably more so.
Thanks for your help,
Steve

 

[TomBascom]

8 hours is a very long window in which you could lose data. You should be transferring those ai files *much* more frequently. Every 15 minutes is "typical".

 

[Cringer]

That's true unless they only make changes every 8 hours or so in which case it's probably sufficient

 

[RealHeavyDude]

Please bear with me as I step in, but:

Very rarely have I seen a Progress application partner to be able to assist their customers in any form to create a disaster recovery strategy that covers the requirements of the customers. In fact, in most cases where I was involved, they were telling their customers - and here I am inclined to use strong words - just b**ls**t. Most of them, even large ones, have no understanding of the database administration and not only that they don't advice their customers, they even give bad advice. I do understand that the intention of these application Partners of Progress is to sell their application in the first place, but some of their advices is just downright ridiculous.

You must understand that most of these partners might have had some knowledgeable person long time ago in the past and the stuff they created still ends up on the customers database servers 20 years later.

Just some pointers (questions I got asked by application partners):

• With OE10 is the after image finally usable?
• Now that we have 2008 does Progress finally provide a backup utility?
• In 2010, can a Progress database be backed up?

I could continue all day ...

Therefore, if there is any value in the data in your Progress database, take an investment into the future and make sure the processes is adapted to your needs. I can only second that 8 hours is a very long time and, while it might be great fun when you have to tell you users that they have to reproduce what they have done the last 8 hours, I would think that they won't like it - not one bit.

Heavy Regards, RealHeavyDude.