VMware vmotion on a running DB?

[Rob Fitzpatrick]

Hi folks,

I have a client that uses VMware ESX (4.1, I think) and is asking whether they can use vmotion to move a VM from one physical host to another while the DB in it is running.

I have found very little info about this topic online, and I don't have VMware experience. I checked with PSC support and they said they have tested this and it works, and that it shouldn't even require a proquiet during the move; the DB should be oblivious to it.

Personally I wouldn't take this approach. I get the sense that part of the appeal for the client is being able to do server maintenance during business hours. If it were up to me I would do it after hours after taking down the database, but maybe I'm just being old-fashioned (and risk-averse).

I would be interested to know whether anyone has hands-on experience and success with this scenario in a production environment. I would put a lot more faith in real-world experience than a test done in a lab. Thanks in advance.

 

[TomBascom]

I've never seen it tried in real life with a Progress database.

I have, however, observed IT people running other databases get the bright idea that this approach would allow them to perform maintenance during business hours.

I've never seen anyone who wanted to keep their job try it twice

I think it might be fine if you have some trivial no-load toy application running.

 

[Rob Fitzpatrick]

Thanks for your input Tom. This is a prime example of why I have never believed the old saw that "the customer is always right".

P.S.: I'm looking forward to your Revolution talk on ABP. Cheers.

 

[LynnReas]

We have hundreds of 10.1B databases running on VMWare ESX & ESXi. When it comes to VMotion the Virtual Windows server doesn't see the change. Even Databases presented with 4 cpu's running 30 percent have VMotion successfully. Fact I have never seen VMotion failed.
If the webspeed service is another server it doesn't lose the connection either.
If you ping the DB server (or any server during a vmotion) you might see a ping dropped as a ARP request to the network has to be performed to find the new Ethernet route for the physical location the VM was moved to.

 

[Rob Fitzpatrick]

Thanks Lynn. I'm still not sold on the idea myself, but it is a relief to hear a real-world experience, as my client is going ahead with this model.

Do you have anything to relate on the time it takes to vmotion a VM (as a function of VM and memory size), or the impact on transaction throughput during the transition? Thanks.

 

[TomBascom]

Just to be clear -- I'm not saying that it will fail. In fact, I'm sure that it "works". I am saying that the idea that you can make such a transition during business hours without users noticing (and complaining) is "unlikely". And that people who I have seen do it with non-Progress databases regretted it.

In general I, personally, frown on administrators putting their own convenience before the needs of their users (the suggested motivation above). Grasping at technologies like this without testing the performance impact (which seems to be the case here) is not a responsible way to go about things.

If real world load tests show that it's ok then it would be fine. So far I've not seen or heard of such a test.

 

[LynnReas]

Actually one of my tasks is to continually determine best practices with our databases / systems. My typical day doesn't even involve Progress software as I'm working on Hardware / Networking or Software deployment. The best part is getting to work with thousands of schools all across this great nation. However for at least 1/3 of the year I'm working on making OpenEdge work better. With that come a huge assortment of servers with some SAN units being Fiber Channel others are iSCSI.
Our programming staff has created awesome performance diagnostic tools that count records, create temp tables and much more to assist our IT staff in determining slow hardware vs problematic programming.
Back 2008 I started using VMWare for testing in house. Using old servers (I'm talking 5 - 7 years old) we put together a virtual platform. Using couple of these old server to provide iSCSI with Open Filer (Open Source) and few with ESX Servers. If you ever get the chance to check out ESX /ESXi head to the performance tab and then advance options. This is the Consumer Reports of anything running on the platform. Once you learn these tools it easy to pin point bad code or determine hardware capabilities like maximum Kbps reads.
While I have spreadsheet after spreadsheet of different aspects here is a little taste. One of the tests I like is the good old restore of a Progress 10.x database into a fixed size Type II extents of 20GB for each of the 32, 64, 128, 256 and Lobs. Now run that against the old MSA 1000 Fiber Channel SAN. When prorest creates the initial 100GB space we have lots of Disk Writes. Windows 2003 (32bit): 186,000 Kbps, Windows 2008R2 (Yes I know there is only 64bit): 225,000 Kbps, Linux SLES 10SP2 (64bit) LVM EXT3: 203,000 Kbps.
Now using a better SAN like the EqualLogic PS4000 same test Windows 2008R2: 305,000 Kbps. Take a Dell R510 (BTW the R stands for Rocket) and use the local SAS 10,000 RPM drives: Windows 2008R2 528,000 Kbps.
Now with the real world testing. Having the a process to create as many reads (without an index and have seen as many as 45,000 per second) and writes then VMotion the database to another physical host. Even with a Virtual Windows 2008R2 server at 20GB doesn't slow down the transactions. Using OpenEdge Management we monitor the performance / responsiveness.
Think about it, most of the time our Servers are spinning doing nothing (spin pun:>)
I'm surely hoping this wasn't enough for you Tom. As a few paragraphs mean nothing without ones own testing. If you are in Boston during Revolution please look me up and I will be more then happy to show you production VMotion and the numbers behind it.

Lynn

 

[TomBascom]

Yes, I'll be there. I look forward to meeting you!

 

[Rob Fitzpatrick]

First round is on me.

 

[paulpetersen412]

Hi Lynn,
We are running over 200 Progress OpenEdge 10.2A03 DBs across 7 VMWare ESX 4.1 Servers with Windows Server 2008 R2. We have successfully used VMotion on running Progress DBs when there has been capacity issues.
We are looking to move up to VMWare 5.0. We are NOT planning to use Vmotion for this exercise.
Has you upgraded to ESX 5.0 yet and have you had any issues?
Thanks
Paul

 

[LynnReas]

Paul,
Yes on ESXi5 and found no issues after 10 months. In fact have seen that VMware File System 5 is a huge improvement from version 3. Unfortunately you have to move all the VM's to another location and reformat the datastore. Memory snapshots that took 4 minutes before take 40 seconds now. The I/O performance is really outstanding.

Lynn