Sun T2000 And Open Edge Index Rebuild

[Robert Jackson]

Environment
========================
SunFire T2000 SPARC (1 CPU, 8 core, 4 threads/core, 32Gb RAM, 2x73Gb (RAID-1) HD's For OS, 2x146Gb (RAID-1) HD's For Open Edge, log files, scripts etc)
StorageTek 2540 RAID Array (12x300Gb HD's)
Solaris 10
Progress Open Edge 10.1B03 (Enterprise RDBMS)

We're currently moving our production databases from an Intel/Red Hat Linux platform to Sun T2000 servers (SPARC Processor)/Solaris 10 platform.

The issue we're having at the moment is with the performance of an index rebuild. Even if we supply the multi threaded options to an index rebuild on our Solaris platform, it runs twice as long as it does on our Intel/Red Hat platform.

Speaking with Sun (and supply them with hardware diagnostic information) they are saying the problem lies in the fact the index rebuild is being run single-threaded. They see one thread doing all the work while the other 31 are just sitting around doing nothing.

Has anyone else had this issue or could point me in the direction of a solution.

TIA.

 

[TomBascom]

You should talk to Progress tech support. They need to hear about these sorts of things.

IMHO it isn't an uncommon problem - but they need a few "smoking guns" to help them figure out why and get motivated to fix it...

Beyond that - what is the full configuration (what RAID level is the StoarageTek? How big is the db? How many tables? How many indexes?) and exactly what parameters are you using for the index rebuild, how is the srt file defined and so on and so forth?

 

[cj_brandt]

In my experience, when idxbuild is scanning the area, only 1 thread is used. After the area is scanned and the utility is actually building the index, that is when the multiple threads are used.

 

[Robert Jackson]

Many thanks for the reply Tom. The configuration is as follows:

The server has 1 CPU (but is multi core/threaded). It is a SunFire T2000 (using the UltraSPARC T1 processor) with 1 CPU, 8 cores, 4 threads pre core giving a mex of 32 threads and has 32Gb RAM. There are 2 pairs of mirrors internally on the server. The O/S (Solaris 10) resides on the first mirrored pair (2x73Gb disks). The second mirrored pair holds Progress and any custom scripts etc and is 2x146Gb disks.

The database is held on a Sun StorageTek 2540 (fully populated) with 12x300Gb disks. The StorageTek 2540 has dual 4Gb/s fibre channel links to the server and has a split RAID configuration as follows:

RAID-1 (disk 1+2 for BI files)
RAID-10 (disk 3->10 for DB files/extents)
RAID-1 (disk 11+12 for AI files)

Each "RAID container" outline above is recognised as a separate volume under CAMS.

When running an idxbuild we also ran a "guds" diagnostic program supplied by Sun to gather various system information. When they had analyzed this, they found that only 1 of the 32 threads was doing any work for the dbutil processes. All others were idle (this was even when we issued the multi- threaded options for the idxbuild). Exactly the same processing time occurs if we do not supply the multi-threaded options to idxbuild.

Progress have since asked for the database.lg file to have a look at. Although I find it a bit strange that according to the log file, the idxbuild process is indeed multi-threaded!

TIA.

 

[Robert Jackson]

Sorry Tom,

In answer to your other questions regarding database sizing:

The database is 31Gb, has 274 tables and 1438 indexes.

TIA.

 

[TomBascom]

Tech support should be asking for more than the db log file. You may need to escalate the issue...

 

[Robert Jackson]

Hi Tom,

Again many thanks for your speedy response. Since my last post I've emailed the diagnostic program Sun asked me to run along with the output generated by it. They were making noises about the fact the box only had 1 CPU, but we await (as always, with a sharp intake of breath) the response.

TIA.

 

[Robert Jackson]

Guys,

Does anyone know of T2000's being used as OE 10.1x DB servers and if there is poor performance with idxbuild?

I'm getting nowhere at the moment with Progress Tech Support. I'm ready to drop these servers into the next available trash can I can find!!


TIA.

 

[TomBascom]

I wouldn't necessarily blame the servers... nor would I exonerate them

Aside from the observation that your tools say that idxbuild is single-threaded what can you tell us?

You mentioned, for instance, that you're moving from RH to Solaris. Why?

You also mentioned that idxbuild is taking twice as long on Solaris. How long is twice as long?

Also -- presumably you're comparing apples to apples -- you've recently done a dump & load of the same db on RH which is where those numbers come from?

Same version of Progress?

Were you able to gather a similar level of detailed information about how well the RH idxbuild utilizes resources?

 

[Robert Jackson]

Hi Tom,

Again my sincere thanks for the reply. I'll attempt to answer as best I can:

1. We were finding that the Dell hardware we are currently running with was/is giving us reliability issues - to the extent that we have lost data in the past. Therefore it was decided (especially for some of our bigger customers) we should move to a more reliable/robust platform. Sun Solaris seemed, for us, to be the perfect choice.

2. The current Dell kit: PE 6850, Quad Intel Xeon processor, 16 Gb RAM, 2x146Gb drives mirrored running RedHat Linux AS 2.1A. Connected to the server is a PV220S fully populated with 14x146Gb drives configured with RAID-1 for BI files, RAID-10 for database extents and RAID-1 for AI files. This kit is running Progress v9.1D08. For a 31Gb database with around 274 tables and 1438 indexes, it takes roughly 30 mins on the current Dell set-up to complete an idxbuild of all indexes/tables. Under the Sun Solaris configuration it takes a consistent 60-63 mins to complete, with/without multi-threaded options supplied to the idxbuild!

3. You are correct in your assumption.

4. Current Dell kit runs v9.1D08, Sun kit runs OE10.1B03 (also tried 10.1C01).

5. No.

TIA.

 

[TomBascom]

1) With properly implemented and managed after-imaging you should not have lost data. Although 9.1D08 isn't helping you there -- as I recall there were some issues with after-imaging at around that time.

2) Personally I'm not much of a fan of that particular replacement HW choice. I'm sure that their gear can work and I have customers running it (I even have one with a T2000 although they're running v9) but I've been less than impressed with them for quite some time now. It does not surprise me that your PC gear is faster. OTOH a more comprehensive set of benchmarks would be interesting. I wouldn't decide based solely on idxbuild times.

I'm sorry but no solutions or suggestions spring to mind. 30 minutes vs 60 minutes isn't ideal but it hardly strikes me as the end of the world unless your downtime requirement is much more stringent than I would expect for a system running this level of HW.

On the other hand I would think that this is just the sort of repeatable, well instrumented, index rebuild issue on an up to date release that the Progress development staff would want very much to hear about. I strongly suggest that you escalate the issue with tech support.

 

[cmedina]

Hello Robert:

I am experiencing a very similar case, regarding the hardware, OS, database and problem symptoms during the index building.

A difference in my case would be the size of the data base, and that the server freezes and does not respond to any command in the console.

I am expecting support form the sun hardware from another country, since the local support (Guatemala) has given no progress in the solution of this issue.

I found your post, and I would appreciate if you could tell me if you resolved your problem (if so, how?), or any clue that you might consider interesting to resolve this unexpected behavior.
Thanks, Christian

 

[Robert Jackson]

Hi Christian,

Nothing to repoprt as of yet. Progress Developmet have asked some specific questions of Sun, which have yet to be answered. Hopefully we'll get somewhere soon and I'll be able to post something concrete for those interested.

 

[TomBascom]

You should probably take it with a grain of salt but the comments in this article are interesting. In particular:

From The Register -- Stephen Jones responding to an Anonymous Coward:

"Since IBM have been bashing the UltraSPARC T1/T2/T2+ at every event I've been too its quite funny to see them make an almost exact Power-based copy."

You clearly haven't the foggiest notion about the architecture of either. The Power Processor is a rip-snorting, single-thread monster and sticking 8 of those on a chip is not going to change that; just the economics. No doubt it will come at the price of using a serious amount of electricity. It's vastly better suited to applications which require fast single-thread performance than is the T1/T2. For those that care, that includes things like high transactional databases where it is essential to keep code-paths short where exclusive access is required. If that's not done, then your multi-core machines will end up burning up a higher and higher proportion of their CPU waiting for spin locks.

The T1/T2 range is fine for low-power, high-throughput workloads where there are lots of largely independent threads. The T1./T2 hardware threading effectively treats the core resources as a system in its own right and despatches different parts of different hardware threads onto the core resources. A sort of multi-tasking within the core (for instance, the T1 has 4 threads per core and one integer unit whilst the T2 has 8 hardware threads and two integer processors per core). Now this is very efficient as resources that would otherwise be unused due to thread stalls for things like memory access can be used. However, it comes at the cost of seriously slower single-thread speed, especially so when the underlying core resources are over-committed (which, incidentally, means that CPU utilisation as reported by the operating system is seriously non-linear as the OS sees the hardware thread, effectively a virtual CPU, as the resource and not the core utilisation).

The T1/T2 is not a competitor to Power. Either a shop has committed itself to AIX (in which case Power is a done deal). The real competition to T1/T2 comes from the x86 side where sheer volume (and better single thread speed) is the real challenge from the latter.

The Power competes with Itanium and the SUN/Fujitsu SPARC64 IV (and the coming Rock processor), not the T1/T2. Frankly the SPARC64 IV is not in the same performance league at the moment. SUN will have to up their game on Rock, and they are going to have a hard time with selling volumes. Incidentally, Power, Itanium & SPARC64 IV all support a form of hardware multi-threading, but it's of a fundamentally different type to that of the T1/T2 which has the feature as its heart and soul. It's innovative, and clever, but it has its limitations and I think it will get steam-rollered by the x86 bandwagon.

This comment rings true to me and it could go a ways toward explaining some of the results reported in this thread.