jurriaan
New Member
I'm testing a dump/load for a move from windows (enterprise 10.1C) to linux (enterprise 10.2B07)
The dump-part has been solved by our friendly system admins, who created a windows 2003 VMWare instance with SSD storage. Now the dump takes just an hour for 49 gigabytes. We have a downtime window of about 12 hours, but any time saved will enable me to get some rest before the next day
The load will take place on a Red Hat Enterprice VMWare instance, 1 single Xeon 2.9 Ghz cpu, 16 Gb memory, storage on a metrocluster (Netapp). I can't influence the storage configuration, no idea what raid-levels lurk in those murky depths.
I've made great speed advances already:
sequential import of all tables + build indexes on the fly (-TB 24 -TM 32 -SS ... -G 1): 3 hours 40 minutes
parallel import of the one big table (20G)/ all other tables +build indexes on the fly (-TB 24 -TM 32 -SS .. -G 1): 2 hours 40 minutes
parallel import of the one big table/all other tables, build indexes afterwards (
-C idxbuild all -thread 1 -threadnum 4 -SS ... -TB 64 -TF 85 -TM 32 -datascanthreads 2 -B 1024 -TMB 64 -rusage): 1 hour 40 minutes
so I have 2 hours saved already. When building the indexes for the big table, I read
Multi-threaded index sorting and building complete. Elapsed time: 951.483
Resource usage: CPU user 804.163749, system 42.930474
Resource usage: DISK reads: 35005288 KB at 36 MB/sec, writes: 32735828 KB at 34 MB/sec
16 indexes were rebuilt in area 10. Total elapsed time: 1859.124
Resource usage: CPU user 1638.285943, system 72.874921
Resource usage: DISK reads: 56857840 KB at 30 MB/sec, writes: 41998548 KB at 22 MB/sec
Temporary sort file at /srt/1/ used up 3118720K of disk space. (11480)
Temporary sort file at /srt/2/ used up 3390656K of disk space. (11480)
Temporary sort file at /srt/3/ used up 2507072K of disk space. (11480)
Temporary sort file at /srt/4/ used up 2494272K of disk space. (11480)
A total of 11510720K of temporary sort disk space was used for area 10. (11483)
Do the experts see any good avenues to speed the process up any further? I could ask for more memory (would 32 Gb mean the temporary sort files disappear?) or a dual cpu or tweak parameters, but I'd like to get a hint on where to look first.
Thanks for reading!
The dump-part has been solved by our friendly system admins, who created a windows 2003 VMWare instance with SSD storage. Now the dump takes just an hour for 49 gigabytes. We have a downtime window of about 12 hours, but any time saved will enable me to get some rest before the next day
The load will take place on a Red Hat Enterprice VMWare instance, 1 single Xeon 2.9 Ghz cpu, 16 Gb memory, storage on a metrocluster (Netapp). I can't influence the storage configuration, no idea what raid-levels lurk in those murky depths.
I've made great speed advances already:
sequential import of all tables + build indexes on the fly (-TB 24 -TM 32 -SS ... -G 1): 3 hours 40 minutes
parallel import of the one big table (20G)/ all other tables +build indexes on the fly (-TB 24 -TM 32 -SS .. -G 1): 2 hours 40 minutes
parallel import of the one big table/all other tables, build indexes afterwards (
-C idxbuild all -thread 1 -threadnum 4 -SS ... -TB 64 -TF 85 -TM 32 -datascanthreads 2 -B 1024 -TMB 64 -rusage): 1 hour 40 minutes
so I have 2 hours saved already. When building the indexes for the big table, I read
Multi-threaded index sorting and building complete. Elapsed time: 951.483
Resource usage: CPU user 804.163749, system 42.930474
Resource usage: DISK reads: 35005288 KB at 36 MB/sec, writes: 32735828 KB at 34 MB/sec
16 indexes were rebuilt in area 10. Total elapsed time: 1859.124
Resource usage: CPU user 1638.285943, system 72.874921
Resource usage: DISK reads: 56857840 KB at 30 MB/sec, writes: 41998548 KB at 22 MB/sec
Temporary sort file at /srt/1/ used up 3118720K of disk space. (11480)
Temporary sort file at /srt/2/ used up 3390656K of disk space. (11480)
Temporary sort file at /srt/3/ used up 2507072K of disk space. (11480)
Temporary sort file at /srt/4/ used up 2494272K of disk space. (11480)
A total of 11510720K of temporary sort disk space was used for area 10. (11483)
Do the experts see any good avenues to speed the process up any further? I could ask for more memory (would 32 Gb mean the temporary sort files disappear?) or a dual cpu or tweak parameters, but I'd like to get a hint on where to look first.
Thanks for reading!