[Progress Communities] [Progress OpenEdge ABL] Forum Post: RE: Loading schema into a new, empty database is slow (prodict/load_df.p)

Status
Not open for further replies.
D

dbeavon

Guest
@onnodehaan For anything CPU-bound you should always be able to scale-up the workload on a big server with multiple CPU's. The first thing to do is just open the task manager while that "build street" is running and see if the CPU is fully utilized (all cores simultaneously). If not then there may be something fishy going on. In your case, the limiting factor might be some third-party dependencies that your colleages might not be able to control (eg PCT or similar). There may be limitations in these third-party dependencies that limit the utilization of CPU (similar to the limitation of OE when it is creating an empty schema). Otherwise the steps for OE alone should be something like so: Delete pre-existing r-code Enumerate all the sources/programs you want to compile Divide them up into batches (eg. 1000 things to compile in each batch) For each batch, output a new ABL program that compiles all the sources/programs (example below) Compile all batches in separate concurrent _progres processes, while using -RO against a database connected with -1 (the db does *not* need to be a served database when using -RO) Make sure to use load-balancing so if the batches aren't equal, then additional batches will be assigned to worker threads that finish early (ie. in .Net you can use Parallel.ForEach and specify parallelization and load-balancing options) Here is an example of a compile program that would build and save all 1000 source programs in a batch. SESSION:SUPPRESS-WARNINGS-LIST = "15090". PROPATH = "Whatever". MESSAGE "...compiling... cyc\p\cyc0001.p". COMPILE cyc\p\cyc0001.p GENERATE-MD5 SAVE. MESSAGE "...compiling... cyc\p\cyc0002.p". COMPILE cyc\p\cyc0002.p GENERATE-MD5 SAVE. MESSAGE "...compiling... cyc\p\cyc0003.p". COMPILE cyc\p\cyc0003.p GENERATE-MD5 SAVE. ... If there are no unusual third-party dependencies, then you should be able to double-check your colleage's findings. Something good will probably come out of it. There is always room for improvement. If you spend some of your day waiting on builds to finish, then there is ROI in improving the build process. Even the saving of 10 seconds can add up ... if you do builds for four staging environments, with five databases in each environment (which need their own r-code because of schema drift) and multiple iterations of this is required on a given day (a bad bug-day). Below is what the task manager looks like during compiles (eight cores with full utilization). Make sure that disk isn't ever the bottleneck by using SSD. Also, make sure you have plenty of RAM because the file system cache is even faster than the SSD. If you are the one that has to wait on builds, It is important that you have some control over improving them. If someone else is managing your builds for you, then you will have the troublesome task of trying to convince them why 50 seconds is better than 60 seconds. (Although they would probably understand if it was their own time that was being wasted ;) Good luck.

Continue reading...
 
Status
Not open for further replies.
Top