Question Compiling Programs In Parallel

[catch.saravana]

@All - We have 1300+ programs scattered across 15 folders which takes 1 hr 20 minutes for a full build. We are planning to automate our build process. Is there a way to do compilation in parallel? We don't have too many user licenses and we can't run parallel compilation on different sessions. Is there an option to do it in parallel using single session (like using CALLBACK, PUBLISH/SUBSCRIBE)? Or any other technique followed in your application?

 

[Rob Fitzpatrick]

We do compiles serially, in a single session. You want to avoid client-server if you can. As we target different OSes and window systems, some customers need one compile (Windows-only) and some need two (front-end/Windows and back-end/*nix). For the front-end compile for *nix customers we connect to the DB(s), dump schema, build local schema-holders on the Windows side and connect the compile session to it. It's a lot faster than connecting remotely to the actual databases.

On a good server, server-side compiles take about 10-15 minutes; client-based Windows compiles might take double that. For reference, there might be something like 15,000+ source files, maybe 9,500 compile units, and probably 60-80 source directories in the compile propath, along with a handful of .PLs. So all in all your compile looks to me like it should run a lot faster than it does.

 

[catch.saravana]

@Rob Fitzpatrick - my bad, I missed one zero on it. It's 13000+ programs scattered across 15 folders. Sorry for the inconvenience.

 

[catch.saravana]

So in this case do you mean 13000+ programs should not take 1 hr 20 minutes when we compile one by one on a good server?

Is it advisable to do it in serially? If yes, rather than investing time on this, I can invest time on workaround with the deployment schedule.

 

[Rob Fitzpatrick]

I've certainly seen compiles of that many programs take this long, and longer. But in my experience it was either when running on a slow machine (e.g. a PC) or when connected client/server to the database(s).

Whether it's "advisable" is a matter of your priorities: if serial is fast enough, given the allowed window of time, then it's the most straightforward approach. If it is too slow then look at your setup and see if it can be optimized. If not then you may have to get more creative, e.g. with parallel threads or with a dedicated build server that has better specs.

 

[TomBascom]

I have seen compiles of many, many more programs than that go much faster.

And I have seen compiles of many fewer go slower.

Client/server connections are a very bad idea for compiling. So are slow IO subsystems.

Multi-session parallel compiles can help a lot.

 

[catch.saravana]

@TomBascom @Rob Fitzpatrick - Thanks for your advise. Compilation on Client Server to the database took 1 hr 20 minutes. When I ran the compilation with DB on the same server it hardly took less than 6 minutes for the same set of programs.

As suggested by Tom, we are also planning to run couple of session in parallel to get the compilation faster (we don't have too many user licenses but 2 or 3 should be fine). We are in the process of automating the build process. Thanks!