Monday, 12 March 2007

My Unix way to parallelize PDI/Kettle jobs ...

Hi all,

Today I want to share some code I used in order to parallelize jobs, at OS level (for me, it's Solaris).
My needs are :
- run 2 jobs in parallel, each on a separate engine,
- run a third job when first 2 parallel jobs are both complete.

I decided to design 2 separate jobs, then manage the execution with simple bash scripting using temp files as flags.
Here is the final code (extract from larger sh file) :

  • First 2 blue lines : 2 jobs to run in parallel, create a flag file for each command.
  • Red lines : a flag control to detect if both jobs are complete (= presence of flag file).
  • Last blue line : the third job to run after the completion of the parallel ones.

1 comment:

cq said...

job1 & job2 & wait ; job3

would probably suffice ?