jzhu
Posts: 19
Joined: Tue Jul 19, 2016 12:34 pm

How to limit the total number of running instances of a file listener job flow

Postby jzhu » Wed May 31, 2017 12:39 pm

File listener is used to trigger a job flow. It takes a while to process each file (typically 15 minutes in our examples). There are lots of files, arriving at random time.
In case of dozens (even hundreds) of such files arriving in a short time period, many instances of the job flow are fired up, potentially causing problems (out-of-memory etc., which actually happened in real environment).

Tried some ways to avoid the issue using external info (such as files/directories etc.).
For example, each instance moves its file(s) to own work directory (named with RUN_ID) before processing. The job flow counts such work directories first, and starts processing only if the count is under certain limit.
It does not seem to work well, as every instance fired up simultaneously does not see each other (and their work directories). The counting (first operation) in each instance seems to be done before the other operations of all instances fired up simultaneously.

Does CloverETL have any way to detect total number of running instances of a particular job flow (such as one triggered by a file listener)?
Or is there any way to limit the access of a job flow (such as singleton)?
Or is there any semaphore accessible from different instances of the same job flow?

The CloverETL used is version 4.3 (all-in-one bundle: Tomcat 8.0.30).

Thanks.

vazquezrosariop
Posts: 107
Joined: Mon Feb 29, 2016 5:33 pm

Re: How to limit the total number of running instances of a file listener job flow

Postby vazquezrosariop » Thu Jun 01, 2017 5:53 pm

Hi jzhu,

CloverETL does have a way to limit the total number of running instances. You can achieve this via Job Config Properties on the Sandbox (Server Console - Sandboxes - CloverETLProject - Config Properties). The two properties you want to set are max_running_concurrently = <the amount of Jobflow you want to run at the same time> and enqueue_executions = true

Best Regards,
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

jzhu
Posts: 19
Joined: Tue Jul 19, 2016 12:34 pm

Re: How to limit the total number of running instances of a file listener job flow

Postby jzhu » Fri Jun 02, 2017 3:17 pm

Thanks, Pedro


cron