Search found 9 matches

by hewills
Fri May 26, 2017 6:09 pm
Forum: CloverETL Server
Topic: Tomcat user running CloverETL
Replies: 1
Views: 86

Tomcat user running CloverETL

Currently we are using tomcat as our application server, and as such the tomcat user is running CloverETL. Since it's normally recommended to have tomcat as a user without login, it makes it tricky having Clover run shell scripts. Basically we have to have tomcat sudo as another user. Is there a rec...
by hewills
Thu May 25, 2017 7:54 pm
Forum: CloverETL Support
Topic: skip graph initialization
Replies: 1
Views: 43

skip graph initialization

I know there is a way to skip the graph configuration, using skip_check_config . But is there a way to have a graph skip the initialization, and just start running? Basically, I have a child graph with dynamic metadata using a sql query - SELECT * FROM SOURCE_TBL WHERE 1=0; Before running the graph,...
by hewills
Tue May 02, 2017 5:02 pm
Forum: CloverETL Server
Topic: Using allocation with DBInputTable
Replies: 5
Views: 182

Re: Using allocation with DBInputTable

Thanks Pedro! I had a chance to use your design, and it sped up the graph as expected. As you said, using the 'ParallelPartition' seems to be key.
Using 'PARTITION' wouldn't be an option for the database we are currently working with. But I'll keep it in mind for the future.
by hewills
Tue Apr 25, 2017 11:42 pm
Forum: CloverETL Server
Topic: Setting number of executors to a jobflow
Replies: 5
Views: 168

Re: Setting number of executors to a jobflow

Now that I understand how the executors work, I don't think the 'max_running_concurrently' parameter is needed. But it's nice to know in case we need it for a different scenario.
I setup the ExecuteGraph with dynamic table parameters and metadata, and so far it's working great. Thanks!
by hewills
Mon Apr 24, 2017 7:50 pm
Forum: CloverETL Server
Topic: Setting number of executors to a jobflow
Replies: 5
Views: 168

Re: Setting number of executors to a jobflow

I decided to try setting up the "dynamic table load", where I feed a list of table names and parameters to one ExecuteGraph. This looks like a better solution for us.
by hewills
Fri Apr 21, 2017 6:07 pm
Forum: CloverETL Server
Topic: Using allocation with DBInputTable
Replies: 5
Views: 182

Re: Using allocation with DBInputTable

Hi, thanks for taking a look. How many records are you processing? ~ 3,000,000 How long does this process usually take? 7.5 minutes What Database Server are you utilizing (vendor, version)? Oracle 12c database, on Red Hat 7.2 Clover Server is running on CentOS I don't have access to the source serve...
by hewills
Fri Apr 21, 2017 5:51 pm
Forum: CloverETL Server
Topic: Setting number of executors to a jobflow
Replies: 5
Views: 168

Re: Setting number of executors to a jobflow

Thanks for the reply Eva, that makes sense. You're right that in my example the jobs could be run in parallel, except if too many run at a time, it causes issues. We have about 500 graphs that need to be executed, so I'm trying to figure out the best way to organize them. I was hoping that there was...
by hewills
Tue Apr 18, 2017 5:00 pm
Forum: CloverETL Server
Topic: Using allocation with DBInputTable
Replies: 5
Views: 182

Using allocation with DBInputTable

I'm using Corp Plus Server, and saw that the data partition objects are available to use, including the Allocation parameter. I've been testing how to use it on a DBInputTable object, where each allocation pulls a different set of data from our source table, and I use ParallelSimpleGather to bring t...
by hewills
Tue Apr 18, 2017 4:46 pm
Forum: CloverETL Server
Topic: Setting number of executors to a jobflow
Replies: 5
Views: 168

Setting number of executors to a jobflow

I have a jobflow with 8 graphs, that I'm running on Clover Server. Each graph is truncating a target table, selecting data from my source, and loading the target table. They are connected and running synchronously in the jobflow. I created a "global" parameter called EXECUTORS, with a valu...

cron