Conflict between EmailSender and 7-zip in parallel

Support/help with CloverETL implementation problems

jzhu
Posts: 19
Joined: Tue Jul 19, 2016 12:34 pm

Conflict between EmailSender and 7-zip in parallel

Postby jzhu » Wed Feb 08, 2017 12:45 pm

Just noticed the strange behavior.
A job flow (A) calls another job flow (B), and sends the result through an email with attachment (in A). The procedure is repeated for multiple inputs (tar.gz files).
(B) runs a script to decompress a tar.gz file using 7-zip and then process.
In (A), if EmailSender is in the same phase as calling job flow (B), calling flow (B) for the next round is aborted automatically while sending email. If they are in the different phases, there is no problem.
The email attachment is some result information from previous round, not directly related with the tar.gz being decompressed for the next round.
The downside to send email in different phase as calling job flow (B) is, the emails are sent in batches with long delay (could be hours -- each tar.gz file takes quite a while to process and there could be a lot of such files).

The related error logs is attached.

The CloverETL server is 4.3.0 (Tomcat).

Any clue what causes the conflict?
Attachments
localhost.2017-02-07.log
(7.21 KiB) Downloaded 11 times

vazquezrosariop
Posts: 92
Joined: Mon Feb 29, 2016 5:33 pm

Re: Conflict between EmailSender and 7-zip in parallel

Postby vazquezrosariop » Mon Feb 13, 2017 3:39 pm

Hi jzhu,

The reason that the Jobflow is failing is related to the fact that the SMTP connection to the EmailSender is being created at the beginning of the phase. So if the whole job has only one phase, and the job takes longer than 90 seconds before the email is sent, the connection will fail (timeout). We advise to have the EmailSender in a different phase, that being said in order to improve the processing time, I would recommend placing the EmailSender into a child jobflow or a child graph. Furthermore, I will add an improvement request in our tracking system to add a setting to manually set the timeout of the EmailSender.
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

vazquezrosariop
Posts: 92
Joined: Mon Feb 29, 2016 5:33 pm

Re: Conflict between EmailSender and 7-zip in parallel

Postby vazquezrosariop » Tue Feb 14, 2017 3:39 pm

Hi jhzu,

We noticed that in the log you sent there was a APR error and we would like to get some more information:

  • Can you please share your server xml file (please remove any sensitive data)
  • Did you download the clover.war file or did you download the bundle that includes tomcat and the clover.war file
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support


cron