Working With Multiple Files

Support/help with CloverETL implementation problems

jcatoe
Posts: 2
Joined: Wed Jan 17, 2018 4:50 pm

Working With Multiple Files

Postby jcatoe » Wed Jan 17, 2018 5:03 pm

I'm trying to replicate this process with CloverETL:

    - download multiple zip files from a website
    - unzip the text files within; each has the same name as the zip file they're in
    - the text files get converted into tables within a Postgres database
    - then some tables get joined together

My question is: do I need to create multiple paths for each file or can they be transformed together without explicitly referencing individual files?

I have successfully been able to download all of the zipped files from the website by reading in a CSV of filenames and mapping it through the HTTPConnector, but from there, I don't know how to handle the different metadata for each file without working with each file separately.

I'm hesitant to work with them separately because there are 20-30 files and new ones get generated each year so it would require yearly maintenance that I may not be here for and it would get unwieldy quickly.

I was hoping that http://forum.cloveretl.com/viewtopic.php?t=3622 would help me, but it was written in CLT1, and I haven't been able to replicate it successfully.

Thank you for the help!

the_goat
Posts: 14
Joined: Fri Dec 01, 2017 6:16 am

Re: Working With Multiple Files

Postby the_goat » Thu Jan 18, 2018 3:41 pm

Are we talking about some BLOBs or CLOBs in that database? Or as I suspect, you need to have them parsed into fields/columns and then put to the database in some sort of format?

jcatoe
Posts: 2
Joined: Wed Jan 17, 2018 4:50 pm

Re: Working With Multiple Files

Postby jcatoe » Fri Jan 19, 2018 4:51 pm

Columns and tables. And we really just drop all of the tables and recreate them from the text files for all but one table.

vazquezrosariop
Posts: 119
Joined: Mon Feb 29, 2016 5:33 pm

Re: Working With Multiple Files

Postby vazquezrosariop » Tue Jan 23, 2018 6:16 pm

I created an example of how you can achieve your use case. I used a combination of Jobflows and Graphs to generate the metadata, and the database tables dynamically based on the data in each file and then inserted the data based on the table name.
Attachments
MultipleFilesExample.zip
(38.99 KiB) Downloaded 4 times
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support