- - download multiple zip files from a website
- unzip the text files within; each has the same name as the zip file they're in
- the text files get converted into tables within a Postgres database
- then some tables get joined together
My question is: do I need to create multiple paths for each file or can they be transformed together without explicitly referencing individual files?
I have successfully been able to download all of the zipped files from the website by reading in a CSV of filenames and mapping it through the HTTPConnector, but from there, I don't know how to handle the different metadata for each file without working with each file separately.
I'm hesitant to work with them separately because there are 20-30 files and new ones get generated each year so it would require yearly maintenance that I may not be here for and it would get unwieldy quickly.
I was hoping that http://forum.cloveretl.com/viewtopic.php?t=3622 would help me, but it was written in CLT1, and I haven't been able to replicate it successfully.
Thank you for the help!