starsailor
Posts: 31
Joined: Tue Jun 04, 2013 9:05 pm

Archive execution logs to Amazon S3 using Archivator.

Postby starsailor » Thu Jan 15, 2015 7:11 pm

I need to be able to archive execution history and task history to Amazon S3 instead of a local filesystem. From the documentation it looks like the "Output path for archives" field only accepts filesystem paths relative to the local host. How can remote cloud storage be specified and an output target?

imriskal
Posts: 383
Joined: Wed Aug 15, 2012 8:18 am

Re: Archive execution logs to Amazon S3 using Archivator.

Postby imriskal » Fri Jan 16, 2015 4:00 pm

Hi starsailor,

I would save the history into the local filesystem and then use File Event Listener with a graph uploading it into Amazon S3 using the following URL format:

Code: Select all

http://access_key_id:secret_access_key@bucketname.s3.amazonaws.com/filename.out

Or alternatively, you can download the execution history via HTTP API and upload it right away.

Regards,
---
Lubos Imriska
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

starsailor
Posts: 31
Joined: Tue Jun 04, 2013 9:05 pm

Re: Archive execution logs to Amazon S3 using Archivator.

Postby starsailor » Fri Jan 16, 2015 4:46 pm

So what you are saying is the Archivator doesn't know how to access Amazon S3 or other cloud based file stores, right? Based on your reply am I correct to say that the Archivator only know how to write archive output thru the local system file api?

imriskal
Posts: 383
Joined: Wed Aug 15, 2012 8:18 am

Re: Archive execution logs to Amazon S3 using Archivator.

Postby imriskal » Mon Jan 19, 2015 9:02 am

Correct, it can save the archive only into the local file system.
---
Lubos Imriska
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support