How to pass set of records between two graphs

Discussion on developing CloverETL engine, transformation components etc.

varkashy
Posts: 23
Joined: Wed Oct 17, 2012 4:11 pm

How to pass set of records between two graphs

Postby varkashy » Wed Aug 28, 2013 10:17 am

Hi

I have a requirement, where in i have a central sandbox which is being invoked by different individual sandboxes, the idea here is to avoid redo the same code changes at multiple places.

However, what the above design is causing a kind of issue. Basically the individual sandbox will read a set of records and then invoke central solution . The central solution need to act on these records. I am not sure how to pass records read in one graph to another graph.

What i did for now is to write the records to a flat file. And then read the records from that flat file in central solution. This is working but i need a more optimized solution and not writing to disk.

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: How to pass set of records between two graphs

Postby kubosj » Thu Aug 29, 2013 2:57 pm

Hi varkashy,

2 options came to my mind:

1] Use files as you suggested; but instead of flat files use CloverDataReader/CloverDataWriter which should provide better performance and safer passing of data (Clover file is binary, not textual).

2] Use JMSReader/JMSWriter to pass records via JMS.

I hope this helps.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

varkashy
Posts: 23
Joined: Wed Oct 17, 2012 4:11 pm

Re: How to pass set of records between two graphs

Postby varkashy » Thu Aug 29, 2013 3:18 pm

Hi

Thanks for the inputs. Both sound interesting way for me.
For JMS way , could you elaborate? I need to open a listener for that , how does it work ?

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: How to pass set of records between two graphs

Postby kubosj » Fri Aug 30, 2013 8:31 am

Hi varkashy,

for the JMS approach: you just need to create JMS queue somewere. There is plenty of options - you can create it on application server (Tomcat, WebLogic, ...), you can have standalone JMS server (ActiveMQ), you can build simple Java app providing one, ...

Once JMS queue is defined you can write to it via JMSWriter and read from it via JMSReader. There is plenty of possible transport options - you can connect to queue in same JVM, JVM on local computer, remote computer, ... see http://activemq.apache.org/activemq-4-c ... -uris.html
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com


cron