Writing raw data to an output port

Discussion on developing CloverETL engine, transformation components etc.

jsinglet
Posts: 6
Joined: Mon Aug 01, 2016 11:49 am

Writing raw data to an output port

Postby jsinglet » Tue Oct 11, 2016 7:03 pm

Hi,

I'm trying to write a Custom Java Component that will read a file, decrypt it, then write the data to the output port which would then be connected to a standard FlatFileReader.

I've managed to get the component to read the file and decrypt it, which works as I can print the decrypted data onto the console. However I cannot work out how to write the data to the output port.

I have a byte[] array, but how do I write this to the output port? All I can find is write methods that take a DataRecord, but I don't want to parse the data as the CSV will be better handled by the standard Clover components.

Can anyone help?

Thanks

John

jsinglet
Posts: 6
Joined: Mon Aug 01, 2016 11:49 am

Re: Writing raw data to an output port

Postby jsinglet » Wed Oct 12, 2016 3:05 pm

OK, I see now how to do it:

Code: Select all

         
         DataRecord record = DataRecordFactory.newRecord(metadata);
         record.getField(0).setValue(decrypted.toByteArray());
         
         writeRecordToPort(0, record);


Where decrypted is a ByteArrayOutputStream where the decrypted data was written.

cholastal
Posts: 107
Joined: Tue Sep 01, 2015 1:22 pm

Re: Writing raw data to an output port

Postby cholastal » Wed Oct 12, 2016 3:17 pm

Hi,

The data you want to write to an output port has to have the form of a DataRecord. Every data flowing through a CloverETL graph has to be mapped to some metadata. This is the way you do it in a custom Java transformation. You don't have to parse the data yourself, you can create a metadata containing only one field of byte type and map your whole byte array to it. This way you can get the array to an edge and parse it by other components. The code may look like this:

Code: Select all

public void execute() {
      
   DataRecord record = outRecords[0];
         
   try (InputStream in = getInputStream("data-in/data2.txt")) {
      
      record.getField("field1").setValue(IOUtils.toByteArray(in));
      writeRecordToPort(0, record);
      
   } catch (IOException e) {
      throw new JetelRuntimeException("Problem with reading input file", e);
   }      
}


The code reads data from a file and the "IOUtils.toByteArray(in)" is just a substitution of a byte[] array.

Hope this helps.
---
Lukas Cholasta
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support