Looping problem when using the CustomJavaTransformer

Discussion on developing CloverETL engine, transformation components etc.

thefat86
Posts: 1
Joined: Thu May 26, 2016 5:30 pm

Looping problem when using the CustomJavaTransformer

Postby thefat86 » Thu May 26, 2016 5:45 pm

Hi,
can someone help me ?

I'm working on Clover and the issue is that :

I have two input metadatas in the port 0 and 1.
for every lines in the port 0, I want to compare the values with all the lines in the second file

in programing language, it could be something like that :

for (I=0,i<10,i++){
for (j=0,j<10,j++){

}
}

for doing that, it's okay for the first metadata, for the second one, the issue is that :

First Loop

while ((inRecordFab2500 = readRecordFromPort(0)) != null) {

DataRecord outRecord=mappingWithJoy2000Table(outRecord, inRecordFab2500);

while ((inputDatasJoy2000 = readRecordFromPort(1)) != null) {

}
}

The problem is, after the first time, the value of readRecordFromPort(1)) remain to null and I don't know how to loop in the second while for every lines of the first while... as outputs, I have the values for only the first lines of the first while.

Can someone help me ?? thank you.

vazquezrosariop
Posts: 92
Joined: Mon Feb 29, 2016 5:33 pm

Re: Looping problem when using the CustomJavaTransformer

Postby vazquezrosariop » Wed Jun 01, 2016 7:50 pm

Hi,

From your message I was able to deduce that you would like to compare two records to determine if they are new or have been updated. If this is the case I would recommend that you use the following approach:

  • Have both records sorted via a unique key, then pass them into the input ports of the DataIntersection component. In the component you will have to specify the key that you will do the join on, the result will produce 3 outputs (e.g. 0 new records, 1 data that will be use for the updated records, 2 old records)
  • Using the data in output port 1 sort it via all fields in the record using ExtSort or FastSort depending on the amount of data you have, and pass it into the first input port of a new DataIntersection. In the second input port you will pass a copy of the new data you are trying to compare with by using a SimpleCopy (make sure to sort via all fields as well). This will result in validating which records have been updated from the prior run

I have taken the liberty of adding a sample graph with this process. By using this method, you can wrap it into a subgraph, thus having a standard way of comparing records.

If for any reason this was not what you are trying to accomplish, please provide me with more information.
Attachments
comparefiles.zip
(7.35 KiB) Downloaded 47 times
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

Garcy
Posts: 3
Joined: Thu Oct 27, 2016 5:02 am

Re: Looping problem when using the CustomJavaTransformer

Postby Garcy » Thu Oct 27, 2016 5:14 am

Hello,

So i have the exact same issue as above, however my end goal is different.

In the 'CustomJavaTrasformer' i need to be able to integrate through all the records of the second input, for each of the records from the first.

Is there a way to reset the 'readRecordFromPort' function so that it can start at the beginning of the list of records?

Look forward to your reply!


Cheers
Marc

vazquezrosariop
Posts: 92
Joined: Mon Feb 29, 2016 5:33 pm

Re: Looping problem when using the CustomJavaTransformer

Postby vazquezrosariop » Mon Nov 07, 2016 5:31 pm

Hi Garcy,

I have attached a example graph/subgraph compares two files and joins them based on category ID. The CustomJoiner.sgrf demonstrates how to read all records from one input port, store them in a temporary file and then iterate thought them repeatedly. For this example CustomJavaTransformer_iterationes.grf shows how the joiner is sued to replace product category ID with category name.

CustomJavaTransformer.zip
(174.16 KiB) Downloaded 26 times
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

Garcy
Posts: 3
Joined: Thu Oct 27, 2016 5:02 am

Re: Looping problem when using the CustomJavaTransformer

Postby Garcy » Tue Nov 08, 2016 6:17 am

vazquezrosariop wrote:Hi Garcy,

I have attached a example graph/subgraph compares two files and joins them based on category ID. The CustomJoiner.sgrf demonstrates how to read all records from one input port, store them in a temporary file and then iterate thought them repeatedly. For this example CustomJavaTransformer_iterationes.grf shows how the joiner is sued to replace product category ID with category name.

CustomJavaTransformer.zip


Hi,

That is perfect! Exactly what i was after and a very detailed example with multiple options, thank you!

I do have another (hopefully quick) query.. logging from a sub-graph, is this possible? I cannot seem to get anything logged from the sub-graph, however when the custom transformer is place onto the main graph, all logging occurs.

Is this simply not a feature yet, or is there something i need to do?


Cheers
Marc

vazquezrosariop
Posts: 92
Joined: Mon Feb 29, 2016 5:33 pm

Re: Looping problem when using the CustomJavaTransformer

Postby vazquezrosariop » Wed Nov 09, 2016 4:18 pm

Hi Garcy,

I have attached and updated version of the project where we have created a method called logParent(). It will allow you to temporarily switch the logging context to the parent graph and back again.

CustomJavaTransformer_logging.zip
(174.45 KiB) Downloaded 26 times
---
Pedro Vazquez Rosario
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com
How to speed up communication with CloverCARE support

Garcy
Posts: 3
Joined: Thu Oct 27, 2016 5:02 am

Re: Looping problem when using the CustomJavaTransformer

Postby Garcy » Wed Nov 09, 2016 10:12 pm

Perfect, thank you very much for the help!


Cheers
Marc