Global static variables

Discussion on developing CloverETL engine, transformation components etc.

yabel
Posts: 1
Joined: Fri Feb 03, 2012 8:20 am

Global static variables

Postby yabel » Fri Feb 03, 2012 8:31 am

Hi!
Is there any way to use global static variables of any type and share them between nodes?
For example, in Reformat node1 declare static int VAR1, set the value of VAR1 in this node and then use the value of VAR1 in reformat node2?

I tried to declare class Global { public static int VAR1;} in node1 for this purpose, set Global.VAR1 = 1234567 in transform(...) in node1, but in the node2 the value of Global.VAR1 is always 0.

Any suggestions?

Thanks

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Fri Feb 03, 2012 1:53 pm

Hi yabel,

I would suggest usage of dictionary - see http://www.cloveretl.com/documentation/ ... onary.html

If you need both read and write it, it make sense to divide reading a writing components into 2 phases. Otherwise behavior won't be deterministic.

Your CloverETL Support team
Jarek
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Wed Mar 14, 2012 1:08 am

Hi Jarek,

How can I use dictionary in the FileURL field of XLSDataWriter? I've read the docs here and here, but couldn't seem to get it to work.
Also, I tried to use the dictionary in the command field of SystemExecute, and it didn't work. Please help.

Thanks!
Jus

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Wed Mar 14, 2012 5:14 am

Hi Jarek,

How do I use dictionary entries in the FileURL field of XLSDataWriter? I tried the examples from here and here but it didn't work.
Also, how can I use the dictionary in a commnd field of SystemExecute?

Thanks!
Jus

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Thu Mar 15, 2012 11:15 am

Hi jus,

SystemExecute: You can prepare batch script executed by SystemExecute. Then pass arguments through input port to it - so for bash script it will be stdin. Then you need to parse stdin and launch commands you need. Here are some hints: viewtopic.php?f=4&t=5241&p=8231&sid=78de9d6fa4b335ca1a17520c79df67e1#p8231

XLSDataWriter: The only way I know to pass dynamic file name is via "-P:name=value" to graph. There you can use ${name} substitution.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Thu Mar 15, 2012 5:18 pm

Hi Jaroslav, thanks for your help.
I can continue working with the SystemExecute example.

Though I'm not sure what to do with the XLSDataWriter, since the filename that I am writing to will be dependent on the output from the ComplexDataReader I'm using. Is there any way at all to do this besides passing a parameter to the graph? Because I don't have the value of the parameter before running it. Also, I can't use a higher level graph that calls RunGraphs because of the bug that was fixed in 3.3.0-M1, because we need this to be in production by the end of the month.

I was thinking of writing to a generic filename, and then run SystemExecute to rename it. But can you think of another way?

Thank you so much!
Jus

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Fri Mar 16, 2012 9:20 am

Hi jus,

generic file name is feasible if there is certainty that graph will not run in more parallel instances. So I would recommend to pass unique file name as graph parameter. Unfortunately there is currently no way how you can generate it by "randomUUID()" and use in both Writer and SystemExec on graph side.

I can't think out better solution. In near, but not yet estimated future, there will be workflows support, which will solve issues you have.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Thu Mar 22, 2012 10:08 pm

Hi Jaroslav, thanks for your answer.
My next question is, how should we handle those workflow things meanwhile?
Should we use external scripts? Do it in the server? How do people do it usually? and what's your suggestion?
Thanks!
Jus

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Fri Mar 23, 2012 9:02 am

Hi jus,

usually solution is based on:
* SystemExecute (scripting extra logic)
* RunGraph (run other graph somehow parametrized, possibly in loop)
* HttpConnector (can invoke tasks on Server via simple HTTP API)

All of them have currently some limitation, so it is necessary to use workarounds. That is the why we are preparing workflows support.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Fri Mar 23, 2012 8:28 pm

Thanks Jaroslav for your prompt reply! I really appreciate it.

I now have a question about closing file handles.
I am using ComplexDataReader to read in a flat file. After processing is finished, I'm using SystemExecute on the next phase to move the file into an archive or reject folder. But I get this error message:

Code: Select all

ERROR [SYS_EXECUTE1_0] - The process cannot access the file because it is being used by another process.

I get the same problem when using XLSDataWriter and then trying to rename it using SystemExecute on the next phase (in the same graph).

How can I close those file handles from within the same graph? Or is there any other solution you can think of?

Thanks a bunch!
Jus

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Tue Mar 27, 2012 9:56 am

Hi jus,

ComplexDataReader closes opened file when graph is finished. That is not correct behavior and it will be fixed by: https://bug.javlin.eu/browse/CL-2229

For now, only workaround is to use 2 graphs - first, which will run second and then move/delete file. In second will be complex data reader.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

jus
Posts: 12
Joined: Wed Feb 29, 2012 12:13 am

Re: Global static variables

Postby jus » Tue Mar 27, 2012 6:01 pm

I guess it's the same problem with the XlsDataWriter? Will it also be fixed in 3.3.0-M2?
Thanks again Jaroslav!
Jus

kubosj
Posts: 372
Joined: Thu Jan 12, 2012 9:10 am

Re: Global static variables

Postby kubosj » Wed Mar 28, 2012 8:25 am

Hi jus,

sorry I missed that. There is new issue https://bug.javlin.eu/browse/CL-2235

I would not expect it will be solved in 3.3.0-M2. The same may happen to CL-2229, although it is currently planned for M2.
Jaroslav Kubos
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com


cron