Using Runtime.exec() to execute the graph

Discussion on developing CloverETL engine, transformation components etc.

angel
Posts: 8
Joined: Thu Aug 25, 2011 5:06 pm

Using Runtime.exec() to execute the graph

Postby angel » Thu Aug 25, 2011 5:33 pm

I am using Runtime.exec() to execute the graph from Java.Graph runs pretty well and outputs 800000 records and it doesn't stop either the JVM or the graph but stops writing to the output.
When I see the log's the out: and KB never change for the rest of the time.I tried increasing the JVM, log level off,no debug no change.
I assumed my command or my graph was wrong so tried executing the same graph via command line it executes perfectly and gives me result more than 800000 records.

Below is the sample code.please suggest me what's wrong with my code or did i miss any arguments?

public String Graph() {
graph ="java -Djava.io.tmpdir=/opt/tomcat/ -cp com.clover/lib/lib/* org.jetel.main.runGraph
-plugins clover/lib/plugins -loglevel OFF -checkconfig -nodebug -config properties/defaultProperties.cfg
-debugdirectory Debug graph.grf";

return graph;
}

public void runGraph() {
try {
p = Runtime.getRuntime().exec(Graph());
readProcess(p,"graph failed");
} catch (IOException e) {
e.printStackTrace();
}
}
public void readProcess(Process p2, String sub) {
String s="",message="";
try{
stdInput = new BufferedReader(new InputStreamReader(p2.getInputStream()));
stdErr = new BufferedReader(new InputStreamReader(p2.getErrorStream()));
int i=0;
while ((s = stdInput.readLine()) != null) {
if (s.contains("ERROR") && !s.isEmpty()) {

message += s + "\r\n";
while ((s = stdInput.readLine()) != null) {
if (s.contains("TIMER") || s.contains("INFO") || s.startsWith("DEBUG") || s.startsWith("INFO"))
break;
message += s + "\r\n";
}
}
}
while((s = stdErr.readLine()) != null){
message += s + "\r\n";
}
if (message != "") {
email.setBody(message);
email.setSubject(sub);
email.send();
System.out.println(sub);
System.exit(0);
}
}catch(IOException e){
e.printStackTrace();
}

}

avackova
Posts: 841
Joined: Fri Jul 20, 2007 9:28 am

Re: Using Runtime.exec() to execute the graph

Postby avackova » Fri Aug 26, 2011 12:48 pm

Hello Angel,
at least you don't wait for the process to finish its job (p2.waitFor()).
By the way: why do you execute the graph from java by calling the external system process? You'd rather call our classes, that are designed to execute the graph from java. For more information see Embedding clover and Tips for Integrating CloverETL to Third Party Applications.
Agata Vackova
Javlin a.s.
agata.vackova@javlin.eu

angel
Posts: 8
Joined: Thu Aug 25, 2011 5:06 pm

Re: Using Runtime.exec() to execute the graph

Postby angel » Fri Aug 26, 2011 4:35 pm

Hi Agata,

Thank you for the respone.

But my logic runs perfectly for other Graphs without the waitfor() (lets say around 80million records) except for this graph which uses the Initiate Connection to search the data.

The Graph uses these components
datareader->partition->initiate(16 instances)->reformat(32)->simplegather->datawriter

Thanks,
Angel

angel
Posts: 8
Joined: Thu Aug 25, 2011 5:06 pm

Re: Using Runtime.exec() to execute the graph

Postby angel » Fri Aug 26, 2011 5:21 pm

Hi,

Sorry new to forums forgot to mention why we haven't used your classes to execute from java since we didn't want to perform any sort of maintenance on this java application.Any type of edit to graphs will be done in the workspace and move the graph to our Linux and execute multiple graphs one by one.

Thanks,
Angel

angel
Posts: 8
Joined: Thu Aug 25, 2011 5:06 pm

Re: Using Runtime.exec() to execute the graph

Postby angel » Fri Aug 26, 2011 10:26 pm

Hello Agata,

Please ignore the other two messages.I assumed that clover class need maintenance but it seems using the below logic
I might be able to do the same.But now I got this exception saying
ERROR [WatchDog] - Node DATA_READER0 error details:
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)

I know the cause its cause due to the temp space. But Is there any field to change the temp path like command line below
java -Djava.io.tmpdir=/opt/tomcat/ that can be added to the logic or do i have to change it in the env variable in linux


//logic
EngineInitializer.initEngine("plugins", "defaultProperties.cfg", null);
String graphPath = "Graph.grf";
InputStream is = new BufferedInputStream(new FileInputStream(graphPath));

TransformationGraph graph = null;
GraphRuntimeContext runtimeContext = new GraphRuntimeContext();
runtimeContext.setDebugDirectory("Debug");

graph = TransformationGraphXMLReaderWriter.loadGraph(is,runtimeContext.getAdditionalProperties());
EngineInitializer.initGraph(graph, runtimeContext);
Future<Result> futureResult = runGraph.executeGraph(graph, runtimeContext);

avackova
Posts: 841
Joined: Fri Jul 20, 2007 9:28 am

Re: Using Runtime.exec() to execute the graph

Postby avackova » Mon Aug 29, 2011 10:27 am

Hello,
unfortunately it is not possible to change CloverETL temp directory. The only way to do this is to run java with the -Djava.io.tmpdir=/opt/tomcat/ option.
Agata Vackova

Javlin a.s.

agata.vackova@javlin.eu

angel
Posts: 8
Joined: Thu Aug 25, 2011 5:06 pm

Re: Using Runtime.exec() to execute the graph

Postby angel » Mon Aug 29, 2011 5:44 pm

Hello,

Is there a way to redirect the console output to a file or just set to loglevel to ERROR.

Tried setting it to
runtimeContext.setloglevel(Level.OFF|LEVEL.ERROR);

It seems no effect and the logging start from the EngineInitializer


Thanks,
Angel

avackova
Posts: 841
Joined: Fri Jul 20, 2007 9:28 am

Re: Using Runtime.exec() to execute the graph

Postby avackova » Wed Aug 31, 2011 12:02 pm

Hello Angel,
CloverETL uses log4j for logging. You can create your configuration file (see Logging) and use it in your program:

Code: Select all

PropertyConfigurator.configure("<path_to_your_log4j_properties_file>");
Agata Vackova

Javlin a.s.

agata.vackova@javlin.eu