Using Runtime.exec() to execute the graph

[angel]

I am using Runtime.exec() to execute the graph from Java.Graph runs pretty well and outputs 800000 records and it doesn't stop either the JVM or the graph but stops writing to the output.
When I see the log's the out: and KB never change for the rest of the time.I tried increasing the JVM, log level off,no debug no change.
I assumed my command or my graph was wrong so tried executing the same graph via command line it executes perfectly and gives me result more than 800000 records.

Below is the sample code.please suggest me what's wrong with my code or did i miss any arguments?

public String Graph() {
graph ="java -Djava.io.tmpdir=/opt/tomcat/ -cp com.clover/lib/lib/* org.jetel.main.runGraph
-plugins clover/lib/plugins -loglevel OFF -checkconfig -nodebug -config properties/defaultProperties.cfg
-debugdirectory Debug graph.grf";

return graph;
}

public void runGraph() {
try {
p = Runtime.getRuntime().exec(Graph());
readProcess(p,"graph failed");
} catch (IOException e) {
e.printStackTrace();
}
}
public void readProcess(Process p2, String sub) {
String s="",message="";
try{
stdInput = new BufferedReader(new InputStreamReader(p2.getInputStream()));
stdErr = new BufferedReader(new InputStreamReader(p2.getErrorStream()));
int i=0;
while ((s = stdInput.readLine()) != null) {
if (s.contains("ERROR") && !s.isEmpty()) {

message += s + "\r\n";
while ((s = stdInput.readLine()) != null) {
if (s.contains("TIMER") || s.contains("INFO") || s.startsWith("DEBUG") || s.startsWith("INFO"))
break;
message += s + "\r\n";
}
}
}
while((s = stdErr.readLine()) != null){
message += s + "\r\n";
}
if (message != "") {
email.setBody(message);
email.setSubject(sub);
email.send();
System.out.println(sub);
System.exit(0);
}
}catch(IOException e){
e.printStackTrace();
}

}

[avackova]

Hello Angel,
at least you don't wait for the process to finish its job (p2.waitFor()).
By the way: why do you execute the graph from java by calling the external system process? You'd rather call our classes, that are designed to execute the graph from java. For more information see Embedding clover and Tips for Integrating CloverETL to Third Party Applications.

[angel]

Hi Agata,

Thank you for the respone.

But my logic runs perfectly for other Graphs without the waitfor() (lets say around 80million records) except for this graph which uses the Initiate Connection to search the data.

The Graph uses these components
datareader->partition->initiate(16 instances)->reformat(32)->simplegather->datawriter

Thanks,
Angel

[angel]

Hi,

Sorry new to forums forgot to mention why we haven't used your classes to execute from java since we didn't want to perform any sort of maintenance on this java application.Any type of edit to graphs will be done in the workspace and move the graph to our Linux and execute multiple graphs one by one.

Thanks,
Angel

[angel]

Hello Agata,

Please ignore the other two messages.I assumed that clover class need maintenance but it seems using the below logic
I might be able to do the same.But now I got this exception saying
ERROR [WatchDog] - Node DATA_READER0 error details:
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)

I know the cause its cause due to the temp space. But Is there any field to change the temp path like command line below
java -Djava.io.tmpdir=/opt/tomcat/ that can be added to the logic or do i have to change it in the env variable in linux


//logic
EngineInitializer.initEngine("plugins", "defaultProperties.cfg", null);
String graphPath = "Graph.grf";
InputStream is = new BufferedInputStream(new FileInputStream(graphPath));

TransformationGraph graph = null;
GraphRuntimeContext runtimeContext = new GraphRuntimeContext();
runtimeContext.setDebugDirectory("Debug");

graph = TransformationGraphXMLReaderWriter.loadGraph(is,runtimeContext.getAdditionalProperties());
EngineInitializer.initGraph(graph, runtimeContext);
Future<Result> futureResult = runGraph.executeGraph(graph, runtimeContext);

[avackova]

Hello,
unfortunately it is not possible to change CloverETL temp directory. The only way to do this is to run java with the -Djava.io.tmpdir=/opt/tomcat/ option.

[angel]

Hello,

Is there a way to redirect the console output to a file or just set to loglevel to ERROR.

Tried setting it to
runtimeContext.setloglevel(Level.OFF|LEVEL.ERROR);

It seems no effect and the logging start from the EngineInitializer


Thanks,
Angel

[avackova]

Hello Angel,
CloverETL uses log4j for logging. You can create your configuration file (see Logging) and use it in your program:

Code: Select all

PropertyConfigurator.configure("<path_to_your_log4j_properties_file>");