Graph executed many times leaks memory

Discussion on developing CloverETL engine, transformation components etc.

esegal
Posts: 6
Joined: Wed Jun 18, 2014 9:58 am

Graph executed many times leaks memory

Postby esegal » Sun Jun 22, 2014 6:42 pm

Another thing please - my application run Clover Graph multiple times - mainly for aggregation purpose.
As I see by JConsole the application consume more and more memory until I get:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at org.jetel.graph.DirectEdge.init(DirectEdge.java:107)
at org.jetel.graph.Edge.init(Edge.java:265)
at org.jetel.graph.Phase.init(Phase.java:147)
at org.jetel.graph.TransformationGraph.init(TransformationGraph.java:463)
at org.jetel.graph.runtime.EngineInitializer.initGraph(EngineInitializer.java:176)

When I set the JVM for enlarge the Direct buffer memory I get the same result - but after more time.
I took dump file of the application after running a while and I see 41,040 instances of org.jetel.component.DataReader (!!!)
- Well something is malfunction here - I think I use Clover as needed - mybe I'm wrong ?
my code call EngineInitializer.initEngine() only once, and after each graph running call also:
Map<String,Node> nodes= graph.getNodes();
for (Node node : nodes.values())
node.free();
graph.free();

Is any one know about memory-leak in CloverETL?? or am I doing something wrong ??

Thanks in advance !
Segal

jurban
Posts: 162
Joined: Fri Jul 20, 2007 9:25 am

Re: Graph executed many times leaks memory

Postby jurban » Thu Jun 26, 2014 10:48 am

Hi esegal,

can you please post the graph you're using? Then we'll analyse it.

Regards,
Jaro
Jaroslav Urban
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

esegal
Posts: 6
Joined: Wed Jun 18, 2014 9:58 am

Re: Graph executed many times leaks memory

Postby esegal » Thu Jun 26, 2014 3:10 pm

Hi,
Please find attached the Graph I use.
Thanks !
Segal
Attachments
AGGREGATE_NRTRDE.grf
Clover Graph that my application uses
(5.53 KiB) Downloaded 94 times

jurban
Posts: 162
Joined: Fri Jul 20, 2007 9:25 am

Re: Graph executed many times leaks memory

Postby jurban » Thu Jun 26, 2014 4:07 pm

I didn't find anything suspicious in the graph. However, you're running the graph multiple times - how are you doing that? Are you running it from your own Java code? You might be not calling proper cleanup after the graph execution (e.g. you should call graph.free()).
Jaroslav Urban
CloverCARE Support
CloverETL | Rapid Data Integration

Visit us online at http://www.cloveretl.com

esegal
Posts: 6
Joined: Wed Jun 18, 2014 9:58 am

Re: Graph executed many times leaks memory

Postby esegal » Mon Jun 30, 2014 10:53 am

The graph is running multiple times by Java code.
I mentioned before that I call graph.free(), and even free() for each node in graph...
I'm trying to compile Clover code in order to try to debug what's happening .
I'll update what's going on.
Thanks !

mahendran
Posts: 1
Joined: Tue Dec 29, 2015 11:49 am

Re: Graph executed many times leaks memory

Postby mahendran » Tue Dec 29, 2015 11:53 am

Hi esegal,

Can you update us on what issue you found and how you fixed it.