Search found 25 matches

by svecp
Thu May 04, 2017 6:28 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

Alright, that may complicate things a tiny bit, but that's not necessarily a problem. To have more elegant solution, I'd probably need to know what exactly is happening in these:
url_prep.png
url_prep.png (64.59 KiB) Viewed 226 times
by svecp
Thu May 04, 2017 1:37 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

Correct, that is suppose to be done in "Build target filename", using CTL lookup functions.
by svecp
Wed May 03, 2017 8:31 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

Sounds good. Just a small alteration should do the trick. Notice two new fields in proposed metadata.

KMac21_download_files.grf
(9.47 KiB) Downloaded 25 times
by svecp
Wed May 03, 2017 7:08 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

That should with small modification work if I understand it correctly. I stitched together a graph with some notes to better understand. Hopefully reflects your situation accurately, if not, let's talk about more context in terms of where and how those files can be downloaded, if they are on a secur...
by svecp
Tue May 02, 2017 11:27 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

Well, if your files are on S3 datastore, then you might have it really simple. Because you can construct an S3 URL and pass it to whatever reader you need to use. So for example, that could be XMLExtract if file you have on your S3 is a zipped XML file. There's a bunch of options, all covered here: ...
by svecp
Tue May 02, 2017 11:14 pm
Forum: CloverETL Support
Topic: Normalized Data to Denormalized Number Columns - Use Denormalize Component?
Replies: 2
Views: 337

Re: Normalized Data to Denormalized Number Columns - Use Denormalize Component?

Well, this is fairly easily achievable by Denormalizer , as you suggested. If you can rely on number and structure of input records - which won't exceed projected output record's width and will have fields specified in correct order, you should be able to use following code (assuming all fields are ...
by svecp
Mon May 01, 2017 10:53 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

No, link I sent is correct. Actually the most important part is the -Dclover.engine.config.file=/full/path/to/file.properties and the actual property which should be set in that file, i.e. Record.RECORD_LIMIT_SIZE . The first snippet specifies which attribute needs to be passed to the java process (...
by svecp
Mon May 01, 2017 8:34 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

So if I understand correctly, you're running those graphs on the server? In such a case (given you're using Tomcat installation), change path to engine properties file in: <tomcat home>/bin/setenv.sh (or .bat) as described here: http://doc.cloveretl.com/documentation/UserGuide/topic/com.cloveretl.gu...
by svecp
Sat Apr 29, 2017 3:37 am
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1108

Re: BufferOverflowException with Large Files

Well your first solution should've worked. I just tried to download 120 MB using following configuration: ffr_http_read.PNG There is also solution I'd prefer when dealing with this kind of use case - read it directly using FlatFileReader. Case below illustrates how you can use FlatFileReader to down...
by svecp
Tue Apr 25, 2017 3:41 pm
Forum: CloverETL Support
Topic: Modifying existing input table for output table
Replies: 1
Views: 213

Re: Modifying existing input table for output table

Yes, solution might look like the one attached. In case incoming data are sorted, you can lose the ExtSort component.
by svecp
Tue Apr 25, 2017 3:11 pm
Forum: CloverETL Support
Topic: Having a changeable database connection login parameters
Replies: 1
Views: 1834

Re: Having a changeable database connection login parameters

Yes, you can use (externalized) parameter files. For example, you can have parameters DB_USER, DB_PASSWORD, DB_URL, ... which they can be referenced in that connection using expressions ${DB_USER}, ${DB_PASSWORD}, ${DB_URL}, ... This (external) parameter file has to be linked to the graph, where you...
by svecp
Tue Apr 25, 2017 2:51 pm
Forum: CloverETL Support
Topic: ComplexDataReader Java Issue
Replies: 6
Views: 536

Re: ComplexDataReader Java Issue

Well, if you set JAVA_HOME to setenv.sh file and echo $JAVA_HOME, you will not see any output since whenever startup.sh is executed, it starts in a new terminal session. If you'd like to add it globally, at least on a Linux machine, you'd need to add it into /etc/profile.d directory or /etc/profile ...
by svecp
Tue Apr 25, 2017 2:28 pm
Forum: CloverETL Server
Topic: Using allocation with DBInputTable
Replies: 5
Views: 554

Re: Using allocation with DBInputTable

There might be a major performance improvement in case that particular table is partitioned. You can query each partition separately, using "PARTITION" clause and instead of using ROW_NUMBER() you can select the whole partition. That should lower DB overhead considerably. See https://docs....
by svecp
Mon Mar 20, 2017 7:15 pm
Forum: CloverETL Support
Topic: MoveFiles error
Replies: 2
Views: 443

Re: MoveFiles error

Hi, are you using embedded browser to select path on a remote server, or you're copy&pasting address? Error you're getting might relate to file system restrictions. What kind of remote system we're talking about? Linux? Windows? The thing is, when you're trying to reach a directory let's say &qu...
by svecp
Fri Mar 17, 2017 1:51 am
Forum: CloverETL Support
Topic: warning converting clover type: date to sql type: DATE
Replies: 2
Views: 457

Re: warning converting clover type: date to sql type: DATE

Hi, I came across exactly the same thing a while back, wondering what is wrong. The answer is actually in PostgreSQL documentation - PostgreSQL's DATE is a plain date information without any notion about the time. CloverETL's date is on the other hand essentially timestamp. The warning basically mea...

cron