Search found 20 matches

by KMac21
Fri Sep 08, 2017 5:34 pm
Forum: CloverETL Support
Topic: Input Port 0 still contains some unread records
Replies: 1
Views: 120

Input Port 0 still contains some unread records

I am receiving an error I have not seen before: DBINPUT_TABLE finished with status ERROR. Component has finished and input port 0 still contains some unread records. Contents of my graph: I have two DBINPUT_TABLE components that each get a max date of the same table in different databases. I then us...
by KMac21
Thu May 04, 2017 4:29 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

I cannot attach the call_id to the end of the URL because it doesn't recognize it as a valid URL when I try to pass it. Also, is your example missing a LookupJoin component in order to get the values from the lookup table?
by KMac21
Wed May 03, 2017 11:04 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

I'm not sure how the components access the lookup table after it has been created in your example. It doesn't look like you are using it for the FlatFileWriter.
by KMac21
Wed May 03, 2017 7:38 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

So the metadata that I want to pass through does not come from the URL Response. The metadata is passed in with the URL at the output of the "Build URL" component into the HTTPConnector. So the metadata coming from the Build URL component would look something like: URL api_key game_name fi...
by KMac21
Wed May 03, 2017 4:33 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

Hi Pavel, The files are not already in S3, my goal is to get them into S3. This is the root of the problem I am having: being able to download the files from API URLs and get them into multiple S3 locations. Also, the directory in S3 does not exist ahead of time. This is why along with the large fil...
by KMac21
Tue May 02, 2017 4:04 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

So we increased the Java heap space on the server to 2 GB and that did not work. We are on a 32 bit machine so I'm not sure it would be smart to increase that any further. I'm not sure I can use the FlatFileReader for what I would like to accomplish. I extract the URLs from the downloaded JSON since...
by KMac21
Mon May 01, 2017 9:46 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

I ran the graph in my designer in a Server sandbox as well as on my designer not on the server. This will ultimately be on the server. I'm not sure the link you gave was the correct one? Did you mean http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.server.docs/docs/tom...
by KMac21
Mon May 01, 2017 8:16 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

Re: BufferOverflowException with Large Files

So one thing that did help is that we did not have the metadata set correctly which I have since changed and that helped the graph make more forward progress than it did before. However, I still have files that one file alone is larger than the RECORD_LIMIT_SIZE. Based on your screenshots and aside ...
by KMac21
Wed Apr 26, 2017 7:03 pm
Forum: CloverETL Support
Topic: BufferOverflowException with Large Files
Replies: 15
Views: 1105

BufferOverflowException with Large Files

A little background about what I am trying to accomplish in a certain portion of my graph (we are on 4.4.0.11): I have multiple URLs that each downloads a zipped data file. I am passing the URLs to the HTTPConnector component which I am then passing the content to a UniversalDataWriter to be written...
by KMac21
Thu Apr 06, 2017 7:13 pm
Forum: CloverETL Support
Topic: Generating Dates for the Last 30 Days
Replies: 2
Views: 268

Re: Generating Dates for the Last 30 Days

This was exactly what I was looking for. Thank you for your help!
by KMac21
Tue Apr 04, 2017 9:34 pm
Forum: CloverETL Support
Topic: Generating Dates for the Last 30 Days
Replies: 2
Views: 268

Generating Dates for the Last 30 Days

I want to generate a list of dates to pass to a URL for a total of 30 days ending with yesterday's date in the format YYYY-MM-DD. I don't know CTL that well so it probably can be done withing a DataGenerator by starting with today - 1 then keep subtracting 1 day until you have a total of 30 days, bu...
by KMac21
Thu Feb 23, 2017 7:52 pm
Forum: CloverETL Support
Topic: Parameterizing Several Lists to Build a URL
Replies: 7
Views: 499

Re: Parameterizing Several Lists to Build a URL

The setting "Partition file tag: Key file tag" worked. The reason I thought I was getting duplicate files was that even though I removed them from designer it had not completely synced with the server and the files showed up again so I thought they were being duplicated. Thanks for the hel...
by KMac21
Thu Feb 23, 2017 7:15 pm
Forum: CloverETL Support
Topic: Parameterizing Several Lists to Build a URL
Replies: 7
Views: 499

Re: Parameterizing Several Lists to Build a URL

So I am almost there! I added the setting "Partition file tag: Key file tag" to the UniversalDataWriter properties while keeping the other settings mentioned above. This works and gives me the file with the correct file name, but it also gives me duplicate files of everything except that t...
by KMac21
Thu Feb 23, 2017 6:34 pm
Forum: CloverETL Support
Topic: Parameterizing Several Lists to Build a URL
Replies: 7
Views: 499

Re: Parameterizing Several Lists to Build a URL

Here is what I tried that has not worked: In the HTTPConnector in Output Mapping: function integer transform() { $out.0.content = $in.1.content; $out.0.FileName = concat($in.0.game,"_",$in.0.kpi); return ALL; } I am connecting the HTTPConnector to the UniversalDataWriter component with the...
by KMac21
Wed Feb 22, 2017 7:55 pm
Forum: CloverETL Support
Topic: Parameterizing Several Lists to Build a URL
Replies: 7
Views: 499

Re: Parameterizing Several Lists to Build a URL

That worked beautifully! Thank you! I have one last question. Because I was using parameters before I was writing the output file from the HTTPConnector as ${game}_${kpi}.json, but now that I am not using parameters so how can I get the same result?

cron