MoveFiles error

Support/help with CloverETL implementation problems

anyeone
Posts: 14
Joined: Thu Feb 09, 2017 4:30 pm

MoveFiles error

Postby anyeone » Thu Mar 16, 2017 10:26 pm

Greetings,

I'm trying to use the MoveFiles component in a job flow (in a Server project) to move a file from the data-out directory to a remote SFTP location to archive it.

I am able to create the SFTP connection and it validates. However, when I run the jobflow, I get an error:
Component [MoveFiles:MOVE_FILES] finished with status ERROR. (In0: 1 recs, Out0: 0 recs)
Failed to move sandbox://SDX/./data-out/categories_4_2 ... 00000.json to sftp://{redacted}:***@{redacted}/4/archive/
Move failed
Copy failed
No such file

The file does exist in the data out directory. The 4/archive folder structure does also exist on the SFTP site.
I have tried including a file name as well (i.e. sftp://{redacted}:***@{redacted}/4/archive/test.json) and I get the same error.

If I try to copy to the root directory of the SFTP site I get a different error (permission denied), but anywhere else I try to copy it says no such file.

I have tried including the file both explicitly by selecting it from the URL dialog and by dynamically specifying the filepath (which is how it will work eventually) but both ways it tells me no such file.

Screen Shot 2017-03-16 at 4.24.14 PM.png
Screen Shot 2017-03-16 at 4.24.14 PM.png (248.49 KiB) Viewed 449 times


The stack trace is:
org.jetel.exception.JetelRuntimeException: Component [MoveFiles:MOVE_FILES] finished with status ERROR. (In0: 1 recs, Out0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:636)
at org.jetel.graph.Node.run(Node.java:603)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jetel.exception.JetelRuntimeException: Failed to move sandbox://SDX/./data-out/categories_4_2 ... 00000.json to sftp://{redacted}:***@{redacted}/4/test.json
at com.opensys.cloveretl.component.MoveFiles.fail(Main:187)
at com.opensys.cloveretl.component.AbstractFileOperation.processError(Main:531)
at com.opensys.cloveretl.component.AbstractFileOperation.processResult(Main:563)
at com.opensys.cloveretl.component.AbstractFileOperation.mainExecuteOperation(Main:351)
at com.opensys.cloveretl.component.AbstractFileOperation.execute(Main:381)
at org.jetel.graph.Node.run(Node.java:563)
... 3 more
Caused by: java.io.IOException: Move failed
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:448)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:374)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Main:202)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Main:40)
at com.opensys.cloveretl.component.AbstractFileOperation.mainExecuteOperation(Main:339)
... 5 more
Caused by: java.io.IOException: Copy failed
at org.jetel.component.fileoperation.DefaultOperationHandler.move(DefaultOperationHandler.java:233)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:441)
... 9 more
Caused by: java.io.IOException: Copy failed
at org.jetel.component.fileoperation.FileManager.copy(FileManager.java:354)
at org.jetel.component.fileoperation.DefaultOperationHandler.move(DefaultOperationHandler.java:231)
... 10 more
Caused by: java.io.IOException
at org.jetel.util.ExceptionUtils.getIOException(ExceptionUtils.java:581)
at org.jetel.component.fileoperation.pool.AbstractPoolableConnection.getIOException(AbstractPoolableConnection.java:46)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getOutputStream(PooledSFTPConnection.java:441)
at org.jetel.component.fileoperation.PooledSFTPOperationHandler$SFTPContent.write(PooledSFTPOperationHandler.java:648)
at org.jetel.component.fileoperation.FileManager$WritableContentProvider.channel(FileManager.java:573)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyFile(DefaultOperationHandler.java:74)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyInternal(DefaultOperationHandler.java:148)
at org.jetel.component.fileoperation.DefaultOperationHandler.copy(DefaultOperationHandler.java:189)
at org.jetel.component.fileoperation.FileManager.copy(FileManager.java:347)
... 11 more
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:753)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:700)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:697)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getOutputStream(PooledSFTPConnection.java:428)
... 17 more

I'm a bit stumped so assistance is greatly appreciated!
The only thing that seems odd to me is that there is a . in the path in the error message that doesn't appear in the raw path, i.e.
sandbox://SDX/./data-out/categories_4_2 ... 00000.json
vs.
sandbox://SDX/data-out/categories_4_201 ... 00000.json

Thanks,
Anye

svecp
Posts: 25
Joined: Wed Nov 09, 2016 11:51 pm
Location: 2111 Wilson Blvd., Arlington VA 22201
Contact:

Re: MoveFiles error

Postby svecp » Mon Mar 20, 2017 7:15 pm

Hi,

are you using embedded browser to select path on a remote server, or you're copy&pasting address? Error you're getting might relate to file system restrictions. What kind of remote system we're talking about? Linux? Windows?

The thing is, when you're trying to reach a directory let's say "/my/directory/structure/is/very/deep.txt" and you don't have "execute" permission to directory "structure" at Linux OS, you'll get "directory does not exist" instead of expected "permission denied" error. Could not this be your case?

For example, I get following error:

Code: Select all

--------------------------------- Error details ----------------------------------
  Component [ListFiles:LIST_FILES] finished with status ERROR. (Out0: 0 recs)
   Failed to list ftp://clover:***@griffin/opt/VBoxGuestAdditions-5.0.14/bin
    Directory listing failed
     File not found: "ftp://clover:***@griffin/opt/VBoxGuestAdditions-5.0.14/bin"
----------------------------------------------------------------------------------


When "VBoxGuestAdditions-5.0.14" has following permissions:

Code: Select all

drwxr-xr--.  2 root   root      6 Jun 21  2016 VBoxGuestAdditions-5.0.14


If I:

Code: Select all

chmod o+w /opt/VBoxGuestAdditions-5.0.14

my graph finishes successfully. Using vsFTP@CentOS 7.
--
Pavel Švec | CloverETL | Sales Engineer | 2111 Wilson Blvd | Suite 320 | Arlington, VA 22201

anyeone
Posts: 14
Joined: Thu Feb 09, 2017 4:30 pm

Re: MoveFiles error

Postby anyeone » Mon Mar 20, 2017 8:45 pm

Hi Pavel,
That wasn't it but your suggestion helped me figure it out.

I added a ListFiles component beforehand to try to view the contents of the directory I was trying to write to. It turns on that when I *manually* sftp to that server and log in as the user in question it drops me in that user's folder automatically. However, when the Clover server sftp's there, you have to additionally add the user folder to the path or it is trying to look under the root.

So the problem was that it was actually trying to drop to serverdomain/4/archive rather than serverdomain/userfolder/4/archive
and once I manually added the username folder to the path, it worked.

So thank you very much :)


cron