In my last blog I talked about automating file transfers using FTP. There are three issues with using FTP. First, your password is sent across the network in clear text making it available to anyone with a protocol analyzer. Second, your data is also sent in clear text. Third, you have to record your password somewhere on the module, either in the macro or in the .netrc file. Because of these issues many sites have mandated or are thinking of mandating that SFTP, the secure FTP subsystem of SSH, be used instead of FTP. However, you just cannot replace ftp with sftp in your command macro (figure 1) and expect it to work (figure 2).

 

&attach_input
sftp 172.16.1.116
nd
MYPASSWORD
put foo
&if (command_status) ^= 226 &then &goto ERROR1
get bar
&if (command_status) ^= 226 &then &goto ERROR2
quit
&
&label ERROR1
..display_line Could not put file
quit
&return
&
&label ERROR2
..display_line Could not get file
quit
&return

Figure 1 – sftp1.cm – replacing the ftp command with sftp in a command macro

 

sftp1
Connecting to 172.16.1.116...
dup2: Bad file number.
Error on line 3 of sftp1.
command_processor: Object not found. nd. In command macro
%phx_vos#m15_mas>SysAdmin>Noah_Davids>sftp1.cm
ready  12:39:59

 

Figure 2 – running sftp1.cm
The problem is an incompatibility between the way that native VOS commands and POSIX open source treat input. However, SFTP, like FTP allows you to create a file with a list of requests (figure 3) which it will automagically run (figure 4).

 


put foo
get bar

 

Figure 3 – sftp2.input – file containing requests for sftp to execute

 


sftp -b sftp2_input nd@172.16.1.116
nd@172.16.1.116's password:
sftp> put foo
Uploading foo to /SysAdmin/Noah_Davids/foo
sftp> get bar
Couldn't stat remote file: No such file or directory
File "/SysAdmin/Noah_Davids/bar" not found.
Ready  13:06:45

 

Figure 4 – execution with an input file of requests and a password prompt
You’ll notice in figure 4 that sftp is prompting for a password. There is no way to put the password in the input file. The ONLY way to bypass the password prompt is to use public key authentication (figure 5).

 

sftp -b sftp2_input nd@172.16.1.116
sftp> put foo
Uploading foo to /SysAdmin/Noah_Davids/foo
sftp> get bar
Couldn't stat remote file: No such file or directory
File "/SysAdmin/Noah_Davids/bar" not found.
Ready  13:12:28

 

Figure 5 – execution with an input file of requests, using public key authentication to eliminate the password prompt

 

You can find an article on how to set up public key authentication here – http://members.cox.net/ndav1/self_published/publickey_authentication_setup.html

Unfortunately, sftp will just execute all the requests in the input file one after another, there is no mechanism for testing if the transfer worked or not.

Since command macros cannot be used are there any alternatives? The gnu_tools product ships with a program called expect. It can be used to send commands, wait for any number of different responds and do something based on what it sees in the output stream. I am no expect expert but the script in figure 6 will help get you started. If you search the web for “expect scripts” you will find many references that will help you customize my simple example.

 

# If we get an end-of-file (eof) it means that the sftp process
# terminated. Report the last command that was sent.
#
proc goteof {cmd} {
puts "sftp connection unexpectedly closed n"
puts "last text received wasn"
puts "<<"
puts $cmd
puts ">>nnn"
exit
}
# This procedure does regular expression matches looking for key strings
# in the output collected from the command that was execured. In this case
# I am just reporting the type of error but other things can be done
# as well. I also only check for 2 errors. There are others. You will have
# to add them as you find them.
#
proc checkforerrors {buf cmd} {
if [regexp {.*not found} $buf] {
puts “$cmd FAILED : not foundnnn”
return 1
}
if [regexp {.*Permission denied} $buf] {
puts “$cmd FAILED : access problemsnnn”
return 1
}
return 0
}

# set the timeout to -1 so there is no timeout. The default is 10 seconds
# and most file transfers take longer than that. I decided to set no
# timeout, you can change that.
set timeout -1

# start sftp
spawn sftp nd@172.16.1.116

# wait for the sftp prompt but if we get an authentication prompt, ending in
# yes/no report a security issue and exit.
expect {
“yes/no?” { puts
“nnnWe have connected to the wrong server or the server has been reloaded.”
puts
“The server key must be validated before this script can be run again.nnn”
exit }
sftp>
}

# change to a convient directory for testing
send “cd sftp_testr”
expect {
eof   { goteof “cd sftp_test” }
sftp>
}

# call check for errors passing it all collected characters up the
# sftp> prompt. Also the command that was sent to sftp. If the
# checkforerrors procedure returns 1 exit the script. Again, you can
# do other things.
if {[checkforerrors $expect_out(buffer) “cd sftp_test”] == 1} { exit }

send “put foor”
expect {
eof   { goteof “put foo” }
sftp>
}
if {[checkforerrors $expect_out(buffer) “put foo”] == 1} { exit }

send “get barr”
expect {
eof   { goteof “get bar” }
sftp>
}
if {[checkforerrors $expect_out(buffer) “get bar”] == 1} { exit }

send “quitr”
expect  eof
puts “script donenn”

 

Figure 6 – sftp3.exp – expect script to automate SFTP

expect sftp3.exp
spawn sftp nd@172.16.1.116
Connecting to 172.16.1.116...
The authenticity of host 172.16.1.116(172.16.1.116)' can't be established
+.
RSA key fingerprint is 37:f4:1a:56:64:af:ab:8a:7c:0b:36:47:c5:6c:1d:1a.
Are you sure you want to continue connecting (yes/no)?
We have connected to the wrong server or the server has been reloaded.
The server key must be validated before this script can be run again.

ready  10:41:10

 

Figure 7 – execution of sftp3.exp expect script with security warning

 

expect sftp3.exp
spawn sftp nd@172.16.1.116
Connecting to 172.16.1.116...
sftp> cd sftp_test
sftp> put foo
Uploading foo to /SysAdmin/Noah_Davids/sftp_test/foo
sftp> get bar
Couldn’t stat remote file: No such file or directory
File “/SysAdmin/Noah_Davids/sftp_test/bar” not found.
sftp> get bar FAILED : not found

ready  10:42:11

 

Figure 8 – execution of sftp3.exp expect script without warning but wiuth a failed transfer

Finally, in my FTP blog I mentioned that FTP was able to read files that were still open and this on occasion has resulted in incomplete files being transferred, I suggested that if the FTP macro waits for a file to appear at a location and then transfers it, the macro should check to be sure that it is no longer locked. The same problem can occur with SFTP and the solution is the same. You can place the file lock check in a command macro (figure 9) and then when the file is no longer locked call expect with the appropriate script.

&label AGAIN
&if (exists new_file) = 1 & (locked new_file) = 0 &then &goto EXPECT
display_line new_file not ready for transfer as of (time)
sleep -seconds 15
&goto AGAIN
&
&
&label EXPECT
expect sftp3.exp

 

Figure 9 – command macro testing for file lock before calling expect