SFTP is the FTP subsystem that is part of SSH, it allows you to do encrypted file transfers. In the past week I have had 2 issues about SFTP not properly transferring files. In one case OpenVOS was the source of  the file and in the other case OpenVOS was receiving the file. In both cases the files contained ASCII data and the problems had to do with the differences between how Microsoft Windows and OpenVOS terminate lines in a text file.

Let’s start with OpenVOS as the source. If you create a text file using your favorite editor you create a sequential file. For example

d test
%phx_vos#m16_mas>SysAdmin>Noah_Davids>test  10-09-05 08:13:49 mst
12345
67890

ready 08:13:49

Figure 1 – display of a text file
The display_file_status command shows that it is a sequential file and that while the data byte count is 10 there are actually 20 bytes in the file.
display_file_status test
name: %phx_vos#m16_mas>SysAdmin>Noah_Davids>test
file organization: sequential file
. . .
next byte:                 20
blocks used: 1
. . .
record count: 2
data byte count:           10

ready 08:14:01

Figure 2 – display_file status
The dump_file command shows the actual structure of the file. Each “line” is preceded and terminated with 2 bytes that indicate its length. These length bytes have to start on an even byte boundary so there is also a filler byte. The significant thing to notice is that there are no typical line termination characters, either a line feed (0x0A) or a carriage return line feed (0x0D0A) sequence.
dump_file test%phx_vos#m16_mas>SysAdmin>Noah_Davids>test  10-09-05 08:14:07 mst
Block number         1000  00053132 333435FF 00050005 36373839  |..12345…..6789|010  30FF0005 FFFFFFFF FFFFFFFF FFFFFFFF  |0……………|

020  FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |…………….|

=

FF0  FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |…………….|

ready  08:14:07

Figure 3 – dump_file showing the actual structure of the file

 

When you transfer the file using SFTP, OpenVOS strips the length bytes from the line and appends a line feed character. The result is that while the original file contained 10 data types the transferred file contains 12.
C:Documents and SettingsnoahMy Documentstemp>"C:Program FilesPuTTYpsftp" nd@164.152.77.128
Using username "nd".
nd@164.152.77.128's password:
Remote working directory is /SysAdmin/Noah_Davids
psftp> get test test.txt
remote:/SysAdmin/Noah_Davids/test => local:test.txt
psftp> quit
C:Documents and SettingsnoahMy Documentstemp>dir
Volume in drive C has no label.
Volume Serial Number is 38B1-9C13
Directory of C:Documents and SettingsnoahMy Documentstempblog - sftp
08/27/2010 01:50 PM <DIR> .
08/27/2010 01:50 PM <DIR> ..
08/27/2010 01:50 PM 12 test.txt
1 File(s) 12 bytes
2 Dir(s) 39,471,644,672 bytes free
Figure 4 – Transferred file length doesn’t match source file length
In addition, if you display the file in something like Notepad, it displays as one line with funny symbols where the lines should terminate.
Figure 5 – display of file in Notepad
If you use a utility to get a hex dump of the file you will see that at the end of each line there is a line feed character (0x0A). The standard Microsoft Windows line termination sequence is carriage return line feed (0x0D0A), Notepad doesn’t know how to interpret just the line feed character, although there are other editors that will display the file correctly.

 

Figure 6 – Hex dump of the transferred file
The result is that the transferred file appears to be corrupt, the length is different and you cannot display it – at least in Notepad. Converting the file to a stream file before transferring it will correct the length issue; both files will now show the same length but the line termination issue will remain.

 

Now what if OpenVOS is receiving a file from a Microsoft Windows System? As I already said, Microsoft Windows uses a carriage return line feed sequence to terminate a line.
Figure 7 – Hex dump of a file created on a Microsoft Windows system
The OpenVOS SFTP subsystem will create a stream file. The line feed characters will be taken as line termination characters and the carriage return characters will be treated as part of the line’s data.

dump_file pc1.txt

%phx_vos#m16_mas>SysAdmin>Noah_Davids>pc1.txt  10-09-05 08:52:21 mst
Block number         1

000  61626364 650D0A66 6768696A 0D0AFFFF  |abcde..fghij….|

010  FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |…………….|

=

FF0  FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |…………….|

ready  08:52:21

Figure 8 – dump file of the transferred file
The display command will display the file correctly.

d pc1.txt

%phx_vos#m16_mas>SysAdmin>Noah_Davids>pc1.txt  10-09-05 08:54:25 mst

abcde

fghij

ready  08:54:25

Figure 9 – display of the transferred file
But when you go to actually read the file the carriage return character will show up at the end of every line.
test_system_calls
tsc: s$attach_port p pc1.txt
tsc: s$open
tsc: s$seq_read p
Buffer length = 6
00000000 61626364 650D |abcde. |
tsc: s$seq_read p
Buffer length = 6
00000000 66676869 6A0D |fghij. |

tsc:

Figure 10 – carriage return characters show up when you read the file
Again, the result is that the file appears to be corrupt.
The following Perl script can be used to add or remove the carriage return characters. You want to add them before transferring the file to a Microsoft Windows system and remove them after transferring the file from the Microsoft Windows system.
# cr.pl begins here
#
# cr
# version 1.0 10-08-27
# Noah.Davids@stratus.com
#
use strict;
use warnings;
use Getopt::Long;
my ($inFile, $outFile, @files, $add, $remove);
my ($INFILE);
my ($result, $count, $verbose, $addremove);
$result = GetOptions ('in=s' => $inFile,
'out=s' => $outFile,
'add' => $add,
'remove' => $remove,
'verbose=s' => $verbose);
if (($result != 1) || !defined ($inFile) || !defined ($outFile))
{
print "nnUsage:n";
print "perl cr.pl -in PATH -out PATH [[-add] | [-remove]] [-verbose]}n";
exit;
}
if (defined ($add) && defined ($remove))
{
print "You can only specify -add or -remove not bothnn";
print "nnUsage:n";
print "perl cr.pl -in PATH -out PATH [[-add] | [-remove]] [-verbose]}n";
exit;
}
@files = glob ($inFile);
if (@files < 1) {print "nNo files found for " . $inFile . "nn";}
if (@files > 1) {print "nMore than 1 file found for " . $inFile . "nn";}
open (OUT, ">".$outFile) || die "Can't open output file " . $outFile . "nn";
open ($INFILE, $files[0]) || die "Can't open input file " . $files[0] . "nn";
if (!defined ($verbose)) { $verbose = -1; }
$count = 0;
while ($_ = <$INFILE>)
{
if (defined ($remove))
{
s/r//;
print OUT $_ ;
}
else
{
s/n//;
print OUT $_ . "rn";
}
$count++;
if (($verbose > 0) && ($count % $verbose) == 0)
{ print "Line " . $count . " of " . $files[0] . " processedn"; }
}
close $INFILE;
#

# cr ends here

Figure 11 – Perl script to remove or add carriage return characters
Why does FTP seem to work? FTP has two modes, ASCII and binary. In ASCII mode transfers from a Microsoft Windows system to OpenVOS result in a sequential file with the carriage return and line feed characters both removed from the records. Binary mode transfers from the Microsoft Windows system to Open VOS result in a stream file with the carriage return and line feed characters present – exactly like the binary mode transfer of SFTP. ASCII mode transfers from OpenVOS to a Microsoft Windows system result in a file with carriage return and line feed characters at the end of each line while binary mode transfers result in a file with just the line feed character at the end of each line. Again just like binary mode transfer of SFTP. The FTP RFC goes into detail about how to terminate lines but the bottom line is that in ASCII mode both client and server are free to terminate the lines as necessary for the local operating system while in binary mode client and server have no such discretion.

© 2019 Stratus Technologies.