Line Endings get eaten?

[ryanfaerman]

For some reason, whenever I connect to the FTP server running from IIS on a Windows Server, my line endings go away. I open files, and the entire thing is a single line.

My old FTP client... Does not have this issue. It shows the same files as having file endings.

When I connect to various Linux Servers, this is not an issue.

Recreating all the line endings is no fun.

[JD]

This is most definitely a transfer encoding issue, and it sounds like all your files are being sent in Binary format, which would explain why your files are correct when uploaded to a Linux server, when they should be sent in ASCII format.

Basically, the problem is this:

Your files have line endings which are denoted by an LF character - the Unix/Linux/OSX standard
Windows files have line endings which are denoted by both a CR and LF character pair

Uploading in Binary encoding transfers the files 'as is' so you files with LF as the line ending will work fine when sent to a Linux server, but will appear as one big line when sent to a Windows server.

So, the question is: have you manually selected Binary encoding? Look at the menu Transfer -> Encoding

By default, Automatic encoding is set, which will select ASCII encoding automatically if the name suffix of the file (ie .txt) indicates that the file is a text file - the Suffix Mapping table (Transfer menu -> Suffix Mapping) is used to determine how  files with particular name suffixes should be transferred.

If your files have no suffix then there's no way for the Automatic encoding setting to determine the correct encoding to use, so Binary is used in that case.

Your old FTP client either must be set to ASCII (Text) encoding already or has a file suffix mapping that Yummy doesn't have set in it's Suffix Mapping table.

Does this make sense to you?

[ryanfaerman]

The suffix map shows that ASCII encoding is selected for the suffix in use (PHP).

Is it possible my old client (Cyberduck) was doing things in binary the entire time? Or am I missing a piece of the puzzle.

[JD]

Ok, the Suffix Map entry is certainly correct. But do you have Automatic selected as the Encoding method?

Check from the menu bar : Transfer -> Encoding -> ?

If you don't have Automatic selected then the Suffix Mapping table isn't used - the encoding that you have selected in the menu is used instead. Is it set to Binary? It should be set to Automatic

I don't think that CyberDuck would have been set to Binary, otherwise you would have had the problem with line endings before.

The other question I have is - are you using SFTP or FTP?

If you're using SFTP, you need to check Preferences -> Transfers -> SFTP Text Upload Mode

[ryanfaerman]

I have Automatic set at the encoding method.

I am using FTP for both of the servers I am testing this against. SFTP on my Linux server does not have this issue. But then again, FTP with a Linux server doesn't have this issue either.

[JD]

That's bizarre! But it must be an isolated case or a misconfiguration, otherwise a lot of people would be reporting problems by now

Ok, please connect to your Windows server, double check when connected that Automatic is set as the encoding method (because this setting is remembered on a per-Bookmark basis), then upload your .php file. When the upload is complete and you have verified that the line endings are incorrect on the server, look at the FTP Transcript (View menu -> Show FTP Transcript) and scroll up to where the file was transferred. Has Yummy issued a TYPE A command prior to uploading the file?

If you're unsure, send me the FTP Transcript (File menu -> Save FTP Transcript) and tell me the name of the uploaded file.

Thanks, and sorry for the trouble

[ryanfaerman]

Yes!

Right as I connect, before the LIST command even goes to the server, the TYPE gets set to A.

What does this mean?

I just uploaded the file I downloaded using Cyberduck, and the line endings were preserved. I overwrote the file.

ie. i downloaded index.php with cyberduck (which locked up finder a few times) then uploaded it with yummy - and line endings stayed intact.

Here is the excerpt, if you want the entire thing, please just let me know.

...

SYST
215 Windows_NT
SITE DIRSTYLE
200 MSDOS-like directory output is off
SITE DIRSTYLE
200 MSDOS-like directory output is on
CWD /www/
250 CWD command successful.
PWD
257 "/www" is current directory.
TYPE A
200 Type set to A.
PASV
227 Entering Passive Mode (64,34,166,113,15,14).
LIST

...

[JD]

Ok, now you have me really confused! 

So, as far as I can see, Yummy uploaded the file correctly? Then the file must have been damaged in some way before you uploaded it previously when got the line ending problem, or you uploaded it with the wrong encoding.

For what it's worth, TYPE A means "set the encoding to ASCII" but actually, I wanted to see the commands up to and including the STOR index.php command, so that I could verify that TYPE A was set before uploading it. If no TYPE A was sent prior to the upload (STOR command) then the last set TYPE would be in force - probably A because of the LIST command.

So has the problem disappeared now, or what?

[ryanfaerman]

The problem is gone, and a co-worker pointed out a good way to fix this.

Set my transfer type to Binary, download the files. Use SubEthaEdit to convert all the line endings to Unix ones, then upload them binary. Afterwards, I should be able to set it back to automatic.

[JD]

Ok good

I was trying to resolve the problem with the assumption that your original files were ok. It appears that wasn't the case, so your colleague's solution will likely iron out any further anomalies.

Please post back if you have any more problems.

Thanks,
Jason

[scott]

I'm actually having the same problem. My coworker too. The files are file on the server, I've checked with both Cyberduck and VI, it's just Yummy that seems to kill these endings.

I've gone through and checked the settings as described above, and I'm set up correctly. Not too sure what else could be wrong...

[JD]

Please check in CyberDuck's transcript whether it is downloading the files in Binary or Text mode.

If Binary is being used you'll see TYPE I
If Text is being used you'll see TYPE A

I'm assuming you're using FTP?

[JD]

Just to follow up while waiting for scott ...

I checked out CyberDuck: It appears to transfer everything in Binary mode by default, which is why there is now this problem. The FTP protocol dictates that text files should be sent in ASCII mode in order to preserve the correct line endings at the revieving end.

So, to sum up, CyberDuck is totally quackers!   ... sorry, couldn't resist

[hutmark]

This is the major issue that has prevented me from purchasing Yummy FTP. I'm using OSX and I save all my text files with Unix line endings using TextMate.

Nearly all the servers I work on are linux based. YummyFTP ate my line endings. With LF locally and remotely binary/ascii shouldn't differ. Straight binary transfers using Cyberduck have been working flawlessly.

I think that it only happened when I was using the edit feature. I'm sorry I can't confirm... my trial ran out.

On a side note, I wasn't able to locate a place to specify extensions for files that should receive ASCII transfers... but as I've said that's a mute point. I transfer in binary and it works fine. I use Cyberduck, ncftp, and others... and the only trouble I've had with line endings getting eaten is with Yummy.

Very interested in developments on this issue. Glad to see the problem isn't isolated.

[JD]

Hi hutmark,

I have to say that I find it difficult to believe that there is a bug in Yummy FTP in this area. If there was, I think I would be out of business by now

That said, you're obviously seeing some kind of problem, so I definitely want to know more.

If you install the 1.5.2 version of Yummy FTP you will get another 30 days trial time. Please see if you can reproduce the problem and we'll take it from there.

One thing that strikes me as odd in your report is that your servers are all Linux-based. Both the other posters reported that the problem only occurred when connecting to Windows-based servers. This is important because, as you note, on a Linux server it doesn't matter if you transfer text files in Binary mode so long as it has LF line endings, since that is its natural line ending format. In the case of the Windows server, which has a CRLF line ending format by default, it seems that the server strips singular LF line endings while sending files in ASCII mode.

If anyone else has this problem with disappearing line endings and you're able to grant me access to your server to see it first hand, please contact me.

Thanks,
Jason