Subject: Re: Attachments via ange-ftp

From: Andrew Innes <[email-masked]>
Date: Tue, 8 Apr 1997 13:29:59 +0100 (BST)

On Tue, 1 Apr 1997 13:42:36 -0600, [email-masked] (Douglas Gray Stephens) said:
>I suspect that my problem is PC related, but I'm not sure if it
>can/should be fixed in VM, or nt-emacs, hence I'm cross posting this
>to [email-masked] to see if the nt-emacs side have
>any suggestions.

Yes, this problem is PC specific (for the most part).

On Tue, 1 Apr 1997 21:34:19 -0500 (EST), Kyle Jones <[email-masked]> said:
>Douglas Gray Stephens writes:
>>[...]
>>This ^M will be causing vm to encode the message in base64.
>>
>>I am not sure why you've used
>>insert-file-contents-literally
>>instead of
>>insert-file-contents
>
>To avoid problems with file handlers uncompressing or otherwise
>fiddling with the input.  Maybe this is the wrong thing to do.
>I'm willing to switch to insert-file-contents and see if that
>works better.

Given that we are talking about including files as MIME attachments, I
think using insert-file-contents-literally is, in principle, the right
thing to do; the "problem" in this context is that it disables the
(imperfect) file type detection code used on Windows as well as
inhibiting the various handlers and hook functions.

Strictly speaking, if the original file uses DOS line endings, then that
is what should be transmitted (in base64 encoding if required).
However, if it is simply a plain text file, it would generally be more
helpful to treat it as such, and convert it to whatever line ending
convention is most suitable - in this case, convert to Unix line endings
so that the contents are transmitted in the clear.

So, although insert-file-contents-literally is strictly correct, in this
instance it would be more helpful to use a modified version which only
inhibits the handlers and hook functions, but leaves the file type code
in place.  Such a change should be safe to make, since it will only
affect Windows where it will generally do the right thing.

Aside:

The whole issue of how text files are handled, by the DOS and Windows
ports of Emacs at least, is really overdue for a major rethink.

The current method for determining whether a file is text (implicitly
meaning DOS text) or binary is based on regular expression matching
against the file name.  This leads to all sorts of hassles, most of
which could be easily avoided by using a simple content scanning
heuristic to identify whether a file is text or binary, and the line
ending convention (DOS, Mac, Unix) if text.

Personally, I would like to see this heuristic incorporated into Emacs
(on all platforms, not just DOS and Windows) - it would make editing and
manipulating text files from different sources mostly transparent.  I
don't know how likely it is this will happen though, since the Mule
capabilities currently being added to Emacs (which must deal with the
more general language/charset encoding properties of files and other
data streams) will probably subsume this issue, and may do so in a
completely different and more general way.

Still, I expect that the line ending convention is usually orthogonal to
charset encoding, so maybe there is a chance to do this anyway.

AndrewI

---

From: "John R. Dennis" <[email-masked]>
Date: Tue, 8 Apr 1997 12:06:41 -0400 (EDT)

>>>>> "Andrew" == Andrew Innes <[email-masked]> writes:

    Andrew> Given that we are talking about including files as MIME
    Andrew> attachments, I think using insert-file-contents-literally
    Andrew> is, in principle, the right thing to do; the "problem" in
    Andrew> this context is that it disables the (imperfect) file type
    Andrew> detection code used on Windows as well as inhibiting the
    Andrew> various handlers and hook functions.

    Andrew> The whole issue of how text files are handled, by the DOS
    Andrew> and Windows ports of Emacs at least, is really overdue for
    Andrew> a major rethink.

I cannot believe how topical this issue is. I just spent all Friday
morning debugging a similar problem in mime.el.

Even though I had set all the variables I knew of that caused CRLF
translation when inserting into a buffer...

      (let ((start (point))
	    (emx-binary-mode t)		;Stop LF to CRLF conversion in OS/2
	    (buffer-file-type t)	;Stop LF to CRLF conversion in DOS/NT
	    (binary-process-input t))	;Stop LF to CRLF conversion in DOS/NT

the conversion was still happening because in fileio.c the
implementation of insert-file-contents overwrites the user supplied
value of buffer-file-type:

    current_buffer->buffer_file_type
      = call1 (Qfind_buffer_file_type, filename);

The elisp code knew it wanted to insert the contents of the file as
binary so it explicitly set buffer-file-type, but the implementation
of insert-file-contents ignored that setting and tried to determine
the translation mode by a regular expression match on the filename. I
fixed the problem by calling insert-file-contents-literally which
undefines find-buffer-file-type so the call in insert-file-contents to
find-buffer-file-type won't succeed. But I don't think the C code in
insert-buffer-contents should ignore the documented variable
(buffer-file-type) that is supposed to toggle the CRLF translation!

All of this is pretty ugly, prone to failure, and more to the point
undocumented for the most part as far as I can tell. After spending
the better part of day digging through the binary vs. text issues I
was left with the distinct impression that most of this code is a
"hack" waiting to break. I absolutely agree with Andrew that this is
in need of a major rethink.

To begin the discussion I will make the following observations:

* Determining binary/text based on regular expression matching of filenames
  is fundamentally flawed. There is not enough naming discipline with
  filenames and extensions to make this work reliably. I have been
  burned by this more times than I care to remember.

* The only way to tell if a file is binary is to scan the file and
  look for non-ascii bytes.

* The documentation on the text/binary issues is woefully inadequate
  and the implementation is inconsistent.

* The binary/text translation should be controlled by a user settable
  variable that is ALWAYS respected. After all, the user is ultimately
  more knowledgable about the contents of a file than the
  implementation.

* There should be second user settable variable that toggles whether
  translation variable is automatically set based on the contents of
  the file. In this way you get automatic translation in the 99% of
  the cases you want it AND you can force the translation on/off when you
  have to.

* We'ed all be happier without operating systems that make the
  artifical distinction between text and binary files and attempts to
  insert/delete/modify bytes that are not in the actual file to undo
  the damage introduced by this ill-conceived distinction in the first
  place (sorry, this last point was a completely personal soapbox
  comment :-)

John Dennis