Subject: New way of handling CRLF

From: Richard Stallman <[email-masked]>
Date: Tue, 1 Jul 1997 20:55:47 -0400

The MULE features include a new way of handling CRLF conversion.
It detects the need to convert CRLF using the same mechanism that
detects the need to convert international character sets.

One consequence of this is that it ought to succeed in editing files
that use LF or files that use CRLF.  Regardless of what type of system
you are on and what type of file system you are using, the file will
appear in the normal Emacs way, with newlines between the lines.

Does this mean that some of the features for text vs binary files
and untranslated file systems are now unnecessary?  Can I simplify
the "Text Files and Binary Files" in the manual?

Please answer me as soon as you can; I am trying to finish the manual.

Note: currently there is a bug: when you visit a file on Unix which
uses CRLF between lines, it recognizes that, but
buffer-file-coding-system is set to nil, which is not right.  I will
forward you the fix for this as soon as I get it.

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 2 Jul 1997 11:03:09 +0300 (IDT)

On Tue, 1 Jul 1997, Richard Stallman wrote:

The following is purely theoretical, based on what you told in your
message.  I didn't have time yet to download and install the pretest,
neither do I know how does MULE detect and convert the file format.

> the "Text Files and Binary Files" in the manual?

I would guess that the manual needs to be changed, but not necessarily
simplified.  The text vs binary thing has two aspects: reading them
into Emacs and writing them back to the filesystem.  No matter how
smart the CRLF detection mechanism is, there will be cases when users
will want a buffer to be written in specific format of their
preference, which might be different from the format of the original
file as read by Emacs.

I'm also not sure that the CRLF detection can be made fully automatic.
Imagine a binary file (like an executable program) that includes a
CRLF pair somewhere; would Emacs 20 strip the CR from it when it reads
that file and treat it as text?

So I think Emacs 20 will need to keep the special varieties of
`find-file' that specify text or binary explicitly (btw, it seems as
if they aren't mentioned anywhere in the 19.34 manual).  There should
also be a way to tell Emacs to write a buffer (or a region) with LFs
translated to CRLFs.  In particular, the (un)?translated filesystem
feature should be kept IMHO.

If the above reasoning is true, there should be minor changes to the
manual (to explain the automatic CRLF detection feature), but the bulk
of the text should be kept.

---

From: Eli Zaretskii <[email-masked]>
Date: Thu, 3 Jul 1997 18:36:11 +0300 (IDT)

On Wed, 2 Jul 1997, Richard Stallman wrote:

> We have two mechanisms for deciding whether a file should have LF, not
> CRLF based on the file name.  One looks for "binary files" and one
> looking for untranslated file systems.
> 
> Could these be unified, I wonder?

On "translated" file systems, Emacs should decide whether the file is 
text (and then convert CRLF -> LF) or binary.

On "untranslated" file systems, all files should be read and written 
verbatim.

---

From: Richard Stallman <[email-masked]>
Date: Wed, 2 Jul 1997 20:39:37 -0400

I agree with Eli that users will still want a mechanism by which files
    are written in a format automatically determined by Emacs.

I agree.

Still, I would really really appreciate it if you would tell
me how things DO work now!  Does Emacs automatically figure
out whether a file has CRLF or LF?

(There is a bug in the pretest that fails to save a file with CRLF if
it was recognized with CRLF.  That has been fixed.)

---

From: Richard Stallman <[email-masked]>
Date: Wed, 2 Jul 1997 20:40:45 -0400

Could these be unified, I wonder?  And could they both be done using
file-coding-system-alist now?

---

From: Richard Stallman <[email-masked]>
Date: Thu, 3 Jul 1997 15:19:06 -0400

That is what it does now--doesn't it?  So what are you trying to say?
Perhaps you misunderstood my question and answered a completely
different one.

Right now we have two separate mechanisms to do two similar jobs.
Can we replace them with one mechanism that can do both jobs?

---

From: Eli Zaretskii <[email-masked]>
Date: Sun, 6 Jul 1997 17:22:00 +0300 (IDT)

> That is what it does now--doesn't it?  So what are you trying to

I was trying to say that the two should be combined.  (un)?translated
says whether the CRLF<->LF conversion is at all an issue, and the
detection of the file type says whether this particular file needs the
conversion, given that it belongs to a "translated" filesystem.

If it already works this way, then my comments are redundant.

---

From: Eli Zaretskii <[email-masked]>
Date: Sun, 6 Jul 1997 17:22:44 +0300 (IDT)

On Fri, 4 Jul 1997, Geoff Voelker wrote:

> correctly (e.g., on a text file with CRLF, both a find-file and a
> find-file-binary create a buffer with the text file stripped of
> CRLF,

What about binary files with embedded CRLFs?  How can Emacs tell which
files are and which aren't ``text''?  If it can't, then the above
behavior is wrong: I might want to use `find-file-binary' to read a
binary file (e.g., an executable program) that just happens to have
embedded CRLF pairs, either as part of text messages or even just an
opcode that happens to look like CRLF.  Will I then be presented with
the file with all CRs in CRLF pairs removed?

> a text file without CRLF, Emacs reads it in correctly, but the
> buffer-file-type is "text", and so the file gets written out with LF
> converted to CRLF.  This is not a bug in the coding-system code, but
> rather due to the fact that, internally, Emacs under DOS_NT looks at
> the buffer-file-type, sees "text", and opens the file in text mode,
> and the operating system changes LF to CRLF.

I'm not sure this is a bug, either.  I can imagine cases where the
user would like Unix-style text files be written as DOS-style text.  I
haven't decided yet what the default should be here, but at least a
user-definable option should be available to get the non-default
behavior.

> Given the new coding-system framework, I think that all file I/O under
> DOS_NT should now be done in binary mode since the data that Emacs
> gives to the operating system does not need any conversion.

If that is the case, how would a user tell Emacs that a file which
originally had no CRs should have them added on output (assuming that
you agree that such cases are possible)?

---

From: Richard Stallman <[email-masked]>
Date: Sun, 6 Jul 1997 20:08:26 -0400

What about binary files with embedded CRLFs?

Specifying that a file is binary means specifying no conversion.
Therefore, CRLF in these files will not be converted.

You can certainly do this by specifying a different coding system
when you save the file.

---

From: Richard Stallman <[email-masked]>
Date: Sun, 6 Jul 1997 04:33:59 -0400

Given the new coding-system framework, I think that all file I/O under
    DOS_NT should now be done in binary mode

Yes, that is true.

If it doesn't work that way now, could someone send me a fix?

    Actually, the new coding-system framework appears to obviate the need
    for buffer-file-type; file-coding-system-alist and
    buffer-file-coding-system appear to be flexible enough to supercede
    it.  I will need to think more about this, though, since it is a
    rather drastic change under DOS_NT.

Another alternative would be to modify some of the Mule functions so
that, on DOS/NT, they look at the same variables which now control the
decision about the buffer file type.  For example, the list of special
extensions and the list of untranslated file systems.

    > it was recognized with CRLF.  That has been fixed.)

    Can you send me the patches for this so that I can test assuming that
    this case should work?

I don't know what the patch is.  If you want, you can log in here
and try to figure out.

    Currently, the default for file-coding-system-alist is 'undecided.
    Under DOS_NT, this should probably be 'emacs-mule so that CRLF is
    decoded and encoded by default.

I don't follow the reasoning.  Why would changing from undecided
to emacs-mule have any effect on EOL conversion?

Perhaps you're being fooled by the bug that Handa fixed (see above)
which made EOL conversion not work when saving a file, since that
may have been only for `undecided'.

At this point, I don't think we should try to eliminate
file-name-buffer-file-type-alist and untranslated file systems, rather
just make them have their effect via the coding system mechanism which
is the right way for now.  file-name-buffer-file-type-alist and
untranslated file systems are documented.