Subject: Re: Coding system issues (3)

From: Eli Zaretskii <[email-masked]>
Date: Tue, 26 Aug 1997 09:52:54 +0300 (IDT)

On Mon, 25 Aug 1997, Richard Stallman wrote:

>     > Besides, even with Unix EOLs, msdos.c causes Emacs to put "=" on the 
>     > modeline, which means binary file.  This is not quite right.
> 
>     Why?  The file doesn't need EOL conversion.  In addition, the file
>     contains random 8bit codes.  So, it should be read/written without any
>     code conversion.
> 
> Eli, please send him a precise test case; tell Handa *exactly* what to
> type so he can observe this.  Describing actions in abstract ways is a
> VERY bad idea, almost guaranteed to lead to misunderstandings.

I did send a precise test case: `C-x C-f src/msdos.c RET'.  After
that, look at the modeline: it says the coding is no-conversion ("="),
as if this were a binary file.  If the file has DOS EOLs, you will see
^M characters at the end of each line.

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 00:31:18 -0400

I did send a precise test case: `C-x C-f src/msdos.c RET'.  After
    that, look at the modeline: it says the coding is no-conversion ("="),
    as if this were a binary file.

Yes, I see this.  The buffer has buffer-file-coding-system = no-conversion,
but enable-multibyte-characters is t.

This is not right.  If the file happens to have a \201 in it, 
Emacs could get quite confused.

The right thing to do, for a file which has byte codes 200-377 which
Emacs can't understand, is to turn off enable-multibyte-characters.
That way it is safe to read in the file no matter what byte values it
has.

In addition, I agree with Eli that it should do EOL conversion
according to the data in the file.

Handa, can you please implement this?

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 09:55:17 +0300 (IDT)

> That way it is safe to read in the file no matter what byte values it

When Emacs detects binary characters that don't fit into any known
coding system, it assumes that coding-category-binary has been
assigned an appropriate coding system.  Currently, this is
no-conversion by default.  Handa suggested to change that default and
assign to it the (new) coding system to be called raw-text that would
still do EOL conversions.

Will this solve the problem?

If it will, then the only problem that remains is how do we make sure
that a truely binary file that happens to have a few CRLF pairs
doesn't get detected as raw-text-dos.  This seems to call for some
kind of heuristic in detect_eol_type, slightly more complicated than
what's there today (which just compares the number of CRLF pairs
against a compile-time threshold, currently set to 3).

I also think the solution proposed by Handa is better than turning off
enable-multibyte-characters, because the latter would probably mean
that users won't be able to assign something different to
coding-category-binary, in case they need to customize the handling of
binary files.

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 10:28:01 +0300 (IDT)

On Wed, 27 Aug 1997, Geoff Voelker wrote:

> This does not seem correct since a file could be on an "untranslated"
> filesystem and still need a coding system (the untranslated only
> refers to EOL).  Do people agree that "binary" in this context really
> means use LFs for EOL?  If so, then undecided-unix should probably be
> used instead of no-conversion.

I agree, but only for files created by Emacs.  An existing file should set
the EOL type according to its content when it is visited and use that EOL
type when it is saved, even on untranslated systems, because that's what
Emacs would do on Unix (where all filesystems are currently treated as
untranslated). 

I think the automatic decoding of EOLs has taken most of the sting out of 
untranslated filesystems feature, except for the case of files created by 
Emacs.

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 16:33:26 +0900

> Will this solve the problem?

I think so, and I beleive Richard too.  Richard asked me to implement
my suggestion, and also asked me to turn off
enable-multibyte-characters when Emacs detects a file is raw-text.
But, of course, we notice that this can't solve all of the situations.

I've just done it.  But, I found one problem.  When we set
enable-multibyte-characters to nil, mode-line doesn't show any
information about coding system (except for EOL format).  Perhaps, we
have to modify mode-line-format so that it shows `B' if
buffer-file-coding-system is no-conversion and `T' in the other cases.

> against a compile-time threshold, currently set to 3).

I don't think it's worth implementing such kind of heuristics, because
there's anyway a case that we can't detect correctly.  In addition,
such a code makes Emacs slower on reading a normal file.

Or, do you have any idea on detecting EOL format without making Emacs
much slower?

---
Ken'ichi HANDA
[email-masked]

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 16:46:50 +0900

[email-masked] (Geoff Voelker) writes:
> This discussion about no-conversion has made me rethink part of
> find-buffer-file-type-coding-system.  For files that are specified to
> be "binary" in file-name-buffer-file-type-alst or
> untranslated-filesystem-list, the no-conversion coding system is used.

I don't know what do you mean by "\"binary\" in this context".

If you mean coding system `binary' or you use the word in a normal
context, e.g. "This is a text file, that is a binary file", then
`binary' measn "no code conversion (including EOL format) required".

But, now I set coding-category-binary to raw-text.  So, in the context
of coding-category-binary, `binary' refer only to text part, and EOL
is automatically detected.

So, your mail makes me think that we had better:
o treat `coding-category-binary' as truely binary even for EOL format,
o set it to `no-conversion',
o make a new category `coding-category-raw-text',
o and set it to raw-text.

What do you think?

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 10:46:08 +0300 (IDT)

On Wed, 27 Aug 1997, Kenichi Handa wrote:

> Richard asked me to implement

I thought setting coding-category-binary to raw-text is enough.  Your 
two-line test was all I needed to read msdos.c correctly.  So why do you 
also need to turn enable-multibyte-character off?

> I've just done it.

Could you please send me the diffs so I could test this?  Thanks.

> there's anyway a case that we can't detect correctly.

Then how would you suggest to solve the case of a true binary file (say, 
an executable program) that happens to have 3 or more CRLF pairs in it?  
As far as I understand, Emacs will convert the CRLF pairs on input and 
add a CR to any LF on output, which is disastrous in such cases.

> Or, do you have any idea on detecting EOL format without making Emacs

The idea is to not give up checking the file after you've seen the first 3
CRLF pairs, but look into the file some more.  I didn't think about this
enough to have a working solution.  I wanted first to be sure that people
agree that this is the way to go.  But generally, I don't think this would
make the input much slower than it is already, if the heuristic is 
implemented in C (inside decode_coding or thereabouts).

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 16:48:35 +0900

> untranslated filesystems feature, except for the case of files created by 

I'm sorry I don't understand the wording "has taken most of the sting
out of ...".  Could you please tell it in an easier English?  ^.^;;;

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 10:54:00 +0300 (IDT)

> o and set it to raw-text.

This might be an OK solution, but I'm afraid I don't understand how would 
Emacs distinguish between these two coding categories (binary and 
raw-text)?

Let's take msdos.c and emacs.exe as two examples.  The former is a text 
file where EOLs should be decoded, the latter is a binary file where EOLs 
should NOT be converted.

Assuming that emacs.exe has at least 3 CRLF pairs in it, how would Emacs 
know which conversion to apply in each of these two cases?

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 10:55:45 +0300 (IDT)

> out of ...".  Could you please tell it in an easier English?  ^.^;;;

I mean it made the intranslated feature unnecessary in many cases where 
it would be usedin Emacs 19.

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 17:24:33 +0900

> also need to turn enable-multibyte-character off?

By turning enable-multibyte-character off, you can avoid seeing some
garbage characters when some part of the buffer contents matches
Emacs' internal format incidentally, can avoid incorrect cursor moving
in such a case.

> Could you please send me the diffs so I could test this?  Thanks.

I'll attach the current diff at the tail.  Please note that it
also contains patches not related to the current discussion.  I'll
update FSF's code as soon as we reach some agreement.

> add a CR to any LF on output, which is disastrous in such cases.

The 100% safe way is:
o set default value of enable-multibyte-character to nil,
o or register the target file name in file-coding-system-alist,
o or visit the file by C-x RET c no-conversion RET FILENAME.

> implemented in C (inside decode_coding or thereabouts). 

The problem with the current file-reading mechanism is that it doesn't
read the whole text at once, instead it does:
1) reads one bunch
2) detects coding
3) if coding is decided, decodes the bunch just read, goto 5)
4) goto 1)
5) reads the remaining bunches while decoding them by the decided coding.

So, currently, if the detecting routine at step 2 can't decide EOL
format, it insert the text as is in a buffer.

I understand that we had better change this mechanism.  But, it
requires another big change in the current code, and I'm afraid it
will delay shipping Emacs much more.

And, I beleive the patch I attached will save most cases.  Considering
the trade off between making code-detection not that slow and making
code-detection more intelligent, I think the former is important as
fas as we can't have a 100% correct code-detection.

---
Ken'ichi HANDA
[email-masked]

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 17:27:58 +0900

> Emacs distinguish between these two coding categories (binary and 

Only by consistency of EOL format.  If consistent, it's raw-text, if
not, it's no-conversion.

> know which conversion to apply in each of these two cases?

No way.  We can assume any rare cases which make any code-detection
mechanisms fail.

---
Ken'ichi HANDA
[email-masked]

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 17:29:18 +0900

Oops, sorry, I forgot to attach the patch.  Here it is.

---
Ken'ichi HANDA
[email-masked]

------------------------------------------------------------

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 12:12:01 +0300 (IDT)

> Emacs' internal format incidentally, can avoid incorrect cursor moving

I see.  In that case, I agree that it would be better to have the modeline
still show the coding system in the case where Emacs sees unknown binary
characters in the file.  ?t and ?b (for text and binary files,
accordingly) are good enough for me. 

> fas as we can't have a 100% correct code-detection.

I agree.  Any heuristic should be based on user experience, and we cannot 
have that unless Emacs is released ;-).

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 20:08:58 +0900

> accordingly) are good enough for me. 

The remaining matter is how to show the state of
enable-multibyte-character in mode line.  Now, two letters (`-' and
coding system mnemonic) before EOL indicator (`:', `\', or `/') means
enable-multibyte-character is t and one character `-' means
enable-multibyte-character it nil.  If we just show `t' or `b', it's
hard for users to know the status of enable-multibyte-character.  My
idea is to turn the first letter `-' to `='.  And, I prefer `=' to `b'
because mnemonic letter of `no-conversion' is `='.

Another change I want to do is to change mnemonic letter of
`emacs-mule' from `=' to `M', then `=' always tells that the buffer
contents are binary code.

Richard, what do you think?  May I change the current code as above?

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 14:16:30 +0300 (IDT)

On Wed, 27 Aug 1997, Kenichi Handa wrote:

> My

Seems OK to me.

> contents are binary code.

It bothers me for some time that emacs-mule and no-conversion have both 
the same mnemonic, so the change is welcome.  However, I would suggest to 
leave emacs-mule be `=' and invent a new letter (`b'? `B'?) for the 
binary case, since `=' means ``the usual case'', which is emacs-mule.  
The binary case is the exception.

---

From: Kenichi Handa <[email-masked]>
Date: Wed, 27 Aug 1997 21:09:41 +0900

> The binary case is the exception.

Unfortunately `B' is already used by `chinese-big5'.  And I want to
keep the letter `b' for a coding system which we may support in the
feature.  And I think `=' is a good mnemonic for `no-conversion'
because the files' external and internal (to Emacs) codings are
`equal'.  In addition, all the other frequently used coding systems
have non-symbol mnemonics.  So, using the symbol `=' for the exception
(no-conversion) seems reasonable.

---
Ken'ichi HANDA
[email-masked]

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 12:25:17 -0400

Handa suggested to change that default and
    assign to it the (new) coding system to be called raw-text that would
    still do EOL conversions.

NO!  It is impossible to solve the problem unless
enable-multibyte-characters is set to nil.

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 12:38:39 -0400

If a file is binary, no-conversion is right; but in addition,
enable-multibyte-characters should be turned off, so that no sequence
of bytes gets misinterpreted as a multibyte character.

    refers to EOL).

That is true.  Files which are on an untranslated file system, whose
individual names do not imply binary files, are not really "binary".

The best thing to do with them is this: if the file exists, read it
normally; if it does not exist, use undecided-unix as the coding
system for creating it.

Can you implement that?

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 12:45:22 -0400

> doesn't get detected as raw-text-dos.

We cannot do that.  For true binary files, the user has to say
somehow that "this is a binary file".

---

From: Eli Zaretskii <[email-masked]>
Date: Wed, 27 Aug 1997 19:44:35 +0300 (IDT)

> of bytes gets misinterpreted as a multibyte character.

I thought that no-conversion already prevents interpretation of multibyte 
sequences in the file, since it does I/O verbatim.  What am I missing?

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 16:21:09 -0400

I think it is a mistake to try to distinguish this automatically.
It cannot be done right, so let's NOT TRY.

Instead, we should simply tell users that they must specify explicitly
which files are true binary files, one way or another.

Handa, please forget about trying to do this, and install the other
changes now.

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 17:16:46 -0400

I thought that no-conversion already prevents interpretation of multibyte 
    sequences in the file, since it does I/O verbatim.

You're lumping together two entirely different issues.  no-conversion
means that the bytes are not translated when they are read in.  What
they mean in the buffer is another matter!

But perhaps no-conversion SHOULD turn off enable-multibyte-characters.
Handa, what do you think?

---

From: Kenichi Handa <[email-masked]>
Date: Thu, 28 Aug 1997 09:59:59 +0900

> which files are true binary files, one way or another.

I agree.  But, we anyway have to define Emacs' behaviour when it
encounter such a file that has random 8-bit code in text but has
consistent EOL format, or and has inconsistent EOL format.  What Emacs
should do in these cases?  I think using raw-text-XXX in the former
case and using no-conversion is the latter case is reasonable.

> Handa, please forget about trying to do this, and install the other

What do you mean by "this"?
o introducing coding-category-raw-text?
o implementing some more intelligent EOL detecter?
o all the changes about handling raw-text?

---
Ken'ichi HANDA
[email-masked]

---

From: Kenichi Handa <[email-masked]>
Date: Thu, 28 Aug 1997 10:14:04 +0900

> Handa, what do you think?

I agree, and your mail just reminded me that you actually asked that
kind of change long ago.

I'll do this change.

---
Ken'ichi HANDA
[email-masked]

---

From: Richard Stallman <[email-masked]>
Date: Wed, 27 Aug 1997 23:20:49 -0400

My
    idea is to turn the first letter `-' to `='.

This change would be ok with me, but it isn't what the manual says,
and changes in this area of the code have tended to introduce bugs.

So please don't change this.

The manual is already printed and says it is =, so don't change this.

Please don't make any change in this area of Emacs, and work on
the other issues which are more important.

---

From: Richard Stallman <[email-masked]>
Date: Thu, 28 Aug 1997 02:16:53 -0400

Please implement the raw-text coding system as we have already
described it.  Please DO NOT try to distinguish heuristically "real
binary" files from "raw-text files".

    I agree.  But, we anyway have to define Emacs' behaviour when it
    encounter such a file that has random 8-bit code in text but has
    consistent EOL format, or and has inconsistent EOL format.  What Emacs
    should do in these cases?

Distinguish raw-text-unix and raw-text-dos and raw-text-mac
just the same way as you do for most other coding systems.

THis issue is NOT IMPORTANT!  Stop spending time on it!
Implement raw-text in the natural way, as I have explained it,
and STOP PAYING ATTENTION TO IT and MOVE ON TO SOMETHING ELSE.

---

From: Kenichi Handa <[email-masked]>
Date: Thu, 28 Aug 1997 16:18:28 +0900

> and STOP PAYING ATTENTION TO IT and MOVE ON TO SOMETHING ELSE.

Ok, then I've already done what necessary.  I'll update FSF's code
soon.

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Thu, 28 Aug 1997 12:05:25 +0300 (IDT)

> It cannot be done right, so let's NOT TRY.

There's nothing wrong IMHO with a mechanism that does detect binary
files most of the time, even if it doesn't work in all cases.  I have
added such capabilities in various DOS ports of GNU tools (e.g., see
the DJGPP port of Grep) and never heard any complaints.  The diffs
that Handa has sent me seem to implement this consistency test
already, and don't seem too resource-consuming.

I agree that further refinement of the binary file detection could be
delayed until more user experience is available, but I don't think it
can be dismissed altogether.

> which files are true binary files, one way or another.

I believe this should prove as a nuisance.  It was enough of a
nuisance in the DOS_NT world to introduce the
file-name-buffer-file-type-alist so frequently-used binary files will
be recognized automatically.  This solution is IMHO not good enough in
the presence of coding, since e.g. *.c and even *.text files could
include strings encoded in non-English languages.  But I believe that
with small changes in the coding-detection code we could make Emacs
recognize most of the binary files.

I don't think users will like the requirement to have in effect two
different ways of visiting files.  Unix users have never before
distinguished between binary files and the other kind, and I think
they will want to keep it that way.

I'm afraid that if we neglect to take some reasonable care of this
issue, it might become the single most important drive for users to
setq enable-multibyte-characters nil.

---

From: Richard Stallman <[email-masked]>
Date: Thu, 28 Aug 1997 12:20:51 -0400

There's nothing wrong IMHO with a mechanism that does detect binary
    files most of the time, even if it doesn't work in all cases.

If it delays the Emacs 20 release even one day, that is something
very wrong with it.

Please drop the subject so that Handa will go back to fixing what he
needs to fix.

    I believe this should prove as a nuisance.  It was enough of a
    nuisance in the DOS_NT world to introduce the
    file-name-buffer-file-type-alist so frequently-used binary files will
    be recognized automatically.

Too bad.  Nothing can be done.