Subject: Coding system issues (3)

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 08:36:03 +0300 (IDT)

There's something in autodetection of a file's coding system which I
find deeply disturbing: it gets in my way when I edit e.g. C sources
with strings that include ASCII characters with the high bit set.  For
example, try to load src/msdos.c or lisp/term/internal.el.  You will
get no-conversion in the first case (which means CRLFs won't be
converted if that file is in DOS format) and in the second you get sjis.
Which is dead wrong in both cases: these are just tables of ASCII
characters with codes beyond 127.

Now, I understand that Emacs cannot possibly know what did I mean when
I put such strings into the file.  These strings might as well be text
in some language other than English, right?  But what annoys me is that
I need to set the coding system explicitly each time I visit these files
to see them as God intended.

Am I missing some function or variable?

If not, then do I have any other way except local file variables to
tell Emacs these are ASCII files?  With major modes, we can specify
the mode on the first nonblank line if we don't like Emacs' choice,
but there seems to be no such feature for coding systems.

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 25 Aug 1997 15:01:28 +0900

> but there seems to be no such feature for coding systems.

In the latest pretest, 

Richard Stallman <[email-masked]> writes:
> I have made a new pretest, which is tarring up now.
> It will soon be in gnu/emacs/{emacs.xtar.gz,leim.xtar.gz}
> on alpha.gnu.ai.mit.edu.

> It has an important new feature:
> You can specify the coding system for a file using the -*-
> construct.  Include `coding: CODINGSYSTEM;' inside the -*-...-*-
> to specify use of coding system CODINGSYSTEM.

So, we can use this feature for src/msdos.c and lisp/term/internal.el.
But, since I followed the way of handling `mode' tag, the `coding' tag
should also be at the first line of a file, which requires making the
first line of internal.el very long.

Richard, shouldn't we loosen this restriction at least for `coding'
tag?  How about consulting at least the first three lines?

By the way, src/msdos.c has Unix-like EOL now, doesn't it?

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 13:10:16 +0300 (IDT)

> So, we can use this feature for src/msdos.c and lisp/term/internal.el.

I've seen Richard's announcement after I wrote the message.  I will add 
"codong: " settings to those two files.

> By the way, src/msdos.c has Unix-like EOL now, doesn't it?

Yes.  So if you stay with the same version of Emacs, you won't see any 
problem.  But I also loaded msdos.c edited by a previous version of 
Emacs, which always added CRs, and then I saw all those ^M characters.

Besides, even with Unix EOLs, msdos.c causes Emacs to put "=" on the 
modeline, which means binary file.  This is not quite right.

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 25 Aug 1997 19:54:42 +0900

> "codong: " settings to those two files.

I recommend to use backslash notation in those files.  Then, there's
no nead of `coding:' tags.

> Emacs, which always added CRs, and then I saw all those ^M characters.

Yah!  Hmmm.  If a file contains random 8-bit code which doesn't fit
the coding system emacs-mule, it is detected as binary file.  This is
a difficult problem.  How can we distinguish such a file from a truely
binnary file which doesn't require any EOL conversion?

> modeline, which means binary file.  This is not quite right.

Why?  The file doesn't need EOL conversion.  In addition, the file
contains random 8bit codes.  So, it should be read/written without any
code conversion.

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 14:07:13 +0300 (IDT)

> binnary file which doesn't require any EOL conversion?

This has been my concern since I first looked at src/coding.c.  The
`coding' tag will have to be the stopgap for now.  Another solution is to
use `find-file-text'.  Perhaps some heuristic could be added in future
based on the relative frequency of CRLF pairs and the binary characters. 

> contains random 8bit codes.  So, it should be read/written without any

The modeline is not for Emacs, it's for the user.  The user should NOT 
see "=" when the file is a text file.  Otherwise, we will need to 
resurrect the T: and B: that we have just nuked in 20.0.93 (because we 
agreed that the coding system tells enough).

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 25 Aug 1997 21:39:16 +0900

> based on the relative frequency of CRLF pairs and the binary characters. 

Hmm, perhaps, we must now give up detecting a coding system of a file
in an incremental manner as being done now, but have to read the whole
file with no conversion, detect a coding system by running
sophisticated Emacs Lisp code on the whole buffer, then decode the
whole buffer at once.  This requires a lot more memory and
time-consuming for reading a huge file, but the advantage of more
appropriate code-detection may be larger than this disadvantage.

> agreed that the coding system tells enough).

The modeline doesn't say anything about the file is text or binary.
It just says how the file was encoded.  They are different things.
Although we have a coding system `binary' (alias of no-conversion),
the term `binary' doesn't means that of DOS file type.  But, hmmm,
perhaps DOS users are too familiar with the concept of file type (text
or binary).  For Unix users, usually there's no difference.

Anyway, these discussions suggests that we have to detect EOL type
even after we detect that a text contains random 8-bit code.  How
about adding a new coding system raw-text, raw-text-dos,
raw-text-unix, raw-text-mac, and set coding-category-binary to
raw-text if we are not in such language environment as Vietnames which
are using such a random 8-bit file for their own language files.

Please try the followings:

(make-coding-system 'raw-text 0 ?t "Raw text")
(setq coding-category-binary 'raw-text)

and find-file msdos.c of LF format and of CRLF format.

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 17:09:56 +0300 (IDT)

> It just says how the file was encoded.  They are different things.

Originally, yes.  Previous versions of Emacs on MS-DOS would display T: 
or B:, accordingly, for text and binary files.  In Emacs 20, we all 
agreed that the coding system/EOL info on the modeline makes those T:/B: 
redundant.  So now they aren't displayed by default.

But this means that the coding system and EOL part of the modeline now
has an additional meaning on DOS and NT.

> even after we detect that a text contains random 8-bit code.

Yep, seems this could be a solution that won't slow down the file loading 
too much.

> and find-file msdos.c of LF format and of CRLF format.

This works (displays "t:" and "t\" respectively on the modeline).

I will have to dig deeper to understand what this does exactly.

Richard, Geoff and Andrew, do you agree that creating such a coding system 
is the way to go?

---

From: Richard Stallman <[email-masked]>
Date: Mon, 25 Aug 1997 23:59:33 -0400

> Emacs, which always added CRs, and then I saw all those ^M characters.

That was a bug in the old version of Emacs.  Let's not worry about old
bugs that have been fixed.

    Yah!  Hmmm.  If a file contains random 8-bit code which doesn't fit
    the coding system emacs-mule, it is detected as binary file.

Could you be more precise?  What does "detected as binary file" really
mean?  What does Emacs DO in this case?

---

From: Kenichi Handa <[email-masked]>
Date: Tue, 26 Aug 1997 14:08:25 +0900

> bugs that have been fixed.

No.  This bug (feature?) still remains.  If a file is formated by
DOS-like EOL, and there exist ramdom 8-bit codes somewhere in the
file, the current version detect it as coding-category-binary, and
does no code conversion because coding-category-binary is set to
`no-conversion' by default.

> mean?  What does Emacs DO in this case?

It does as I wrote above because decode_coding (in coding.c) has the
following code.
----------------------------------------------------------------------
  if (coding->type == coding_type_undecided)
    detect_coding (coding, source, src_bytes);

  if (coding->eol_type == CODING_EOL_UNDECIDED)
    detect_eol (coding, source, src_bytes);
----------------------------------------------------------------------
So, Emacs at first try to detect text coding.  At this time, if the
file contains random 8-bit code, Emacs thinks that the category of
coding is coding-category-binary and setup the coding system
no-conversion in the structure `coding' (coding->eol_type is set to
CODING_EOL_LF).  So, it skips detect_eol.

Even now, if we set coding-category-binary to `emacs-mule', this
problem is avoided, but it is like setting coding-category-sjis to
iso-latin-1 (Richard, do you remember the previous discussion about
handling Microsoft extra latin code?), and not a right thing.
In addtition, mnemonic of `emacs-mule' is `=' (same as no conversion),
which won't help DOS users.

So, I proposed a new coding system `raw-text' (thought I'm not sure
this is a good name or not) which requires only EOL conversion and set
coding-category-binary to raw-text by default.

Then, the call of detect_coding setup raw-text in coding
(coding->eol_type is set to CODING_EOL_UNDECIDED), and Emacs calls
detect_eol which may set coding->eol_type correctly.

The demerit of this method is that a truely binary file is detected as
`raw-text-XXX'.  But, this can be avoided except for a very rare case
by changing the code of detect_eol so that it setup no-conversion to
the struct `coding' if EOL format is not consistent.

---
Ken'ichi HANDA
[email-masked]

---

From: "Richard M. Stallman" <[email-masked]>
Date: Sat, 30 Aug 1997 12:55:04 -0700 (PDT)

whole buffer at once.

What worries me in this, is that this method could be used for
insert-file-contents, but cannot be used for a synchronous subprocess.
I am not sure that it is a good idea to use different methods for
subprocesses and files.

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 1 Sep 1997 17:12:04 +0900

> I am not sure that it is a good idea to use different methods for

?? A synchronous subprocess has no problem, we can read the whole
output into a buffer, and then process it.

The problem is with an asynchronous subprocess because we must detect
a coding system on the fly.  But, I think it is enough to give just a
bunch of data Emacs receives from the subprocess to the sophisticated
Emacs Lisp code which I mentioned above.

And, even in file-reading (insert-file-contents), if BEG and END are
specified, that Emacs Lisp code will detect a coding system only from
the part of a file.

---
Ken'ichi HANDA
[email-masked]

---

From: Richard Stallman <[email-masked]>
Date: Mon, 1 Sep 1997 13:07:20 -0400

Right, I meant to say asynchronous.

I am not sure what "enough" means.  It would do something, it would
choose a coding system, but it would do so in an inconsistent way.

In effect, what I am saying is this: if this method is acceptable for
an asynchronous subprocess, doesn't that mean it is also acceptable
for a file?

---

From: Kenichi Handa <[email-masked]>
Date: Tue, 2 Sep 1997 09:50:57 +0900

> an asynchronous subprocess, doesn't that mean it is also acceptable

Fo an asynchronous subprocess of which communication is hidden from
users such as ispell and nntp, we anyway have to specify a coding
system explicitly.  It is dangerous to beleive automatic code
detection.  But, for shell (for instance), users can see the output
from the subprocess, and they can easily find something is wrong when
Emacs fails to find a correct coding automatically.  And they can
change a coding system by C-x RET p interactively.  The sophisticated
Emacs Lisp code mentioned above won't work that acculate when it is
given a few data (perhaps only one line data), but I think it is
acceptable for subprocesses.

---
Ken'ichi HANDA
[email-masked]