Subject: Coding system issues (2)

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 08:28:23 +0300 (IDT)

I wonder whether insert-file-contents needs to inherit the coding
system from the buffer, if it is set already (not undecided)?  Right
now, the coding system is computed afresh every time, even if REPLACE
is non-nil, or if we are inserting into a buffer which already has
some text in it.  (I'm not talking merely about DOS_NT EOL conversion 
here.)

One case where this subtlety might bite you is when you byte-compile a
.el file.  The byte compiler erases the buffer and re-reads the file
before it begins the compilation (why, btw?), so even if you had set
the coding system before that, you need to set it again with C-x RET c
before compiling.  If you forget, you might get subtle bugs when
running the .elc file, because the strings get written into it in
converted form.

I had this problem with lisp/term/internal.el which leads Emacs to
believe it's encoded in sjis.  Even if I set the coding to emacs-mule
when I visit the file, Emacs will use sjis when it re-reads the file
before compiling it.  The converted strings were used to set
case-conversion tables, so the effect of this was that Fdowncase
mysteriously stopped working for some characters: a particularly nasty
and hard-to-debug problem.

Do you agree that inserting a file into a buffer that already has a
decided coding system should use the same coding system?

If not, what about the case with byte-compiling?  Should coding system
be bound to that of the buffer during the compilation?  I think users
should not be requested to know whether a certain command calls
insert-file-contents or not.  When I set the coding system for a
buffer, I'd expect that all the operations thereafter will use that
coding system.  Won't you?

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 25 Aug 1997 15:36:07 +0900

> some text in it.  (I'm not talking merely about DOS_NT EOL conversion 

I don't agree.  The coding system of a file being read should be
decided only by the file contents unless the coding system is
specified explicitly.

> and hard-to-debug problem.

This can be avoided by putting `coding' tag at the head of a file as I
wrote before.

By the way, I don't think it is a good idea to have random binary
codes in a source file.  For instance, in the case of internal.el (I
have just notived the existence of this file), we can use backslash
notation (e.g. "\207") instead of putting row binary code in string to
keep the information of cases.

With the current file, if a user of Japanese or Chinese version of
Windows sees the file with their own editor (not emacs), they surely
break the file contents.

And, I think the file name "internal.el" is not appropriate.
Something like "codepage.el" is better.  What do you think?

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 13:21:54 +0300 (IDT)

> This can be avoided by putting `coding' tag at the head of a file as I

No, it's not good enough.  Users can override the coding tag with
C-x c RET  when they loaded the file.  It is IMHO not nice to request 
that they use C-x c RET again before invoking the byte compiler.

> keep the information of cases.

Sure, but this is much harder for the programmer ;-).

> Something like "codepage.el" is better.  What do you think?

The truth is, those case tables should be nuked.  As soon as I learn 
enough about international languages support in Emacs, I will change that 
part to set a specific language environment according to the DOS 
codepage.

Other than that, internal.el does perform terminal-specific stuff, like 
key remapping etc.

---

From: Kenichi Handa <[email-masked]>
Date: Mon, 25 Aug 1997 20:14:49 +0900

> that they use C-x c RET again before invoking the byte compiler.

I don't understand why they dare to load the file by C-x RET c (not
C-x c RET)?  Anyway, if there's a reason to use C-x RET c, it means
that coding tag is not correct and the tag should be modified
correctly.

> Sure, but this is much harder for the programmer ;-).

I don't know why putting those raw 8bit codes is easier for
programmers.  When you add more dos-codepage support (e.g. Slavic,
Turkish), you anyway can't see correct characters.  And, the current
code of internal.el doesn't work for multibyte characters.  To make it
work for multibyte characters, I think the better way is to change
those string to vector of multibyte charactes.

> Other than that, internal.el does perform terminal-specific stuff, like 

I see.

---
Ken'ichi HANDA
[email-masked]

---

From: Eli Zaretskii <[email-masked]>
Date: Mon, 25 Aug 1997 16:56:01 +0300 (IDT)

On Mon, 25 Aug 1997, Kenichi Handa wrote:

> programmers.

Because when its mine codepage, I just type the characters on my keyboard 
;-).

---

From: Richard Stallman <[email-masked]>
Date: Mon, 25 Aug 1997 13:51:00 -0400

No, it's not good enough.  Users can override the coding tag with
    C-x c RET  when they loaded the file.

I do not see a problem with this.  Users can do whatever they want to.

      It is IMHO not nice to request 
    that they use C-x c RET again before invoking the byte compiler.

Could you explain what you are talking about?

---

From: Richard Stallman <[email-masked]>
Date: Mon, 25 Aug 1997 16:55:55 -0400

The coding system of a file being read should be
    decided only by the file contents unless the coding system is
    specified explicitly.

Yes, that is right.

Eli, if you disagree, would you please describe *in full* a case where
you think it is wrong?  It isn't useful to have a discussion if we
are not sure we are talking about the same thing.

---

From: Richard Stallman <[email-masked]>
Date: Mon, 25 Aug 1997 17:55:58 -0400

I wonder whether insert-file-contents needs to inherit the coding
    system from the buffer, if it is set already (not undecided)?

This would be incorrect.  If I insert file foo into a buffer visiting
bar, foo should be decoding in the right coding system of file foo,
which has absolutely nothing to do with the coding system of file bar.

      The byte compiler erases the buffer and re-reads the file
    before it begins the compilation (why, btw?),

Because that buffer probably had some other text in it,
perhaps from another file that you compiled.

Remember that byte-compile-file does NOT visit the input file.
It uses a temporary buffer.  This is a normal technique;
many Emacs commands that look at files use temporary buffers.
Some read many different files.

---

From: Richard Stallman <[email-masked]>
Date: Mon, 25 Aug 1997 17:56:43 -0400

That would be completely wrong.

---

From: Eli Zaretskii <[email-masked]>
Date: Tue, 26 Aug 1997 09:58:49 +0300 (IDT)

> Could you explain what you are talking about?

I did that, in my original message.  This message was one of a series
that was generated by a discussion which followed.

The problem (which took me a couple of days to debug, btw) was that
lisp/term/internal.el, when byte-compiled and loaded, would cause
Fdowncase to behave erratically.  Specifically, the letters A, O, and
U would stay in upper case.  To reproduce:

	     emacs -q
	     M-x load-file lisp/term/internal.elc RET
	     C-x b *scratch* RET
	     (downcase "AOU")^J

(I hope that internal.el can be loaded on Unix with no problems, so
you could try it.)

It turned out that internal.el looks to Emacs as sjis-encoded file.  I
then set the coding system to emacs-mule manually when I visited that
file (`C-x RET c emacs-mule RET C-x C-f lisp/term/internal.el RET'),
but the byte-compiled file was still wrong, because
`emacs-lisp-byte-compile' re-reads the file, and when it does, it
decodes it again as sjis.  So the user must set the coding system
again before compiling the file.  This is IMHO counter-intuitive,
since users might have no idea that the byte compiler reads the file
again.

With the introduction of the `coding' tag in the -*- line, this
problem with internal.el is solved.  But I'm still concerned with the
more general case whereby setting coding system for the .el file is
not enough to byte-compile it as that coding system says.  In the case
where a user needs to override Emacs coding detection, this might lead
to subtle bugs.

---

From: Eli Zaretskii <[email-masked]>
Date: Tue, 26 Aug 1997 10:06:05 +0300 (IDT)

On Mon, 25 Aug 1997, Richard Stallman wrote:

> It uses a temporary buffer.

I was using `emacs-lisp-byte-compile' (also available from the menu bar),
which is supposed to compile the file in the current buffer.  I understand
that it calls `byte-compile-file' internally, but it is still not obvious
that it should re-read the file which in this case is already visited.

---

From: Richard Stallman <[email-masked]>
Date: Tue, 26 Aug 1997 12:56:31 -0400

byte-compile-file should not be influenced by the current buffer.