Emacs24 coding system problems

Emacs24 coding system problems

[Uday Reddy]

I have a problem with Emacs 24 that is a bit mystifying.  I manage to put it
in a state, by using VM, where

  - enable-multibyte-characters has the global value of t, but
  - every time I visit a file, it gets turned off locally.

This happens with both Emacs 24.1 and 24.2.  It didn't happen with Emacs 23.

The DOS line endings in files don't get recognized either.

There must be something that VM does which fires this bug, but can't figure
out what.  VM often creates unibyte buffers to load mail folders.  It uses a
function like this to do so:

(defun vm-generate-new-unibyte-buffer (name)
  (if vm-xemacs-p
      (generate-new-buffer name)
    (let* ((buffer (generate-new-buffer name)))
      (when (fboundp 'set-buffer-multibyte)
	(with-current-buffer buffer
	  (set-buffer-multibyte nil)))
      buffer)))

I am not sure if this is violating any protocols.

Cheers,
Uday

Re: Emacs24 coding system problems

[Andreas Schwab]

Uday Reddy <usr.vm.rocks@gmail.com> writes:

> I have a problem with Emacs 24 that is a bit mystifying.  I manage to put it
> in a state, by using VM, where
>
>   - enable-multibyte-characters has the global value of t, but
>   - every time I visit a file, it gets turned off locally.

Maybe you have a non-nil global value of coding-system-for-read, or
file-coding-system-alist is clobbered somehow.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs24 coding system problems

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Fri, 02 Nov 2012 09:39:02 +0100
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Uday Reddy <usr.vm.rocks@gmail.com> writes:
> 
> > I have a problem with Emacs 24 that is a bit mystifying.  I manage to put it
> > in a state, by using VM, where
> >
> >   - enable-multibyte-characters has the global value of t, but
> >   - every time I visit a file, it gets turned off locally.
> 
> Maybe you have a non-nil global value of coding-system-for-read, or
> file-coding-system-alist is clobbered somehow.

Or maybe the file in question has undecodeable byte sequences.  Do you
see "t" (which stands for 'raw-text') in the mode line, per chance?

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Fri, 02 Nov 2012 10:51:23 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: usr.vm.rocks@gmail.com, emacs-devel@gnu.org
> 
> > From: Andreas Schwab <schwab@linux-m68k.org>
> > Date: Fri, 02 Nov 2012 09:39:02 +0100
> > Cc: emacs-devel <emacs-devel@gnu.org>
> > 
> > Uday Reddy <usr.vm.rocks@gmail.com> writes:
> > 
> > > I have a problem with Emacs 24 that is a bit mystifying.  I manage to put it
> > > in a state, by using VM, where
> > >
> > >   - enable-multibyte-characters has the global value of t, but
> > >   - every time I visit a file, it gets turned off locally.
> > 
> > Maybe you have a non-nil global value of coding-system-for-read, or
> > file-coding-system-alist is clobbered somehow.
> 
> Or maybe the file in question has undecodeable byte sequences.  Do you
> see "t" (which stands for 'raw-text') in the mode line, per chance?

But if this happens with _every_ file you visit, then my suggestion is
not relevant, obviously.

Re: Emacs24 coding system problems

[Uday Reddy]

Thanks for all the responses to this problem.

Andreas Schwab writes:

> Maybe you have a non-nil global value of coding-system-for-read, or
> file-coding-system-alist is clobbered somehow.

coding-system-for-read is nil.  file-coding-system-alist looks the same as
the normal.

Eli Zaretskii writes:

> Or maybe the file in question has undecodeable byte sequences.  Do you
> see "t" (which stands for 'raw-text') in the mode line, per chance?

The files are normal unicode text.  But, they appear as sequences of octets,
as if they were visited by find-file-literally.

There is `t' in the modeline.  `buffer-file-coding-system' is getting set to
`no-conversion', whereas normally it is `utf-8-unix'.

These files work fine in Emacs23.  The problems only arise in Emacs24.

Stefan Monnier writes:

> Could you have set coding-system-for-read maybe?

coding-system-for-read is nil.

----

I have made some progress on narrowing down the problem.  It happens only if
VM is used along with emacs-w3m for displaying the HTML content.  Without
emacs-w3m, everything works fine.

Emacs-w3m evidently does a lot of manipulations with coding-systems.  But,
they are only supposed to affect the interaction with the w3m sub-process.
So, it is not clear why they should affect the reading of files.  Indeed, in
Emacs23 they didn't.

If you would like to take a look at what w3m does, here is a linke to the
main file:

  http://cvs.namazu.org/Development/emacs-w3m/w3m.el?view=log

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Sun, 18 Nov 2012 19:06:28 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Uday Reddy <usr.vm.rocks@gmail.com>
> 
> Thanks for all the responses to this problem.

Sorry, I only remember very vaguely what this was about.

> > Or maybe the file in question has undecodeable byte sequences.  Do you
> > see "t" (which stands for 'raw-text') in the mode line, per chance?
> 
> The files are normal unicode text.  But, they appear as sequences of octets,
> as if they were visited by find-file-literally.
> 
> There is `t' in the modeline.  `buffer-file-coding-system' is getting set to
> `no-conversion', whereas normally it is `utf-8-unix'.

't' is not no-conversion, 't' is raw-text.  I wonder what is going on
there.  Is there a file and a recipe from "emacs -Q" I could try to
reproduce this?

> I have made some progress on narrowing down the problem.  It happens only if
> VM is used along with emacs-w3m for displaying the HTML content.  Without
> emacs-w3m, everything works fine.

If this is so complicated, then the only way to track this down is to
set breakpoints on the C level where buffer is decoded
(decode-coding-region, I guess), and see what happens there and why.

> If you would like to take a look at what w3m does, here is a linke to the
> main file:
> 
>   http://cvs.namazu.org/Development/emacs-w3m/w3m.el?view=log

I doubt that this would be efficient, this stuff you are using
probably needs a lot of setup.  If you cannot find a simple recipe,
then GDB is your friend.

Re: Emacs24 coding system problems

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > 't' is not no-conversion, 't' is raw-text.

There's no real difference, except for EOL processing.  I doubt that's
relevant here.

Re: Emacs24 coding system problems

[Uday Reddy]

Eli Zaretskii writes:

> Sorry, I only remember very vaguely what this was about.

The problem is that once Emacs24 is used with VM along with emacs-w3m, then
from then on Emacs24 reads every file "literally", i.e., doesn't recognize
the coding-systems and dos line-endings.

> > > Or maybe the file in question has undecodeable byte sequences.  Do you
> > > see "t" (which stands for 'raw-text') in the mode line, per chance?
> >
> > There is `t' in the modeline.  `buffer-file-coding-system' is getting set to
> > `no-conversion', whereas normally it is `utf-8-unix'.

Sorry, that was a serious typo.  I mean tto say there is *no* `t' in the
modeline.

> If this is so complicated, then the only way to track this down is to
> set breakpoints on the C level where buffer is decoded
> (decode-coding-region, I guess), and see what happens there and why.

Yeah, I was afraid of that.  

Cheers,
Uday

Re: Emacs24 coding system problems

[Stefan Monnier]

> Sorry, that was a serious typo.  I mean tto say there is *no* `t' in the
> modeline.

That confirms the buffer is in unibyte mode, even though you say the
global value of enable-multibyte-characters is t.
Maybe you can try to debug-on-entry of set-buffer-multibyte (tho many
calls are done from C, so won't be caught).


        Stefan

Re: Emacs24 coding system problems

[Uday Reddy]

Stefan Monnier writes:

> Maybe you can try to debug-on-entry of set-buffer-multibyte (tho many
> calls are done from C, so won't be caught).

Thanks, Stefan.

There was only call made to set-buffer-multibyte from find-file-noselect-1.
The argument was t.

Tracing through find-file-noselect-1, I found that `rawfile' was set to
nil.  Then there is call to `insert-file-contents'.  When it returns,
enable-multibyte-characters has been set to nil.

Trying out various revisions of the Emacs trunk, I have narrowed down the
problem to the revisions between Sean Seager's builds 2/14/11 and 2/21/11,
i.e., roughly between trunk revisions 103261 and 103379.

Four commits in this range had changes to multibyte issues:

  103273
  103292
  103304
  103338

The first two seem to be refactorings.  The third one is only dealing with
allout.  The commit 103338 might be the one that is interfering with
emacs-w3m.

Cheers,
Uday

Re: Emacs24 coding system problems

[Uday Reddy]

Uday Reddy writes:

> Four commits in this range had changes to multibyte issues:
> 
>   103273
>   103292
>   103304
>   103338
> 
> The first two seem to be refactorings.  The third one is only dealing with
> allout.  The commit 103338 might be the one that is interfering with
> emacs-w3m.

False alarm.  I managed to get the trunk to build on Win32.

I notice that rev. 103303 already had the problem and 103291 didn't.  So,
the commit 103292 is most likely the source of the problem.

I can't get the revision 103292 to compile.  It gave an error

xdisp.c: In function 'decode_mode_spec':
xdisp.c:19589:14: error: 'buffer_file_type' undeclared (first use in this functi
on)
xdisp.c:19589:14: note: each undeclared identifier is reported only once for eac
h function it appears in
mingw32-make[2]: *** [oo-spd/i386/xdisp.o] Error 1

I am trying the neighbouring revisions.

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Sun, 25 Nov 2012 10:44:26 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>, emacs-devel@gnu.org
> 
> False alarm.  I managed to get the trunk to build on Win32.

I'm confused: is this problem specific to MS-Windows?  I don't think
you ever mentioned Windows in this thread before; did I miss
something?

> I can't get the revision 103292 to compile.  It gave an error
> 
> xdisp.c: In function 'decode_mode_spec':
> xdisp.c:19589:14: error: 'buffer_file_type' undeclared (first use in this functi
> on)
> xdisp.c:19589:14: note: each undeclared identifier is reported only once for eac
> h function it appears in
> mingw32-make[2]: *** [oo-spd/i386/xdisp.o] Error 1
> 
> I am trying the neighbouring revisions.

Uday, this isn't the most efficient way of finding the bug.  It will
waste a lot of your time and energy on overcoming unrelated problems,
especially since 103292 is such a mega-changeset.

The most efficient way is to debug the _current_ codebase.  Not to go
back to prior versions, but go forward.  We are not interested in
finding bugs in past versions, we are interested in finding current
(and future ;-) bugs.

So let's revisit your last finding.  You said:

> Tracing through find-file-noselect-1, I found that `rawfile' was set to
> nil.  Then there is call to `insert-file-contents'.  When it returns,
> enable-multibyte-characters has been set to nil.

The logical next step is to find out what happened during the call to
insert-file-contents.  The most probable suspect is the call to the
function after-insert-file-set-coding.  insert-file-contents calls it
after reading the file into a buffer, in order to find the appropriate
coding-system to set the buffer's buffer-file-coding-system.  Can you
see what happens inside after-insert-file-set-coding?

Re: Emacs24 coding system problems

[Uday Reddy]

Eli Zaretskii writes:

> I'm confused: is this problem specific to MS-Windows?  I don't think
> you ever mentioned Windows in this thread before; did I miss
> something?

I didn't know it was specific to Windows either.  But, given that Kazuhiro's
workaround fixed the problem, it must be.

It appears that Emacs24 is not treating `buffer-file-type' as buffer-local
variable, at least in some places inside its guts.  The Bug Report #12989
demonstrated that.

> So let's revisit your last finding.  You said:
> 
> > Tracing through find-file-noselect-1, I found that `rawfile' was set to
> > nil.  Then there is call to `insert-file-contents'.  When it returns,
> > enable-multibyte-characters has been set to nil.
> 
> The logical next step is to find out what happened during the call to
> insert-file-contents.  The most probable suspect is the call to the
> function after-insert-file-set-coding.  insert-file-contents calls it
> after reading the file into a buffer, in order to find the appropriate
> coding-system to set the buffer's buffer-file-coding-system.  Can you
> see what happens inside after-insert-file-set-coding?

insert-file-contents calls find-buffer-file-type-coding-system, which
returns (no-conversion . no-conversion).  enable-multibyte-characters has
been set to nil already.  after-insert-file-set-coding doesn't make any
difference to this.

If I set buffer-file-type to nil and try again:

- find-buffer-file-type-coding-system returns (undecided . undecided).
enable-multibyte-characters is still nil.

- By the time after-insert-file-set-coding gets called,
enable-multibyte-characters has already been set to t.  All is well after
that.

Cheers,
Uday

 

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Sun, 25 Nov 2012 20:36:29 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Uday Reddy <usr.vm.rocks@gmail.com>,
>     monnier@IRO.UMontreal.CA,
>     emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
> 
> > I'm confused: is this problem specific to MS-Windows?  I don't think
> > you ever mentioned Windows in this thread before; did I miss
> > something?
> 
> I didn't know it was specific to Windows either.

Telling the OS on which problems happen is useful regardless.

> It appears that Emacs24 is not treating `buffer-file-type' as buffer-local
> variable, at least in some places inside its guts.  The Bug Report #12989
> demonstrated that.

Why do you (or is it VM?) use that variable at all?

> If I set buffer-file-type to nil and try again:
> 
> - find-buffer-file-type-coding-system returns (undecided . undecided).
> enable-multibyte-characters is still nil.
> 
> - By the time after-insert-file-set-coding gets called,
> enable-multibyte-characters has already been set to t.  All is well after
> that.

The default value of buffer-file-type is nil.  How did it get set to
something else?

Re: Emacs24 coding system problems

[Kazuhiro Ito]

> Trying out various revisions of the Emacs trunk, I have narrowed down the
> problem to the revisions between Sean Seager's builds 2/14/11 and 2/21/11,
> i.e., roughly between trunk revisions 103261 and 103379.

I guess your problem is related with Bug#12989.  Please check whether
buffer-file-type is non-nil when the problem occurs.  If that is the
cause, write the below code in your ~/.emacs for workaround.

(defvar-local buffer-file-type nil)

-- 
Kazuhiro Ito

Re: Emacs24 coding system problems

[Uday Reddy]

Kazuhiro Ito writes:

> I guess your problem is related with Bug#12989.  Please check whether
> buffer-file-type is non-nil when the problem occurs.  If that is the
> cause, write the below code in your ~/.emacs for workaround.
> 
> (defvar-local buffer-file-type nil)

Thanks very much, Kazuhiro.  Your workaround fixed the problem.  (I had to
add a separate `make-variable-buffer-local' in Emacs 24.2.)

I am surprised that Emacs24 shipped with such a glaring bug.

Perhaps you can mention the work-around as a follow-up to your bug report,
so that people can find it?

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Sun, 25 Nov 2012 19:58:25 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,
> 	Uday Reddy <usr.vm.rocks@gmail.com>, emacs-devel@gnu.org
> 
> I am surprised that Emacs24 shipped with such a glaring bug.

No one uses that variable anymore.  Since it was removed (Feb 2011!)
no one complained until now.  So I think "glaring bug" is a glaring
exaggeration.

Re: Emacs24 coding system problems

[Uday Reddy]

Eli Zaretskii writes:

> > Date: Sun, 25 Nov 2012 19:58:25 +0000
> > From: Uday Reddy <usr.vm.rocks@gmail.com>
> > Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,
> > 	Uday Reddy <usr.vm.rocks@gmail.com>, emacs-devel@gnu.org
> > 
> > I am surprised that Emacs24 shipped with such a glaring bug.
> 
> No one uses that variable anymore.  Since it was removed (Feb 2011!)
> no one complained until now.  So I think "glaring bug" is a glaring
> exaggeration.

Hmm.  Eli, it looks like you haven't read Kazuhiro's bug report closely.
find-file-noselect is setting the variable.  We are not.

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Sun, 25 Nov 2012 23:37:14 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Uday Reddy <usr.vm.rocks@gmail.com>,
>     kzhr@d1.dion.ne.jp,
>     monnier@IRO.UMontreal.CA,
>     emacs-devel@gnu.org
> 
> Hmm.  Eli, it looks like you haven't read Kazuhiro's bug report closely.
> find-file-noselect is setting the variable.

It does?  Can you show me where?  I must be blind, because I don't see
this variable mentioned anywhere in files.el.

Re: Emacs24 coding system problems

[Kazuhiro Ito]

At Mon, 26 Nov 2012 05:41:57 +0200,
Eli Zaretskii wrote:
> 
> > Date: Sun, 25 Nov 2012 23:37:14 +0000
> > From: Uday Reddy <usr.vm.rocks@gmail.com>
> > Cc: Uday Reddy <usr.vm.rocks@gmail.com>,
> >     kzhr@d1.dion.ne.jp,
> >     monnier@IRO.UMontreal.CA,
> >     emacs-devel@gnu.org
> > 
> > Hmm.  Eli, it looks like you haven't read Kazuhiro's bug report closely.
> > find-file-noselect is setting the variable.
> 
> It does?  Can you show me where?

find-file-not-found-set-buffer-file-coding-system in lisp/dos-w32.el.
lisp/dos-w32.el adds find-file-not-found-set-buffer-file-coding-system
to find-file-not-found-functions.

Please try the below code on trunk or emacs-24 on Windows.

(let ((debug-on-error t))
  (defadvice find-file-not-found-set-buffer-file-coding-system
    (after debug activate)
    (when buffer-file-type
      (error "buffer-file-type is non-nil")))
  (list
   (setq buffer-file-type nil)
   (progn
     (let ((coding-system-for-read 'no-conversion))
       ;; Specify non-existing file.
       (kill-buffer (find-file-noselect "c:/zzzzzzz")))
     buffer-file-type)))


Debugger entered--Lisp error: (error "buffer-file-type is non-nil")
  signal(error ("buffer-file-type is non-nil"))
  error("buffer-file-type is non-nil")
  find-file-not-found-set-buffer-file-coding-system()
  run-hook-with-args-until-success(find-file-not-found-set-buffer-file-coding-system)
  (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t))
  (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t))))
  (if rawfile (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents-literally filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (setq error t))) (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t)))))
  (save-current-buffer (set-buffer buf) (kill-local-variable (quote find-file-literally)) (kill-local-variable (quote buffer-file-coding-system)) (kill-local-variable (quote cursor-type)) (let ((inhibit-read-only t)) (erase-buffer)) (and (default-value (quote enable-multibyte-characters)) (not rawfile) (set-buffer-multibyte t)) (if rawfile (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents-literally filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (setq error t))) (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer 
 buf) (signal (quote file-error) (list "File is not readable" filename))) (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t))))) (if (equal filename b
 uffer-file-name) (setq buffer-file-truename truename) (setq buffer-file-truename (abbreviate-file-name (file-truename buffer-file-name)))) (setq buffer-file-number number) (if find-file-visit-truename (setq buffer-file-name (expand-file-name buffer-file-truename))) (setq default-directory (file-name-directory buffer-file-name)) (and backup-enable-predicate (not (funcall backup-enable-predicate buffer-file-name)) (progn (make-local-variable (quote backup-inhibited)) (setq backup-inhibited t))) (if rawfile (progn (set-buffer-multibyte nil) (setq buffer-file-coding-system (quote no-conversion)) (set-buffer-major-mode buf) (make-local-variable (quote find-file-literally)) (setq find-file-literally t)) (after-find-file error (not nowarn))) (current-buffer))
  (with-current-buffer buf (kill-local-variable (quote find-file-literally)) (kill-local-variable (quote buffer-file-coding-system)) (kill-local-variable (quote cursor-type)) (let ((inhibit-read-only t)) (erase-buffer)) (and (default-value (quote enable-multibyte-characters)) (not rawfile) (set-buffer-multibyte t)) (if rawfile (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents-literally filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (setq error t))) (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents filename t)) (file-error (when (and (file-exists-p filename) (not (file-readable-p filename))) (kill-buffer buf) (signal 
 (quote file-error) (list "File is not readable" filename))) (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t))))) (if (equal filename buffer-file-na
 me) (setq buffer-file-truename truename) (setq buffer-file-truename (abbreviate-file-name (file-truename buffer-file-name)))) (setq buffer-file-number number) (if find-file-visit-truename (setq buffer-file-name (expand-file-name buffer-file-truename))) (setq default-directory (file-name-directory buffer-file-name)) (and backup-enable-predicate (not (funcall backup-enable-predicate buffer-file-name)) (progn (make-local-variable (quote backup-inhibited)) (setq backup-inhibited t))) (if rawfile (progn (set-buffer-multibyte nil) (setq buffer-file-coding-system (quote no-conversion)) (set-buffer-major-mode buf) (make-local-variable (quote find-file-literally)) (setq find-file-literally t)) (after-find-file error (not nowarn))) (current-buffer))
  (let (error) (with-current-buffer buf (kill-local-variable (quote find-file-literally)) (kill-local-variable (quote buffer-file-coding-system)) (kill-local-variable (quote cursor-type)) (let ((inhibit-read-only t)) (erase-buffer)) (and (default-value (quote enable-multibyte-characters)) (not rawfile) (set-buffer-multibyte t)) (if rawfile (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents-literally filename t)) (file-error (when (and (file-exists-p filename) (not ...)) (kill-buffer buf) (signal (quote file-error) (list "File is not readable" filename))) (setq error t))) (condition-case nil (let ((inhibit-read-only t)) (insert-file-contents filename t)) (file-error (when (and (file-exists-p filename) (not ...)) (kill-buffer buf) (signal (quote file-error) (list "File is
  not readable" filename))) (or (run-hook-with-args-until-success (quote find-file-not-found-functions)) (setq error t))))) (if (equal filename buffer-file-name) (setq buffer-file-truename tr
 uename) (setq buffer-file-truename (abbreviate-file-name (file-truename buffer-file-name)))) (setq buffer-file-number number) (if find-file-visit-truename (setq buffer-file-name (expand-file-name buffer-file-truename))) (setq default-directory (file-name-directory buffer-file-name)) (and backup-enable-predicate (not (funcall backup-enable-predicate buffer-file-name)) (progn (make-local-variable (quote backup-inhibited)) (setq backup-inhibited t))) (if rawfile (progn (set-buffer-multibyte nil) (setq buffer-file-coding-system (quote no-conversion)) (set-buffer-major-mode buf) (make-local-variable (quote find-file-literally)) (setq find-file-literally t)) (after-find-file error (not nowarn))) (current-buffer)))
  find-file-noselect-1(#<buffer zzzzzzz> "c:/zzzzzzz" nil nil "c:/zzzzzzz" nil)
  find-file-noselect("c:/zzzzzzz")
  (kill-buffer (find-file-noselect "c:/zzzzzzz"))
  (let ((coding-system-for-read (quote no-conversion))) (kill-buffer (find-file-noselect "c:/zzzzzzz")))
  (progn (let ((coding-system-for-read (quote no-conversion))) (kill-buffer (find-file-noselect "c:/zzzzzzz"))) buffer-file-type)
  (list (setq buffer-file-type nil) (progn (let ((coding-system-for-read (quote no-conversion))) (kill-buffer (find-file-noselect "c:/zzzzzzz"))) buffer-file-type))
  (let ((debug-on-error t)) (defadvice find-file-not-found-set-buffer-file-coding-system (after debug activate) (when buffer-file-type (error "buffer-file-type is non-nil"))) (list (setq buffer-file-type nil) (progn (let ((coding-system-for-read (quote no-conversion))) (kill-buffer (find-file-noselect "c:/zzzzzzz"))) buffer-file-type)))
  eval((let ((debug-on-error t)) (defadvice find-file-not-found-set-buffer-file-coding-system (after debug activate) (when buffer-file-type (error "buffer-file-type is non-nil"))) (list (setq buffer-file-type nil) (progn (let ((coding-system-for-read (quote no-conversion))) (kill-buffer (find-file-noselect "c:/zzzzzzz"))) buffer-file-type))) nil)
  eval-last-sexp-1(t)
  eval-last-sexp(t)
  eval-print-last-sexp()
  call-interactively(eval-print-last-sexp nil nil)

-- 
Kazuhiro Ito

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Mon, 26 Nov 2012 17:30:25 +0900
> From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
> Cc: Uday Reddy <usr.vm.rocks@gmail.com>, monnier@IRO.UMontreal.CA,
>  emacs-devel@gnu.org
> 
> At Mon, 26 Nov 2012 05:41:57 +0200,
> Eli Zaretskii wrote:
> > 
> > > Date: Sun, 25 Nov 2012 23:37:14 +0000
> > > From: Uday Reddy <usr.vm.rocks@gmail.com>
> > > Cc: Uday Reddy <usr.vm.rocks@gmail.com>,
> > >     kzhr@d1.dion.ne.jp,
> > >     monnier@IRO.UMontreal.CA,
> > >     emacs-devel@gnu.org
> > > 
> > > Hmm.  Eli, it looks like you haven't read Kazuhiro's bug report closely.
> > > find-file-noselect is setting the variable.
> > 
> > It does?  Can you show me where?
> 
> find-file-not-found-set-buffer-file-coding-system in lisp/dos-w32.el.
> lisp/dos-w32.el adds find-file-not-found-set-buffer-file-coding-system
> to find-file-not-found-functions.

Thanks.  I committed to the emacs-24 branch changes that should fix
this.  In particular, buffer-file-type should be buffer-local again.
Please test (revision 110960).

There are still places in the Lisp sources that set buffer-file-type,
but with that variable buffer local, this should no longer affect the
other buffers.  I will remove the other references to this variable on
the trunk, as doing that might destabilize Emacs, due to the
complexities of the issues involved.

Re: Emacs24 coding system problems

[Kazuhiro Ito]

> > > > find-file-noselect is setting the variable.
> > > 
> > > It does?  Can you show me where?
> > 
> > find-file-not-found-set-buffer-file-coding-system in lisp/dos-w32.el.
> > lisp/dos-w32.el adds find-file-not-found-set-buffer-file-coding-system
> > to find-file-not-found-functions.
> 
> Thanks.  I committed to the emacs-24 branch changes that should fix
> this.  In particular, buffer-file-type should be buffer-local again.
> Please test (revision 110960).

Thanks for the fix. I confirmed the problem is fixed on trunk.

-- 
Kazuhiro Ito

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Wed, 28 Nov 2012 21:39:20 +0900
> From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
> Cc: usr.vm.rocks@gmail.com, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > > > > find-file-noselect is setting the variable.
> > > > 
> > > > It does?  Can you show me where?
> > > 
> > > find-file-not-found-set-buffer-file-coding-system in lisp/dos-w32.el.
> > > lisp/dos-w32.el adds find-file-not-found-set-buffer-file-coding-system
> > > to find-file-not-found-functions.
> > 
> > Thanks.  I committed to the emacs-24 branch changes that should fix
> > this.  In particular, buffer-file-type should be buffer-local again.
> > Please test (revision 110960).
> 
> Thanks for the fix. I confirmed the problem is fixed on trunk.

Thanks for testing.  I will keep this bug open until I remove all
traces of buffer-file-type from the trunk.

Re: Emacs24 coding system problems

[Uday S Reddy]

> Thanks.  I committed to the emacs-24 branch changes that should fix
> this.  In particular, buffer-file-type should be buffer-local again.
> Please test (revision 110960).

I got some very strange behaviours in VM when I tried this revision
(quite complicated to describe).  So not everything is well, and it will
take a while to fully test it.

But the decision to make it buffer-local again is certainly positive and
quite welcome.

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Wed, 28 Nov 2012 21:16:26 +0000
> From: Uday S Reddy <usr.vm.rocks@gmail.com>
> CC: Kazuhiro Ito <kzhr@d1.dion.ne.jp>, monnier@IRO.UMontreal.CA, 
>  emacs-devel@gnu.org
> 
> 
> > Thanks.  I committed to the emacs-24 branch changes that should fix
> > this.  In particular, buffer-file-type should be buffer-local again.
> > Please test (revision 110960).
> 
> I got some very strange behaviours in VM when I tried this revision
> (quite complicated to describe).

Is that strange behavior related to encoding and decoding non-ASCII
characters in any way?

Re: Emacs24 coding system problems

[Uday Reddy]

Eli Zaretskii writes:

> Is that strange behavior related to encoding and decoding non-ASCII
> characters in any way?

I can't say either way definitively.

To give a rough idea of what is happening, when I open an IMAP folder in VM,
it loads the cache folder that has only mail message headers.  Then it
fetches the body of the "current message" from the IMAP server and inserts
it in folder buffer at the right place based on the markers that have been
placed there when the cache folder was first parsed, and displays it to the
user.

The message body is getting inserted at wrong places, which means that the
markers are off somehow.

The folders are all unibyte ASCII buffers.  I was testing it with the
current message getting fetched is also ASCII.  So, the problems are arising
with unibyte buffers.

When I load uncompiled code of VM or place debug-on-entry on the relevant
code, the behaviour changes.  So, it isn't easy narrow down where the
problem is.

The markers are also off in the Summary window, which is a multibyte buffer.

What is worse, the problems are only noticeable at the very beginning of the
Emacs session.  They are not happening consistently throughout the Emacs
session.  So, I have to restart Emacs every time I want to try something
different.

Cheers,
Uday

Re: Emacs24 coding system problems

[Eli Zaretskii]

> Date: Fri, 30 Nov 2012 10:58:04 +0000
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Uday S Reddy <usr.vm.rocks@gmail.com>,
>     kzhr@d1.dion.ne.jp,
>     monnier@IRO.UMontreal.CA,
>     emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
> 
> > Is that strange behavior related to encoding and decoding non-ASCII
> > characters in any way?
> 
> I can't say either way definitively.
> 
> To give a rough idea of what is happening, when I open an IMAP folder in VM,
> it loads the cache folder that has only mail message headers.  Then it
> fetches the body of the "current message" from the IMAP server and inserts
> it in folder buffer at the right place based on the markers that have been
> placed there when the cache folder was first parsed, and displays it to the
> user.
> 
> The message body is getting inserted at wrong places, which means that the
> markers are off somehow.

If you do this on a collection of messages that don't have any
non-ASCII characters, and now CR-LF style EOLs, does the problem
happen then?  If it does, then decoding non-ASCII characters is not
the root cause.

Emacs24 unpredictable behaviour [Was: coding system problems]

[Uday Reddy]

Eli Zaretskii writes:

> If you do this on a collection of messages that don't have any
> non-ASCII characters, and now CR-LF style EOLs, does the problem
> happen then?  If it does, then decoding non-ASCII characters is not
> the root cause.

Messages are always made of ASCII characters.  So, that is not an issue.
The folder I am working with has Unix line endings.  The folder is loaded
into a unibyte buffer.  (VM always does that.)

Here is a sample piece of code that I inserted into the middle of VM, to
figure out how Emacs24 is behaving:

(defun vm-test-retrieve (mm)
  (save-current-buffer
    (set-buffer (vm-buffer-of mm))
    (goto-char (vm-text-of mm))
    (debug nil (list (point)
		     (marker-position (vm-text-of mm))
		     (= (point) (marker-position (vm-text-of mm)))
		     ))
    (debug nil (= (point) (marker-position (vm-text-of mm))))
    ))

Since goto-char is going to (vm-text-of mm), the first debug should
print two identical numbers and `t' for the comparison.  However, here is an
instance where it doesn't:

Debugger entered: ((99947 98744 nil))
  vm-test-retrieve([[#<marker at 93074 in test-imap> #<marker at 93107 in test-imap> nil #<marker at 98744 in test-imap> #<marker at 98744 in test-imap> #<marker at 98745 in test-imap>]...])

Even more crazily, for another message, the first debug prints this:

Debugger entered: ((104925 104925 t))
  vm-test-retrieve([[#<marker at 98745 in test-imap> #<marker at 98778 in test-imap> nil #<marker at 104925 in test-imap> #<marker at 104925 in test-imap> #<marker at 104926 in test-imap>]...])

and the second debug prints this:

Debugger entered: (nil)
  vm-test-retrieve([[#<marker at 98745 in test-imap> #<marker at 98778 in test-imap> nil #<marker at 104925 in test-imap> #<marker at 104925 in test-imap> #<marker at 104926 in test-imap>]...])

How does the point change from one debug to the next?  I have no idea.

--------

The good news is that these problems may not have anything to do with the
buffer-file-type changes.  I see them as far as back as revision 107968. 


Cheers,
Uday

Re: Emacs24 unpredictable behaviour [Was: coding system problems]

[Andreas Schwab]

Uday Reddy <usr.vm.rocks@gmail.com> writes:

> How does the point change from one debug to the next?  I have no idea.

Since (debug) triggers a redisplay it will set point in a displayed
buffer to window-point.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs24 unpredictable behaviour [Was: coding system problems]

[Uday Reddy]

Uday Reddy writes:

> Here is a sample piece of code that I inserted into the middle of VM, to
> figure out how Emacs24 is behaving:
> 
> (defun vm-test-retrieve (mm)
>   (save-current-buffer
>     (set-buffer (vm-buffer-of mm))
>     (goto-char (vm-text-of mm))
>     (debug nil (list (point)
> 		     (marker-position (vm-text-of mm))
> 		     (= (point) (marker-position (vm-text-of mm)))
> 		     ))
>     (debug nil (= (point) (marker-position (vm-text-of mm))))
>     ))
> 
> Since goto-char is going to (vm-text-of mm), the first debug should
> print two identical numbers and `t' for the comparison.  However, here is an
> instance where it doesn't:
> 
> Debugger entered: ((99947 98744 nil))
>   vm-test-retrieve([[#<marker at 93074 in test-imap> #<marker at 93107 in test-imap> nil #<marker at 98744 in test-imap> #<marker at 98744 in test-imap> #<marker at 98745 in test-imap>]...])

Following Andreas Schwab's prompt, I have changed the "debug" calls to
"message".  And, the misbehaviour of goto-char is still found:

(102332 98744 nil)
nil
98744 98744 t

The third line is printed by the regular VM code, which is right after this
call to `vm-test-retrieve'.  Surprisingly, in that instance, goto-char
works.  From this point on, VM appears to work normally.

The problem that causes buffer corruption happens earlier, which produced a
trace like this:

(104925 104925 t)
t
104925 104925 t
just before fetch 104925
just before insert 98778

The problem here is that between the "just before fetch" line and the 
"just before insert" line, the point changes.  The message body gets
inserted in the wrong place as a result.

I have narrowed down this point change to a call to
`auth-source-user-or-password' because, if I set `auth-sources' to nil, this
point change is avoided.  However, `auth-source-user-or-password' couldn't
have guessed the position 98778 where another marker is pointing.  So, the
problem must be some marker code of Emacs.

I suspect that if we can figure out why the `goto-char' above doesn't work,
we might be closer to identifying the problem.

Cheers,
Uday

Re: Emacs24 coding system problems

[Stefan Monnier]

>   - enable-multibyte-characters has the global value of t, but
>   - every time I visit a file, it gets turned off locally.
> The DOS line endings in files don't get recognized either.

Could you have set coding-system-for-read maybe?

> There must be something that VM does which fires this bug, but can't figure
> out what.  VM often creates unibyte buffers to load mail folders.  It uses a
> function like this to do so:

> (defun vm-generate-new-unibyte-buffer (name)
>   (if vm-xemacs-p
>       (generate-new-buffer name)
>     (let* ((buffer (generate-new-buffer name)))
>       (when (fboundp 'set-buffer-multibyte)
> 	(with-current-buffer buffer
> 	  (set-buffer-multibyte nil)))
>       buffer)))

> I am not sure if this is violating any protocols.

No, this code looks sane.


        Stefan