saveplace: don't ask for coding system

saveplace: don't ask for coding system

[David Reitter]

Can I suggest the following two changes to saveplace.el?

- set the coding system so that the user isn't asked about a coding  
system for the places file every time they quit Emacs (when file names  
contain non-standard characters).

- do not show a message about saving - it's not useful information for  
most users and contributes to information overload!
   (if the function was interactive, this would be a different story.)


I have no write access to CVS, so if people like such changes,  
somebody will have to commit them.



*** lisp/saveplace.el	25 Jul 2007 05:47:04 +0100	1.37.2.1
--- lisp/saveplace.el	28 Nov 2007 14:40:39 +0000	
***************
*** 207,213 ****
   (defun save-place-alist-to-file ()
     (let ((file (expand-file-name save-place-file)))
       (save-excursion
!       (message "Saving places to %s..." file)
         (set-buffer (get-buffer-create " *Saved Places*"))
         (delete-region (point-min) (point-max))
         (when save-place-forget-unreadable-files
--- 207,213 ----
   (defun save-place-alist-to-file ()
     (let ((file (expand-file-name save-place-file)))
       (save-excursion
!       ;; (message "Saving places to %s..." file)
         (set-buffer (get-buffer-create " *Saved Places*"))
         (delete-region (point-min) (point-max))
         (when save-place-forget-unreadable-files
***************
*** 224,233 ****
                  t))))
   	(condition-case nil
   	    ;; Don't use write-file; we don't want this buffer to visit it.
! 	    (write-region (point-min) (point-max) file)
! 	  (file-error (message "Can't write %s" file)))
           (kill-buffer (current-buffer))
!         (message "Saving places to %s...done" file)))))

   (defun load-save-place-alist-from-file ()
     (if (not save-place-loaded)
--- 224,235 ----
                  t))))
   	(condition-case nil
   	    ;; Don't use write-file; we don't want this buffer to visit it.
! 	    (let ((coding-system-for-write 'utf-8))
! 	      (write-region (point-min) (point-max) file))
! 	  (file-error (message "Saving places: Can't write %s" file)))
           (kill-buffer (current-buffer))
!         ;;(message "Saving places to %s...done" file)
! 	))))

   (defun load-save-place-alist-from-file ()
     (if (not save-place-loaded)

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> From: David Reitter <dreitter@inf.ed.ac.uk>
> Date: Wed, 28 Nov 2007 14:46:08 +0000
> 
>    	    ;; Don't use write-file; we don't want this buffer to visit it.
> ! 	    (let ((coding-system-for-write 'utf-8))
> ! 	      (write-region (point-min) (point-max) file))

FWIW, I don't think UTF-8 is a good idea here, since it cannot encode
all the characters Emacs supports.  Maybe emacs-mule?

> !       ;; (message "Saving places to %s..." file)

I think there should be a user option to disable this, instead of
disabling it unconditionally.  I, for example, quite like this
message.

Re: saveplace: don't ask for coding system

[David Reitter]

On 28 Nov 2007, at 21:47, Eli Zaretskii wrote:

> FWIW, I don't think UTF-8 is a good idea here, since it cannot encode
> all the characters Emacs supports.  Maybe emacs-mule?

You'll know best what should be used. The coding system should support  
all characters in file names (that are supported).

>> !       ;; (message "Saving places to %s..." file)
>
> I think there should be a user option to disable this, instead of
> disabling it unconditionally.  I, for example, quite like this
> message.


If you like this message, presumably you'll like "such" messages in  
general.

I wonder if this hasn't come up before: could we not have a global  
verbosity level? If it is well-defined, all parts including modes  
could respect it.
One could have an optional argument to `message', indicating the level  
of the particular message.

In a case like the above (and, for the sake of the argument, we assume  
places could be saved interactively), I would go for a medium to low  
level, indicating progress at a usually quick process, which is, by  
default, only displayed when the function was called interactively.

General variable for verbosity level. (was: saveplace: don't ask for coding system)

[Karl Fogel]

David Reitter <david.reitter@gmail.com> writes:
> I wonder if this hasn't come up before: could we not have a global
> verbosity level? If it is well-defined, all parts including modes
> could respect it.
>
> One could have an optional argument to `message', indicating the level
> of the particular message.

In this change:

   http://cvs.savannah.gnu.org/viewvc/emacs/lisp/saveplace.el?root=emacs&r1=1.38&r2=1.39

...I added a boolean that tells saveplace.el whether or not to print
loading/saving messages.  David Reitter rightly points out that it
would make more sense to have this be a generic variable that many
packages in Emacs could use.  For example, it could subsume these two:

  gmm-message
  Function: If LEVEL is lower than `gmm-verbose' print ARGS using `message'.

  gnus-message
  Function: If LEVEL is lower than `gnus-verbose' print ARGS using `message'.

(No doubt there are other examples elsewhere in Emacs.)

The purpose of this email is to figure out whether/how we should
implement such a system.  Proposal: a new function and new variable:

   maybe-message LEVEL FORMAT_STRING &optional FORMAT_ARGS
   Function: Print FORMAT_STRING formatted with FORMAT_ARGS iff LEVEL
             is equal to or higher than `message-verbosity-level'.

   message-verbosity-level
   Variable: Say which messages are printed by `maybe-message' (which see)
             in non-interactive contexts.

For interactive functions, `maybe-message' should behave just like
`message' does today.  But should `message-verbosity-level' be a
number, or something else?  It could be, for example, a symbol:

   message-verbosity-level:
     `none'       ==>  print no messages (except errors)
     `important'  ==>  print only important messages
     `regular'    ==>  ...

This has the advantages of being clearer to users, and of allowing us
to cleanly insert new verbosity levels in the future.

I'm avoiding proposing a system of overlapping categories; that is,
where you'd set the variable to a list of symbols, and maybe-message
would print iff the level specified by the code matched any of them.
While that would be the most flexible system, I'm not sure we really
need that level of complexity here.

Please comment.  Feel free to trash this system in favor of something
entirely different -- I'm not wedded to it, I'm just trying to get the
conversation started.

-Karl

Re: General variable for verbosity level. (was: saveplace: don't ask for coding system)

[David Reitter]

On 3 Dec 2007, at 01:04, Karl Fogel wrote:
>
>   message-verbosity-level
>   Variable: Say which messages are printed by `maybe-message' (which  
> see)
>             in non-interactive contexts.

Yes, I would either log all messages to *Messages*, or also create a  
customization variable:

log-verbosity-level
Variable: Maximum level of messages logged to *Messages* by `maybe- 
message'.

>  But should `message-verbosity-level' be a
> number, or something else?  It could be, for example, a symbol:

I think the symbol would be okay, because it is easier to understand,  
and also quite easy to extend.
But some modes (like tramp-mode) already have a numeric system of up  
to ten verbosity levels.

So, perhaps one could allow for numbers, and allow symbols as a  
clearer form that translate directly to numbers, e.g. `none' would be  
0, `important' 2, `verbose' 5,  `all' 10. That way, verbosity is  
indicates as an integer 0..10, with 10 being the most verbose.

That way, we would not lose any specificity in modes like tramp-mode,  
but retain a clean and simply interface for the user.

(But to be honest, either way looks good to me.)

There could be a recommendation for externally maintained modes to  
take use maybe-message, or take this verbosity level as the default  
for an existing verbosity customization.

Re: General variable for verbosity level.

[Karl Fogel]

I got rid of the regular loading/saving messages in saveplace.el,
which is what David originally suggested.

Eli spoke up in favor of having such messages, but it's not clear that
the majority of users know or care about them, and loading/saving
places is such a fast operation that there's no user-visible delay to
explain.

It may well be that Emacs should have a general variable to control
verbosity level, and if so, I'd certainly be in favor of saveplace.el
using that variable.  But since we don't currently have that variable,
saveplace.el has to make a decision, and I think the best choice is
not to display the messages (even though, as with any behavior, there
is always someone out there who wishes otherwise, no doubt for
perfectly good reasons).

I started a thread on having such a general variable, but -- my
apologies -- I do not have time to actually implement it or even play
honest broker in the discussion.  So I'm withdrawing from that thread.
If it results in actual code, though, I'll happily make whatever
change to saveplace.el is needed to use the new system.

-Karl

Re: General variable for verbosity level.

[Reiner Steib]

On Mon, Dec 03 2007, Karl Fogel wrote:

> For interactive functions, `maybe-message' should behave just like
> `message' does today.  But should `message-verbosity-level' be a
> number, or something else?  It could be, for example, a symbol:
>
>    message-verbosity-level:
>      `none'       ==>  print no messages (except errors)
>      `important'  ==>  print only important messages
>      `regular'    ==>  ...
>
> This has the advantages of being clearer to users, and of allowing us
> to cleanly insert new verbosity levels in the future.

The doc string of `gmm-message' gives some additional information on
the levels:

,----[ <f1> f gmm-message RET ]
| gmm-message is a compiled Lisp function in `gmm-utils.el'.
| (gmm-message level &rest args)
| 
| If level is lower than `gmm-verbose' print args using `message'.
| 
| Guideline for numbers:
| 1 - error messages, 3 - non-serious error messages, 5 - messages for things
| that take a long time, 7 - not very important messages on stuff, 9 - messages
| inside loops.
`----

Related to this: Users have asked to add a timestamp to the message
(in the *Messages* buffers).  I think this might be useful.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Nov 28, 2007 10:47 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> FWIW, I don't think UTF-8 is a good idea here, since it cannot encode
> all the characters Emacs supports.  Maybe emacs-mule?

Hmm... Didn't we agree, after a lengthy discussion, that UTF-8 was the
best coding system to save the ido history file (which contains
filenames, like the saveplace file)?

             Juanma

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Date: Thu, 29 Nov 2007 10:04:04 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: "David Reitter" <dreitter@inf.ed.ac.uk>, emacs-devel@gnu.org
> 
> Hmm... Didn't we agree, after a lengthy discussion, that UTF-8 was the
> best coding system to save the ido history file

Where did "we" agree to that? in what thread?

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Nov 29, 2007 10:04 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Where did "we" agree to that? in what thread?

http://thread.gmane.org/gmane.emacs.devel/82804

Perhaps I understood it wrong, in which case I'd like to ask again
what coding system to use for the ido history file...

             Juanma

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Nov 28, 2007 3:46 PM, David Reitter <dreitter@inf.ed.ac.uk> wrote:

> --- 224,235 ----
>                   t))))
>         (condition-case nil
>             ;; Don't use write-file; we don't want this buffer to visit it.
> !           (let ((coding-system-for-write 'utf-8))
> !             (write-region (point-min) (point-max) file))
> !         (file-error (message "Saving places: Can't write %s" file)))
>            (kill-buffer (current-buffer))
> !         ;;(message "Saving places to %s...done" file)
> !       ))))
>

I think you should also add a "-*- coding: XXX -*-" line to the file.

             Juanma

Re: saveplace: don't ask for coding system

[David Reitter]

On 29 Nov 2007, at 09:05, Juanma Barranquero wrote:
>
>
> I think you should also add a "-*- coding: XXX -*-" line to the file.


Good idea. I left the coding system at `utf-8', because that's what  
ido seems to use. If there are characters not in utf-8 that file  
systems accept  for file names, it should probably be emacs-mule.

I'll post a separate patch for log level based message filtering  
another time.


2007-12-02  David Reitter <david.reitter@gmail.com>

	* saveplace.el (load-save-place-alist-from-file): set coding system
	for places file and improve write error message.




*** lisp/saveplace.el	25 Jul 2007 05:47:04 +0100	1.37.2.1
--- lisp/saveplace.el	02 Dec 2007 16:44:59 +0000	
***************
*** 205,215 ****
         (setq save-place-alist (cdr save-place-alist)))))

   (defun save-place-alist-to-file ()
!   (let ((file (expand-file-name save-place-file)))
       (save-excursion
         (message "Saving places to %s..." file)
         (set-buffer (get-buffer-create " *Saved Places*"))
         (delete-region (point-min) (point-max))
         (when save-place-forget-unreadable-files
   	(save-place-forget-unreadable-files))
         (let ((print-length nil)
--- 205,220 ----
         (setq save-place-alist (cdr save-place-alist)))))

   (defun save-place-alist-to-file ()
!   (let ((file (expand-file-name save-place-file))
! 	(coding-system-for-write 'utf-8))
       (save-excursion
         (message "Saving places to %s..." file)
         (set-buffer (get-buffer-create " *Saved Places*"))
         (delete-region (point-min) (point-max))
+       (insert
+        (format ";; -*- mode: emacs-lisp; coding: %s -*-\n"
+ 	       coding-system-for-write)
+        ";; Positions for files, automatically generated by  
`saveplace'.\n\n")
         (when save-place-forget-unreadable-files
   	(save-place-forget-unreadable-files))
         (let ((print-length nil)
***************
*** 225,231 ****
   	(condition-case nil
   	    ;; Don't use write-file; we don't want this buffer to visit it.
   	    (write-region (point-min) (point-max) file)
! 	  (file-error (message "Can't write %s" file)))
           (kill-buffer (current-buffer))
           (message "Saving places to %s...done" file)))))

--- 230,236 ----
   	(condition-case nil
   	    ;; Don't use write-file; we don't want this buffer to visit it.
   	    (write-region (point-min) (point-max) file)
! 	  (file-error (message "Saving places: Can't write %s" file)))
           (kill-buffer (current-buffer))
           (message "Saving places to %s...done" file)))))

Re: saveplace: don't ask for coding system

[Karl Fogel]

David Reitter <dreitter@inf.ed.ac.uk> writes:
>> I think you should also add a "-*- coding: XXX -*-" line to the file.
>
> Good idea. I left the coding system at `utf-8', because that's what
> ido seems to use. If there are characters not in utf-8 that file
> systems accept  for file names, it should probably be emacs-mule.

I already committed a version of this change:

  $ head -7 lisp/ChangeLog
  2007-12-02  Karl Fogel  <kfogel@red-bean.com>
  
        * saveplace.el (save-place-alist-to-file): Use `utf-8' coding
        system when writing, and set it in the first-line file variables.
        Based on a patch by David Reitter <dreitter{_AT_}inf.ed.ac.uk>
        and a suggestion by Juanma Barranquero.
  
  $ cvs ci -F lisp/msg lisp/saveplace.el lisp/ChangeLog
  /cvsroot/emacs/emacs/lisp/saveplace.el,v  <--  lisp/saveplace.el
  new revision: 1.40; previous revision: 1.39
  /cvsroot/emacs/emacs/lisp/ChangeLog,v  <--  lisp/ChangeLog
  new revision: 1.12354; previous revision: 1.12353
  Mailing notification to emacs-commit@gnu.org... sent.
  Mailing diffs to emacs-diffs@gnu.org... 2 sent.
  $ 

I think utf-8 will be fine, practically speaking.

Just now, I've made a followup tweak based on your most recent patch:

  $ head -6 lisp/ChangeLog
  2007-12-02  Karl Fogel  <kfogel@red-bean.com>
  
        * saveplace.el (save-place-alist-to-file): Set
        coding-system-for-write once and refer to it throughout.
        Based on a patch by David Reitter <dreitter{_AT_}inf.ed.ac.uk>.
  
  $ cvs ci -F lisp/msg lisp/saveplace.el lisp/ChangeLog
  /cvsroot/emacs/emacs/lisp/saveplace.el,v  <--  lisp/saveplace.el
  new revision: 1.41; previous revision: 1.40
  /cvsroot/emacs/emacs/lisp/ChangeLog,v  <--  lisp/ChangeLog
  new revision: 1.12355; previous revision: 1.12354
  Mailing notification to emacs-commit@gnu.org... sent.
  Mailing diffs to emacs-diffs@gnu.org... 2 sent.
  $

> I'll post a separate patch for log level based message filtering
> another time.

I already committed a version of that today too:

   $ head -9 lisp/ChangeLog 
   2007-12-02  Karl Fogel  <kfogel@red-bean.com>
   
   	Offer option for saveplace to be quiet about loading and saving.
   	Suggested by: David Reitter <dreitter{_AT_}inf.ed.ac.uk>
   
   	* lisp/saveplace.el (save-place-quiet): New customizable boolean.
   	(save-place-alist-to-file, load-save-place-alist-from-file): Use it
   	to determine whether to print loading/saving messages.
   
   $ cvs ci -F m lisp/saveplace.el lisp/ChangeLog
   /cvsroot/emacs/emacs/lisp/saveplace.el,v  <--  lisp/saveplace.el
   new revision: 1.39; previous revision: 1.38
   /cvsroot/emacs/emacs/lisp/ChangeLog,v  <--  lisp/ChangeLog
   new revision: 1.12351; previous revision: 1.12350
   Mailing notification to emacs-commit@gnu.org... sent.
   Mailing diffs to emacs-diffs@gnu.org... 2 sent.
   $ 

Best,
-Karl

Re: saveplace: don't ask for coding system

[David Reitter]

On 2 Dec 2007, at 21:26, Karl Fogel wrote:
>
>   	Offer option for saveplace to be quiet about loading and saving.
>   	Suggested by: David Reitter <dreitter{_AT_}inf.ed.ac.uk>

That is not what I suggested. I wanted a global customization option  
that would allow users to specify a "log level", and a change to  
`message' to take the log level into account.

It is overkill to add countless extra options to every package to make  
it shut up. You're not doing users a favor.

Are you committing changes to 22.x? The save-place file coding seems  
like a bug to me and I would welcome a fix for it in the 22 branch as  
well.

Re: saveplace: don't ask for coding system

[Karl Fogel]

David Reitter <dreitter@inf.ed.ac.uk> writes:
>> Offer option for saveplace to be quiet about loading and saving.
>> Suggested by: David Reitter <dreitter{_AT_}inf.ed.ac.uk>
>
> That is not what I suggested. I wanted a global customization option
> that would allow users to specify a "log level", and a change to
> message' to take the log level into account.
>
> It is overkill to add countless extra options to every package to make
> it shut up. You're not doing users a favor.

Yeah, I kind of agree -- already on your side, really :-).

The best strategy would be to start a separate thread, to figure out
what the log-level (or message-level, whatever) variable should be
called, where it should be defined, and -- most importantly -- what
the various verbosity levels should be.  If you would like to initiate
that thread and drive the discussion, we can probably get that settled
reasonably quickly.  Then I'll switch saveplace (and other packages I
help maintain) to use the new system.

If you don't have time to do that, please say so, and I'll try to take
care of it.  (Perhaps I shouldn't have committed the current change to
saveplace first, but I wanted to leave markers for where the new code
should be hooked in.)

> Are you committing changes to 22.x? The save-place file coding seems
> like a bug to me and I would welcome a fix for it in the 22 branch as
> well.

Sure, I can port the coding-system change over to the EMACS_22_BASE
branch (I'm assuming that's the right one; it's sometimes hard to tell
what branches serve what purposes here).

-Karl

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> From: Karl Fogel <kfogel@red-bean.com>
> Date: Sun, 02 Dec 2007 15:23:17 -0800
> Cc: Juanma Barranquero <lekktu@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> 
> > Are you committing changes to 22.x? The save-place file coding seems
> > like a bug to me and I would welcome a fix for it in the 22 branch as
> > well.
> 
> Sure, I can port the coding-system change over to the EMACS_22_BASE
> branch

But please, do not use utf-8 on the EMACS_22_BASE branch: it doesn't
support several character sets that EMACS_22_BASE still uses.  (I
don't know why you disregarded discussions of this for the changes on
the trunk, but since the trunk will not be released before the unicode
merge, maybe that's okay.)

Re: saveplace: don't ask for coding system

[Karl Fogel]

Eli Zaretskii <eliz@gnu.org> writes:
>> Sure, I can port the coding-system change over to the EMACS_22_BASE
>> branch
>
> But please, do not use utf-8 on the EMACS_22_BASE branch: it doesn't
> support several character sets that EMACS_22_BASE still uses.  (I
> don't know why you disregarded discussions of this for the changes on
> the trunk, but since the trunk will not be released before the unicode
> merge, maybe that's okay.)

I didn't disregard the discussions, I simply thought they expressed no
clear preference for another coding system over utf-8.  If there is a
better solution, we should use that, and I'd be happy to make the
change.

In the thread, you objected to utf-8 because Emacs-mule supports some
characters that utf-8 doesn't (I hadn't known that, but I'll take your
word for it).  However, Juanma pointed out that utf-8 was used for the
ido-history file, and that the reason this was okay was that the data
is just filenames (as with saveplace); he pointed to a thread on this.

In *that* thread, there was no clear conclusion, but the last mail I
see is from Kenichi Handa (whose judgement I usually trust!) saying, I
think, that utf-8 ought to be fine for filenames, including on the
EMACS_22_BASE branch.

It would be more helpful if, instead of saying what encoding *not* to
use, you (or someone) would say what encoding *to* use, and why :-).

Thanks,
-Karl

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 3, 2007 11:59 AM, Karl Fogel <kfogel@red-bean.com> wrote:

> It would be more helpful if, instead of saying what encoding *not* to
> use, you (or someone) would say what encoding *to* use, and why :-).

Yes.

I'm in fact following this thread to know what (if anything) to do with ido...

What I find puzzling is that, in the ido thread, I understood Eli's
messages as being *for* using UTF-8:

http://article.gmane.org/gmane.emacs.devel/82823
http://article.gmane.org/gmane.emacs.devel/82855
http://article.gmane.org/gmane.emacs.devel/82863

             Juanma

Re: saveplace: don't ask for coding system

[Stephen J. Turnbull]

Juanma Barranquero writes:
 > On Dec 3, 2007 11:59 AM, Karl Fogel <kfogel@red-bean.com> wrote:
 > 
 > > It would be more helpful if, instead of saying what encoding *not* to
 > > use, you (or someone) would say what encoding *to* use, and why :-).
 > 
 > Yes.

Use UTF-8 in NFD form.  A protocol for handling the gaiji problem
(characters not encoded by whatever system your application uses) will
have to be developed anyway, and you can use that to handle all the
problems Eli brings up that have to do with "what Mule code can do
that Unicode can't".  There are other problems of ambiguity in Unicode
itself (see Unicode TR#31), but since Mule code can (at least
theoretically) represent all Unicode code points, Mule code has them
too.

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Cc: dreitter@inf.ed.ac.uk,  lekktu@gmail.com,  emacs-devel@gnu.org
> From: Karl Fogel <kfogel@red-bean.com>
> Date: Mon, 03 Dec 2007 02:59:33 -0800
> 
> It would be more helpful if, instead of saying what encoding *not* to
> use, you (or someone) would say what encoding *to* use

I thought that was obvious: emacs-mule.

> and why :-).

I thought that was obvious: because it supports all the characters
Emacs can handle.

> In the thread, you objected to utf-8 because Emacs-mule supports some
> characters that utf-8 doesn't (I hadn't known that, but I'll take your
> word for it).

In Emacs 22, only mule-unicode-* characters can be encoded with
UTF-8.  We do a lot of juggling behind the scenes to pretend that
characters from other charsets are somehow equivalent to (or mapped
into) Unicode codepoints, but that's camouflage.  Cyrillic ISO-8859-5
characters and Cyrillic mule-unicode-* characters are still different
characters, for example.

> However, Juanma pointed out that utf-8 was used for the
> ido-history file, and that the reason this was okay was that the data
> is just filenames (as with saveplace); he pointed to a thread on this.

That discussion was about Emacs 23, which will be Unicode based and
where UTF-8 will be able to support all characters (or so I think).
EMACS_22_BASE is a stable branch, so we are not supposed to install
there changes that _might_ work, only changes we know _for_sure_ they
_will_ work.

Re: saveplace: don't ask for coding system

[Karl Fogel]

Eli Zaretskii <eliz@gnu.org> writes:
>> It would be more helpful if, instead of saying what encoding *not* to
>> use, you (or someone) would say what encoding *to* use
>
> I thought that was obvious: emacs-mule.
>
>> and why :-).
>
> I thought that was obvious: because it supports all the characters
> Emacs can handle.

The disadvantage is that emacs-mule is Emacs-specific.  It would be
better if the file were readable by other editors and viewers --
although it's only parsed by Emacs (as far as I know), it *is*
human-readable, and we should take that into account when choosing an
encoding.

I've always thought of emacs-mule as a purely internal encoding
system.  If we're starting to push it out to files, that feels a bit
like mission creep to me; it's not a step I'm very comfortable with.

> That discussion was about Emacs 23, which will be Unicode based and
> where UTF-8 will be able to support all characters (or so I think).
> EMACS_22_BASE is a stable branch, so we are not supposed to install
> there changes that _might_ work, only changes we know _for_sure_ they
> _will_ work.

So a good solution might be:

   1. write it out using utf-8 on the trunk (Emacs 23) line
   2. write it out using emacs-mule on the Emacs 22 line
   3. make sure that saveplace.el reads the coding system from the
      first line of the file and uses that to interpret the rest of
      the file

Manual testing just now indicates that ";;; -*- coding: utf-8 -*-"
will look the same (i.e., be the same sequence of bytes) in both
coding systems, and that therefore step (3) above would work.

Does this seem reasonable to you?

(I'm not volunteering for (2) or (3), unfortunately.  I've done (1),
by installing what seems to be the right change for Emacs 23.  I wish
I could do more, but time is limited.  It is still useful to describe
what's needed for Emacs 22, though, so someone else knows what to do
if they want.  I admit that it's partly that I'm not enthusiastic
about mule as an external storage encoding anyway.)

-Karl

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Cc: lekktu@gmail.com,  dreitter@inf.ed.ac.uk,  emacs-devel@gnu.org
> From: Karl Fogel <kfogel@red-bean.com>
> Date: Mon, 03 Dec 2007 13:32:14 -0800
> 
> The disadvantage is that emacs-mule is Emacs-specific.

I don't see this as a big problem in this case: this file is useless
for other applications, and almost unreadable by humans.  I use this
feature all the time, and I only looked at the file perhaps once or
twice.  It's just a huge Lisp expression.

> I've always thought of emacs-mule as a purely internal encoding
> system.

Yes; and saveplace file is an internal Emacs file, not unlike
auto-save files.

> If we're starting to push it out to files, that feels a bit
> like mission creep to me; it's not a step I'm very comfortable with.

I really don't see a rational reason for this uneasiness.  emacs-mule
is almost iso-2022; would you feel uneasy with that, too?

>    1. write it out using utf-8 on the trunk (Emacs 23) line
>    2. write it out using emacs-mule on the Emacs 22 line
>    3. make sure that saveplace.el reads the coding system from the
>       first line of the file and uses that to interpret the rest of
>       the file
> 
> Manual testing just now indicates that ";;; -*- coding: utf-8 -*-"
> will look the same (i.e., be the same sequence of bytes) in both
> coding systems, and that therefore step (3) above would work.
> 
> Does this seem reasonable to you?

Yes.

> (I'm not volunteering for (2) or (3), unfortunately.

Too bad.

Re: saveplace: don't ask for coding system

[Karl Fogel]

Eli Zaretskii <eliz@gnu.org> writes:
> I don't see this as a big problem in this case: this file is useless
> for other applications, and almost unreadable by humans.  I use this
> feature all the time, and I only looked at the file perhaps once or
> twice.  It's just a huge Lisp expression.

It's not useless for other applications; they could parse it.
However, I don't know of any actual examples of other applications
doing that, so my argument isn't terribly compelling.

>>    1. write it out using utf-8 on the trunk (Emacs 23) line
>>    2. write it out using emacs-mule on the Emacs 22 line
>>    3. make sure that saveplace.el reads the coding system from the
>>       first line of the file and uses that to interpret the rest of
>>       the file
>> 
>> Does this seem reasonable to you?
>
> Yes.
>
>> (I'm not volunteering for (2) or (3), unfortunately.
>
> Too bad.

Further testing indicates that we get (3) for free (probably due to
the way `find-operation-coding-system' works?).  So I've implemented
(2) now, and I think that means this is done.  

   /cvsroot/emacs/emacs/lisp/saveplace.el,v  <--  saveplace.el
   new revision: 1.37.2.2; previous revision: 1.37.2.1

The upgrade from Emacs 22 to later versions should go smoothly, since
Emacs will always read with the coding system specified by the file,
and write whatever that version of Emacs should write.  (And in fact,
even downgrading is unlikely to result in problems, since either utf-8
or emacs-mule will work in the common cases for both versions.)

-Karl

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Cc: lekktu@gmail.com,  dreitter@inf.ed.ac.uk,  emacs-devel@gnu.org
> From: Karl Fogel <kfogel@red-bean.com>
> Date: Mon, 03 Dec 2007 22:17:52 -0800
> 
> Further testing indicates that we get (3) for free (probably due to
> the way `find-operation-coding-system' works?).  So I've implemented
> (2) now, and I think that means this is done.  

Thanks!

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 4, 2007 5:20 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Yes; and saveplace file is an internal Emacs file, not unlike
> auto-save files.

I've followed suit and used emacs-mule for the ido history file in Emacs 22.

Which brings forth a similar case: the savehist-file (from
savehist.el) is by default saved in utf-8 (at least on recent XEmacs
and Emacs). However, the eol-conversion is left for Emacs to decide.

That's Not Good when some of the saved variables contain "^M" or "^J"
on a non-unix environment. For example, I did some "^M^J" -> "^J"
replacements (to fix a garbled patch) on Windows, and in subsequent
invocations of Emacs the contents of query-replace-history got sillier
and sillier: "^M^M^J", "^M^M^M^M^J", etc.

Now, savehist.el is friendly enough to contain a variable
`savehist-coding-system', so it's easy to fix in your own .emacs. The
question is, for a file whose intention is to preserve exactly some
variable's contents, wouldn't be a better default to use utf-8-unix?

             Juanma

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Date: Tue, 4 Dec 2007 17:40:57 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: "Emacs Devel" <emacs-devel@gnu.org>
> 
> Which brings forth a similar case: the savehist-file (from
> savehist.el) is by default saved in utf-8 (at least on recent XEmacs
> and Emacs). However, the eol-conversion is left for Emacs to decide.

When saving history, there's nothing to decide: savehist uses a
temporary buffer, which is born with the native EOL convention of the
underlying platform: -unix on Posix platforms, -dos on Windows.

> That's Not Good when some of the saved variables contain "^M" or "^J"
> on a non-unix environment.

So you mean when reading the history file, yes?

> For example, I did some "^M^J" -> "^J"
> replacements (to fix a garbled patch) on Windows, and in subsequent
> invocations of Emacs the contents of query-replace-history got sillier
> and sillier: "^M^M^J", "^M^M^M^M^J", etc.
> 
> Now, savehist.el is friendly enough to contain a variable
> `savehist-coding-system', so it's easy to fix in your own .emacs. The
> question is, for a file whose intention is to preserve exactly some
> variable's contents, wouldn't be a better default to use utf-8-unix?

I don't think it's better, because a lone ^M can fool Emacs even if we
use -unix.  What would help, I think, is have an explicit EOL
conversion stated in the coding: cookie.  Then the fragile guesswork
will be bypassed.

Re: saveplace: don't ask for coding system

[Stefan Monnier]

> I don't think it's better, because a lone ^M can fool Emacs even if we
> use -unix.  What would help, I think, is have an explicit EOL
> conversion stated in the coding: cookie.  Then the fragile guesswork
> will be bypassed.

You're right that the most important part is to specify the EOL in the
coding cookie.  But I also think that it should always use `unix' EOL
when writing the file, because that's the only coding-system that's
safe: both the `dos' and the `mac' EOLs can result in data loss when the
text being saved includes ^M and ^J characters.  It's minor, but if
we're going to insist on emacs-mule in place of utf-8, then we should
insist on `unix' in place of `dos' or `mac' style EOL.


        Stefan

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 04 Dec 2007 17:24:52 -0500
> Cc: Juanma Barranquero <lekktu@gmail.com>, emacs-devel@gnu.org
> 
> You're right that the most important part is to specify the EOL in the
> coding cookie.  But I also think that it should always use `unix' EOL
> when writing the file, because that's the only coding-system that's
> safe: both the `dos' and the `mac' EOLs can result in data loss when the
> text being saved includes ^M and ^J characters.  It's minor, but if
> we're going to insist on emacs-mule in place of utf-8, then we should
> insist on `unix' in place of `dos' or `mac' style EOL.

I'm fine with -unix, but I won't recommend changing that on the Emacs
22 branch, as there's no bug we are fixing here.

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 5, 2007 5:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> I'm fine with -unix, but I won't recommend changing that on the Emacs
> 22 branch, as there's no bug we are fixing here.

Why is not a bug to pass ^M^J to M-x replace-string and end up with
^M^M^M^M...^M^J in the history (after a suitable number of Emacs
reloads)?

             Juanma

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Date: Wed, 5 Dec 2007 09:49:34 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: "Stefan Monnier" <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> On Dec 5, 2007 5:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I'm fine with -unix, but I won't recommend changing that on the Emacs
> > 22 branch, as there's no bug we are fixing here.
> 
> Why is not a bug to pass ^M^J to M-x replace-string and end up with
> ^M^M^M^M...^M^J in the history (after a suitable number of Emacs
> reloads)?

Because it wasn't reported by anyone until now, and the current
encoding was in savehist from day one.  So IMO this change is not safe
enough and the problem it tries to solve is not important enough to
risk the stability of Emacs 22 branch.

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 5, 2007 7:25 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Because it wasn't reported by anyone until now, and the current
> encoding was in savehist from day one.  So IMO this change is not safe
> enough and the problem it tries to solve is not important enough to
> risk the stability of Emacs 22 branch.

Color me puzzled.

- The problem has one reporter (me), but so do many other bugs we've
been fixing.
- The fix is quite simple: changing utf-8 (which is already used by
savehist) to utf-8-unix.
- The fix affects a file which is only used by Emacs, and in fact,
only by the package that creates it.

How could that not be "safe enough"?

As for "not important", I agree, and moreover, there's an easy
workaround (setting savehist-coding-system in your .emacs); so I'm not
pushing for fixing in in EMACS_22_BASE. But I honestly don't
understand the reticence.

Juanma

Re: saveplace: don't ask for coding system

[Eli Zaretskii]

> Date: Wed, 5 Dec 2007 21:05:39 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> - The problem has one reporter (me), but so do many other bugs we've
> been fixing.

You didn't actually encounter the bug, you deliberately created this
example to verify that your (correct) theory about problems with
leaving the EOL conversion unspecified.

> But I honestly don't understand the reticence.

It's okay to disagree; there's no need to dig deeper trying to
understand differences of judgment.

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 6, 2007 5:21 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> You didn't actually encounter the bug, you deliberately created this
> example to verify that your (correct) theory about problems with
> leaving the EOL conversion unspecified.

Sorry, but that's not true. Excuse my self-quoting:

"For example, I did some "^M^J" -> "^J"
replacements (to fix a garbled patch) on Windows, and in subsequent
invocations of Emacs the contents of query-replace-history got sillier
and sillier: "^M^M^J", "^M^M^M^M^J", etc."

What happened was that I had a patch with garbled EOL sequences. I did
some replacement and applied the patch (from inside Emacs). Next time
I tried to do the same I was puzzled to see ^M^M^J in the replacement
string. Truth be told, it wasn't even the first time it happened to
me, just the first time I took the time to investigate it. I suppose I
have some suboptimal diff/patch tools which make the problem more
likely.

> It's okay to disagree; there's no need to dig deeper trying to
> understand differences of judgment.

I trust your judgement on Emacs issues; I was interested in knowing
what was I missing.

             Juanma

Re: saveplace: don't ask for coding system

[Juanma Barranquero]

On Dec 4, 2007 10:32 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> When saving history, there's nothing to decide: savehist uses a
> temporary buffer, which is born with the native EOL convention of the
> underlying platform: -unix on Posix platforms, -dos on Windows.

I don't understand. Wouldn't a -unix coding force it to be saved with
Unix EOL convention?

> So you mean when reading the history file, yes?

Well, I do "M-x replace-string ^M^J <RET> ^J <RET>", and the savehist
file contains 0x0D 0x0A and 0x0D 0x0D 0x0A.

> What would help, I think, is have an explicit EOL
> conversion stated in the coding: cookie.

We are miscommunicating somehow. The value of `savehist-coding-system'
is used to write the file, and is also written in the coding: cookie.
When I propose to use a -unix variant to write the file, I of course
mean to also write it in the cookie.

             Juanma

Re: saveplace: don't ask for coding system

[Richard Stallman]

    The disadvantage is that emacs-mule is Emacs-specific.  It would be
    better if the file were readable by other editors and viewers --
    although it's only parsed by Emacs (as far as I know), it *is*
    human-readable, and we should take that into account when choosing an
    encoding.

I agree that would be an advantage, but the first priority has to be
to make the feature _work_ in all cases.

    I've always thought of emacs-mule as a purely internal encoding
    system.  If we're starting to push it out to files,

It was never supposed to be "purely internal".  There is nothing wrong
with that.