One example of code I can't understand

One example of code I can't understand

[Richard Stallman]

Here's more code from mm-util.el that I don't understand.
I don't know what this function is supposed to do
or how it works.


(defun mm-iso-8859-x-to-15-region (&optional b e)
  (if (fboundp 'char-charset)
      (let (charset item c inconvertible)
	(save-restriction
	  (if e (narrow-to-region b e))
	  (goto-char (point-min))
	  (skip-chars-forward "\0-\177")
	  (while (not (eobp))
	    (cond
	     ((not (setq item (assq (char-charset (setq c (char-after)))
				    mm-iso-8859-x-to-15-table)))
	      (forward-char))
	     ((memq c (cdr (cdr item)))
	      (setq inconvertible t)
	      (forward-char))
	     (t
	      (insert-before-markers (prog1 (+ c (car (cdr item)))
				       (delete-char 1)))))
	    (skip-chars-forward "\0-\177")))
	(not inconvertible))))

One example of code I can't understand

[Richard Stallman]

Here's more code from mm-util.el that I don't understand.

    ;; We are in a unibyte buffer or XEmacs non-mule, so we futz around a bit.
    (save-excursion
      (save-restriction
	(narrow-to-region b e)
	(goto-char (point-min))
	(skip-chars-forward "\0-\177")
	(if (eobp)
	    '(ascii)
	  (let (charset)
	    (setq charset
		  (and (boundp 'current-language-environment)
		       (car (last (assq 'charset
					(assoc current-language-environment
					       language-info-alist))))))
	    (if (eq charset 'ascii) (setq charset nil))
	    (or charset
		(setq charset
		      (car (last (assq mail-parse-charset
				       mm-mime-mule-charset-alist)))))
	    (list 'ascii (or charset 'latin-iso8859-1)))))))))

One example of code I can't understand

[Richard Stallman]

Here's some code from mm-util.el that I don't understand.
Well, I can understand the first 6 lines, but after that
I am stumped.  The doc string gives no details of what the
value should look like or what it means.


(defvar mm-iso-8859-x-to-15-table
  (and (fboundp 'coding-system-p)
       (mm-coding-system-p 'iso-8859-15)
       (mapcar
	(lambda (cs)
	  (if (mm-coding-system-p (car cs))
	      (let ((c (string-to-char
			(decode-coding-string "\341" (car cs)))))
		(cons (char-charset c)
		      (cons
		       (- (string-to-char
			   (decode-coding-string "\341" 'iso-8859-15)) c)
		       (string-to-list (decode-coding-string (car (cdr cs))
							     (car cs))))))
	    '(gnus-charset 0)))
	mm-iso-8859-15-compatible))
  "A table of the difference character between ISO-8859-X and ISO-8859-15.")

Re: One example of code I can't understand

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 19 Jul 2009 19:21:29 -0400
> 
> Here's more code from mm-util.el that I don't understand.
> 
>     ;; We are in a unibyte buffer or XEmacs non-mule, so we futz around a bit.

Why do we need to cater to unibyte operation in Emacs 23?

Re: One example of code I can't understand

[Stefan Monnier]

> Here's some code from mm-util.el that I don't understand.
> Well, I can understand the first 6 lines, but after that
> I am stumped.  The doc string gives no details of what the
> value should look like or what it means.


> (defvar mm-iso-8859-x-to-15-table
>   (and (fboundp 'coding-system-p)
>        (mm-coding-system-p 'iso-8859-15)
>        (mapcar
> 	(lambda (cs)
> 	  (if (mm-coding-system-p (car cs))
> 	      (let ((c (string-to-char
> 			(decode-coding-string "\341" (car cs)))))
> 		(cons (char-charset c)
> 		      (cons
> 		       (- (string-to-char
> 			   (decode-coding-string "\341" 'iso-8859-15)) c)
> 		       (string-to-list (decode-coding-string (car (cdr cs))
> 							     (car cs))))))
> 	    '(gnus-charset 0)))
> 	mm-iso-8859-15-compatible))
>   "A table of the difference character between ISO-8859-X and ISO-8859-15.")

Entries in this list have the form (CHARSET OFFSET CHARS...)
and it means that characters in CHARSET (except for those in CHARS) can
be converted to iso-8859-15 by adding OFFSET.

In Emacs-23 it doesn't make much sense (because unification, OFFSET is
always 0).  It's used in mm-find-mime-charset-region (via
mm-iso-8859-x-to-15-region) to provide a "poor man's unification":

    (if (and (> (length charsets) 1)
	     (memq 'iso-8859-15 charsets)
	     (memq 'iso-8859-15 hack-charsets)
	     (save-excursion (mm-iso-8859-x-to-15-region b e)))
	(dolist (x mm-iso-8859-15-compatible)
	  (setq charsets (delq (car x) charsets))))

i.e. if we need more than 1 coding-system to encode the region and
iso-8859-15 is among them, then use the above table to turn some of the
other chars into iso-8859-15 in the hope to reduce the number of
coding-systems to use (and hence the number of chunk into which the
text needs to be split).

Now that we have utf-8, this is unnecessary since we can always encode
the whole text with just a single coding-system, without having to break
it down into chunks.


        Stefan

Re: One example of code I can't understand

[Richard Stallman]

    Why do we need to cater to unibyte operation in Emacs 23?

1. Unibyte buffers may be needed for certain internal operations.  I
am not sure -- we need to ask Handa.

2. As for editing of unibyte buffers by users, maybe nobody will mind
if we get rid of that now.  But we should not eliminate it without
first polling the users, to see who uses it and whether it is important.

Re: One example of code I can't understand

[Reiner Steib]

On Mon, Jul 20 2009, Stefan Monnier wrote:

> In Emacs-23 it doesn't make much sense (because unification, OFFSET is
> always 0).  It's used in mm-find-mime-charset-region (via
> mm-iso-8859-x-to-15-region) to provide a "poor man's unification":
[...]
> Now that we have utf-8, this is unnecessary since we can always encode
> the whole text with just a single coding-system, without having to break
> it down into chunks.

Please keep in mind that Gnus wants to support older Emacs versions
and XEmacs, see (info "(gnus)Emacsen").  So please don't remove this
code.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: One example of code I can't understand

[Kenichi Handa]

In article <E1MSy7Q-0005Mn-7H@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     Why do we need to cater to unibyte operation in Emacs 23?
> 1. Unibyte buffers may be needed for certain internal operations.  I
> am not sure -- we need to ask Handa.

Currently, we are using unibyte buffers for various internal
operations (e.g rmail, tar-mode, jka-compr, ...).

It is theoretically possible to modify all of them to use a
multibyte buffer that contains only ASCII and eight-bit
chars.  But, as it may make the operations slow, I don't see
a merit in doing that.

Is that what you want to know?

> 2. As for editing of unibyte buffers by users, maybe nobody will mind
> if we get rid of that now.  But we should not eliminate it without
> first polling the users, to see who uses it and whether it is important.

I agree.

---
Kenichi Handa
handa@m17n.org

Re: One example of code I can't understand

[Stefan Monnier]

>     Why do we need to cater to unibyte operation in Emacs 23?
> 1. Unibyte buffers may be needed for certain internal operations.  I
> am not sure -- we need to ask Handa.

Maybe Gnus maintainers would know better (it seems more related to Gnus
than to Mule).

> 2. As for editing of unibyte buffers by users, maybe nobody will mind
> if we get rid of that now.  But we should not eliminate it without
> first polling the users, to see who uses it and whether it is important.

Editing unibyte buffers is an important feature, which we use whenever
we edit binary files.  I see no need/reason to get rid of it.

In this area, there is something that we should eliminate, it's the
notion of a unibyte-session (which means something like forcing
default-enable-multibyte-characters to t, tho it's a bit trickier than
that since default-enable-multibyte-characters is sometimes let-bound
around buffer-creation in place of calling set-buffer-multibyte
explicitly after the creation).


        Stefan

Re: One example of code I can't understand

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 21 Jul 2009 09:51:08 +0900
> 
> In article <E1MSy7Q-0005Mn-7H@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
> 
> >     Why do we need to cater to unibyte operation in Emacs 23?
> > 1. Unibyte buffers may be needed for certain internal operations.  I
> > am not sure -- we need to ask Handa.
> 
> Currently, we are using unibyte buffers for various internal
> operations (e.g rmail, tar-mode, jka-compr, ...).

The code in question decides how to encode portions of a mail buffer
for sending an email message.  I doubt that the above uses of unibyte
buffers can be ever exposed to this code, since for that they would
need to be parts of an email message being composed.

Re: One example of code I can't understand

[Kenichi Handa]

In article <83ocreohrv.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > >     Why do we need to cater to unibyte operation in Emacs 23?
> > > 1. Unibyte buffers may be needed for certain internal operations.  I
> > > am not sure -- we need to ask Handa.
> > 
> > Currently, we are using unibyte buffers for various internal
> > operations (e.g rmail, tar-mode, jka-compr, ...).

> The code in question decides how to encode portions of a mail buffer
> for sending an email message.

Then, the mail buffer itself may be unibyte if a user starts
Emacs with --unibyte.

But, that answer may be out of focus because I'm now writing
without reading "the code in question".  I'm sorry in that
case.  I don't have a time to read all the mails of this
thread at the moment.

---
Kenichi Handa
handa@m17n.org

Re: One example of code I can't understand

[Richard Stallman]

    In this area, there is something that we should eliminate, it's the
    notion of a unibyte-session (which means something like forcing
    default-enable-multibyte-characters to t, tho it's a bit trickier than
    that since default-enable-multibyte-characters is sometimes let-bound
    around buffer-creation in place of calling set-buffer-multibyte
    explicitly after the creation).

That change might be ok, but we should poll the users first.

Re: One example of code I can't understand

[Richard Stallman]

    Currently, we are using unibyte buffers for various internal
    operations (e.g rmail, tar-mode, jka-compr, ...).

    It is theoretically possible to modify all of them to use a
    multibyte buffer that contains only ASCII and eight-bit
    chars.  But, as it may make the operations slow, I don't see
    a merit in doing that.

    Is that what you want to know?

Yes.  It means we certainly don't want to eliminate unibyte buffers.

Re: One example of code I can't understand

[Richard Stallman]

    Please keep in mind that Gnus wants to support older Emacs versions
    and XEmacs, see (info "(gnus)Emacsen").  So please don't remove this
    code.

I'm going to remove this code in the version of the code that I
install elsewhere in Emacs.  But the functions in my version will have
different names, so the change won't affect Gnus.

Re: One example of code I can't understand

[Richard Stallman]

    In Emacs-23 it doesn't make much sense (because unification, OFFSET is
    always 0).  It's used in mm-find-mime-charset-region (via
    mm-iso-8859-x-to-15-region) to provide a "poor man's unification":

Does that mean I can get rid of the code that uses this?
That is a nice simplification.

Re: One example of code I can't understand

[Reiner Steib]

On Tue, Jul 21 2009, Richard Stallman wrote:

>     Please keep in mind that Gnus wants to support older Emacs versions
>     and XEmacs, see (info "(gnus)Emacsen").  So please don't remove this
>     code.
>
> I'm going to remove this code in the version of the code that I
> install elsewhere in Emacs.  But the functions in my version will have
> different names, so the change won't affect Gnus.

Sounds like code duplication to me.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: One example of code I can't understand

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 21 Jul 2009 10:42:01 -0400
> Cc: emacs-devel@gnu.org
> 
>     In Emacs-23 it doesn't make much sense (because unification, OFFSET is
>     always 0).  It's used in mm-find-mime-charset-region (via
>     mm-iso-8859-x-to-15-region) to provide a "poor man's unification":
> 
> Does that mean I can get rid of the code that uses this?

Yes, I think so.

Re: One example of code I can't understand

[Stephen J. Turnbull]

Kenichi Handa writes:

 > It is theoretically possible to modify all of them to use a
 > multibyte buffer that contains only ASCII and eight-bit
 > chars.  But, as it may make the operations slow, I don't see
 > a merit in doing that.

XEmacs has always done it this way; it is more than a theoretical
possibility.  In XEmacs it doesn't make much difference for the
operations you describe (less than a factor of 2) because the overhead
of interfacing to the pipes is greater than the overhead of
conversion.  This is based on some profiling Ben did many years ago;
ISTR it was basically linear up to 128MB or maybe 256MB.  I don't
recall the exact numbers, but I do remember being very surprised that
it was substantially less than 2.

In XEmacs, I think it makes a much bigger difference for mail buffers
(VM, or Gnus nnfolder) because of the frequency of byte<->char
conversions in those more or less random access applications.

Re: One example of code I can't understand

[Richard Stallman]

    > I'm going to remove this code in the version of the code that I
    > install elsewhere in Emacs.  But the functions in my version will have
    > different names, so the change won't affect Gnus.

    Sounds like code duplication to me.

Yes, for the moment.  Eventually we can delete the corresponding code
from mm-util.el and replace it with aliases pointing at these
functions.  You could keep the old code to include in a compatibility
pack when distributing Gnus separately.