Unibyte strings in Lisp data structures

Unibyte strings in Lisp data structures

[Eli Zaretskii]

Take a look at jka-compr-compression-info-list: each compression
method has a magic signature there, which is the 9th element of the
vector describing that compression method.

Now evaluate this:

  (multibyte-string-p (aref (car jka-compr-compression-info-list) 9))
     => nil

These magic signatures are unibyte strings.  But why are they unibyte?
What code decides that they should be unibyte, when Emacs reads
jka-cmpr-hook.el?  Can we rely on the fact that these strings will
always be unibyte?

I bumped into this while debugging a problem in rmailmm.el: saving
attachments whose file names end in .gz produces a file that is
gzip-compressed twice.  I finally traced this to this fragment in
jka-compr.el:

	     ;; If the contents to be written out
	     ;; are properly compressed already,
	     ;; don't try to compress them over again.
	     (not (and magic
		       (equal (if (stringp start)
				  (substring start 0 (min (length start)
							  (length magic)))
                                (let* ((from (or start (point-min)))
                                       (to (min (or end (point-max))
                                                (+ from (length magic)))))
                                  (buffer-substring from to)))
			      magic))))

This test failed, because `magic' is a unibyte string, while
buffer-substring was returning a multibyte string.

The fix seems to be easy: modify rmail-mime-save to make the temporary
buffer it uses be a unibyte buffer.  But then I started to wonder how
come `magic' is a unibyte string, and can I rely on that?

There is, of course, the alternative to convert both strings to
unibyte and compare that.  Still, I think it would be good to know how
come these strings are unibyte to begin with.

Re: Unibyte strings in Lisp data structures

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> What code decides that they should be unibyte, when Emacs reads
> jka-cmpr-hook.el?

Strings are read as unibyte by default unless they contain non-ascii,
non-8-bit characters. (See (elisp) Converting Representations::).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Unibyte strings in Lisp data structures

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Kenichi Handa <handa@m17n.org>,  emacs-devel@gnu.org
> Date: Tue, 13 Jul 2010 17:05:30 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What code decides that they should be unibyte, when Emacs reads
> > jka-cmpr-hook.el?
> 
> Strings are read as unibyte by default unless they contain non-ascii,
> non-8-bit characters. (See (elisp) Converting Representations::).

Thanks, but I'm not sure this is relevant.  The section you pointed to
deals with conversions and insertions, not with how strings are read
by the Lisp reader.

Note that in jka-cmpr-hook.el, these magic signatures are specified as
octal escapes:

    ["\\.g?z\\(~\\|\\.~[0-9]+~\\)?\\'"
     "compressing"        "gzip"         ("-c" "-q")
     "uncompressing"      "gzip"         ("-c" "-q" "-d")
     t t "\037\213"]

I think the relevant code is this fragment from lread.c:read_escape:

    case '0':
    case '1':
    case '2':
    case '3':
    case '4':
    case '5':
    case '6':
    case '7':
      /* An octal escape, as in ANSI C.  */
      {
	register int i = c - '0';
	register int count = 0;
	while (++count < 3)
	  {
	    if ((c = READCHAR) >= '0' && c <= '7')
	      {
		i *= 8;
		i += c - '0';
	      }
	    else
	      {
		UNREAD (c);
		break;
	      }
	  }

	if (i >= 0x80 && i < 0x100)
	  i = BYTE8_TO_CHAR (i);
	return i;
      }

The BYTE8_TO_CHAR macro returns the multibyte representation of an
eight-bit byte.  Then, in read1, we do:

		if (CHAR_BYTE8_P (c))
		  force_singlebyte = 1;
		...
	else if (force_singlebyte)
	  {
	    nchars = str_as_unibyte (read_buffer, p - read_buffer);

The question is now: will this rule remain stable for time long enough
to rely on it?  Or is it safer to convert both strings to the same
representation for comparison?

Re: Unibyte strings in Lisp data structures

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> The question is now: will this rule remain stable for time long enough
> to rely on it?  Or is it safer to convert both strings to the same
> representation for comparison?

I think we should make it that (equal s (string-to-multibyte s)) is
always true.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."