bug#2416: 23.0.60; decode-coding-region

bug#2416: 23.0.60; decode-coding-region

[Chong Yidong]

Could you try to narrow down the bug, so that it can be reproduced?  One
way would be to copy out buffer contents before decoding into a separate
buffer, and see if decode-coding-region misbehaves when applied manually
to that bufer.

bug#2416: 23.0.60; decode-coding-region

[mj]

I have been having this problem since I switched to Emacs 23 several
weeks ago. I'm using VM to read my mails. There seems to be a problem
in decode-coding-region when VM tries to decode a string. When VM
tries to decode a region or a string, it uses a temporary buffer and
basically runs the following lisp code:

(apply 'decode-coding-region (point-min) (point-max)  'us-ascii nil)

The original buffer  content would be something like this:

B7040400-12
some text here

after decode-coding-region is executed, the buffer content became:

^@7040450-12
some text here

Where ^@ is actually binary code \0 (not ascii ^ and @). There is another instance
that a string was decoded and the result is  ^@ prefixed. 

I could not reproduce this with "Emacs -Q". But it always happens when
thsoe particular messages were processed by VM. 

Strangely enough, if I inserted a few spaces at the beginning of
buffer: (one space in the following buffer)

 B7040400-12
some text here

And, the decoding was done correctly. In another instance mentioned
above, one space is not enough. I had to put several spaces to get the
decoding working. 

I saw another bug report just yesterday regarding decode-coding-region
crashing. I applied the patch, but it did not help in the
decoding. 

Please let me know if you need other information to help understand
the problem. Thanks. 

-----
Emacs version: "GNU Emacs 23.0.60.1 (i386-mingw-nt5.1.2600) of 2009-01-29 on T42"

Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4) --cflags -I../../GnuWin32/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: chinese-big5
  default-enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
  auto-image-file-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

bug#2416: 23.0.60; decode-coding-region

[Eli Zaretskii]

> Date: Fri, 20 Feb 2009 13:13:01 -0800 (PST)
> From: mj <mj54590@gmail.com>
> Cc: 
> 
> I have been having this problem since I switched to Emacs 23 several
> weeks ago. I'm using VM to read my mails. There seems to be a problem
> in decode-coding-region when VM tries to decode a string. When VM
> tries to decode a region or a string, it uses a temporary buffer and
> basically runs the following lisp code:
> 
> (apply 'decode-coding-region (point-min) (point-max)  'us-ascii nil)
> 
> The original buffer  content would be something like this:
> 
> B7040400-12
> some text here
> 
> after decode-coding-region is executed, the buffer content became:
> 
> ^@7040450-12
> some text here
> 
> Where ^@ is actually binary code \0 (not ascii ^ and @). There is another instance
> that a string was decoded and the result is  ^@ prefixed. 
> 
> I could not reproduce this with "Emacs -Q". But it always happens when
> thsoe particular messages were processed by VM. 

Could you please see if the problem still persists in the current CVS?
Your Emacs seems to be about a month old (Jan 29), and a couple of
related bugs were fixed in coding.c since then.

bug#2416: 23.0.60; decode-coding-region

[MJ]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

Thanks for the reply. I just tried with the latest CVS version and the
problem still persists:

"GNU Emacs 23.0.90.1 (i386-mingw-nt5.1.2600) of 2009-02-21 on T42"

On Sat, Feb 21, 2009 at 4:16 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Fri, 20 Feb 2009 13:13:01 -0800 (PST)
> > From: mj <mj54590@gmail.com>
> > Cc:
> >
> > I have been having this problem since I switched to Emacs 23 several
> > weeks ago. I'm using VM to read my mails. There seems to be a problem
> > in decode-coding-region when VM tries to decode a string. When VM
> > tries to decode a region or a string, it uses a temporary buffer and
> > basically runs the following lisp code:
> >
> > (apply 'decode-coding-region (point-min) (point-max)  'us-ascii nil)
> >
> > The original buffer  content would be something like this:
> >
> > B7040400-12
> > some text here
> >
> > after decode-coding-region is executed, the buffer content became:
> >
> > ^@7040450-12
> > some text here
> >
> > Where ^@ is actually binary code \0 (not ascii ^ and @). There is another
> instance
> > that a string was decoded and the result is  ^@ prefixed.
> >
> > I could not reproduce this with "Emacs -Q". But it always happens when
> > thsoe particular messages were processed by VM.
>
> Could you please see if the problem still persists in the current CVS?
> Your Emacs seems to be about a month old (Jan 29), and a couple of
> related bugs were fixed in coding.c since then.
>

[-- Attachment #2: Type: text/html, Size: 1962 bytes --]

bug#2416: 23.0.60; decode-coding-region

[Juanma Barranquero]

On Fri, Feb 20, 2009 at 22:13, mj <mj54590@gmail.com> wrote:

> (apply 'decode-coding-region (point-min) (point-max)  'us-ascii nil)
>
> The original buffer  content would be something like this:
>
> B7040400-12
> some text here
>
> after decode-coding-region is executed, the buffer content became:
>
> ^@7040450-12
> some text here
>
> Where ^@ is actually binary code \0 (not ascii ^ and @). There is another instance
> that a string was decoded and the result is  ^@ prefixed.

Could it be related to bug#1809?

(with-temp-buffer
  (insert (make-string 20 ?.))
  (decode-coding-region 1 (point-max) 'raw-text)
  (buffer-string))

=> "^@..................."

    Juanma

bug#2416: 23.0.60; decode-coding-region

[MJ]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

Juanma, thank you for the lisp code that reproduces the same problem that I
am having.

If a space is inserted at the beginning of the buffer, then the result is
correct (as stated in my bug report):

(with-temp-buffer
 (insert " ")
 (insert (make-string 20 ?.))
 (decode-coding-region 2 (point-max) 'us-ascii)
 (buffer-string))
" ...................."

(I use 'us-ascii just to show the coding does not matter).

Now, hopefully emacs developers will  be able to understand and fix the
problem.


On Sat, Feb 21, 2009 at 9:47 PM, Juanma Barranquero <lekktu@gmail.com>wrote:

> On Fri, Feb 20, 2009 at 22:13, mj <mj54590@gmail.com> wrote:
>
> > (apply 'decode-coding-region (point-min) (point-max)  'us-ascii nil)
> >
> > The original buffer  content would be something like this:
> >
> > B7040400-12
> > some text here
> >
> > after decode-coding-region is executed, the buffer content became:
> >
> > ^@7040450-12
> > some text here
> >
> > Where ^@ is actually binary code \0 (not ascii ^ and @). There is another
> instance
> > that a string was decoded and the result is  ^@ prefixed.
>
> Could it be related to bug#1809?
>
> (with-temp-buffer
>  (insert (make-string 20 ?.))
>  (decode-coding-region 1 (point-max) 'raw-text)
>  (buffer-string))
>
> => "^@..................."
>
>    Juanma
>

[-- Attachment #2: Type: text/html, Size: 1951 bytes --]

bug#2416: 23.0.60; decode-coding-region

[Juanma Barranquero]

On Sun, Feb 22, 2009 at 06:07, MJ <mj54590@gmail.com> wrote:

> If a space is inserted at the beginning of the buffer, then the result is
> correct (as stated in my bug report):

If you look at #1809, you'll see that the problem seems to be
sensitive to string length. Assuming you're using ASCII, do you see it
with strings of length != 20?

    Juanma

bug#2416: 23.0.60; decode-coding-region

[MJ]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

Indeed, including newline, the buffer length is exactly 20 bytes in two
cases where they are ascii. In another case, it's BIG5 Chinese (each BIG5
code is two bytes), and the total is also 20 bytes. Thanks for pointing it
out.

On Sun, Feb 22, 2009 at 12:13 AM, Juanma Barranquero <lekktu@gmail.com>wrote:

> On Sun, Feb 22, 2009 at 06:07, MJ <mj54590@gmail.com> wrote:
>
> > If a space is inserted at the beginning of the buffer, then the result is
> > correct (as stated in my bug report):
>
> If you look at #1809, you'll see that the problem seems to be
> sensitive to string length. Assuming you're using ASCII, do you see it
> with strings of length != 20?
>
>    Juanma
>

[-- Attachment #2: Type: text/html, Size: 1098 bytes --]

bug#2416: 23.0.60; decode-coding-region

[Juanma Barranquero]

merge 1809 2416
quit

> Indeed, including newline, the buffer length is exactly 20 bytes in two
> cases where they are ascii. In another case, it's BIG5 Chinese (each BIG5
> code is two bytes), and the total is also 20 bytes. Thanks for pointing it
> out.

OK, so this is #1809 then. I'm merging the bugs.

  Juanma

bug#2416: 23.0.60; decode-coding-region

[Juanma Barranquero]

On Sun, Feb 22, 2009 at 06:13, Juanma Barranquero <lekktu@gmail.com> wrote:

> If you look at #1809, you'll see that the problem seems to be
> sensitive to string length. Assuming you're using ASCII, do you see it
> with strings of length != 20?

A clue: the bug is sensitive to the value of the BUF_GAP_SIZE set in
`get-buffer-create', at buffer.c:364.

If you change

  BUF_GAP_SIZE (b) = 20;

to another size, so does the bug's triggering length.

    Juanma

bug#2416: 23.0.60; decode-coding-region

[Andreas Schwab]

Juanma Barranquero <lekktu@gmail.com> writes:

> Could it be related to bug#1809?
>
> (with-temp-buffer
>   (insert (make-string 20 ?.))
>   (decode-coding-region 1 (point-max) 'raw-text)
>   (buffer-string))
>
> => "^@..................."

I've installed a fix.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2416: 23.0.60; decode-coding-region

[MJ]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Andreas, thank you for the quick fix, which indeed solves the problem I was
having. -- mj

On Sun, Feb 22, 2009 at 9:31 AM, Andreas Schwab <schwab@suse.de> wrote:

> Juanma Barranquero <lekktu@gmail.com> writes:
>
> > Could it be related to bug#1809?
> >
> > (with-temp-buffer
> >   (insert (make-string 20 ?.))
> >   (decode-coding-region 1 (point-max) 'raw-text)
> >   (buffer-string))
> >
> > => "^@..................."
>
> I've installed a fix.
>
> Andreas.
>
> --
> Andreas Schwab, SuSE Labs, schwab@suse.de
> SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
> PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>

[-- Attachment #2: Type: text/html, Size: 1235 bytes --]