bug#42366: UTF8 QR codes vs. emacs

bug#42366: UTF8 QR codes vs. emacs

[積丹尼 Dan Jacobson]

Compare how
$ qrencode -t UTF8 x
looks in
$ emacs -nw -f shell # vs.
$ emacs     -f shell

As you can see in the latter it is quite mangled,
perhaps beyond machine recognition.

bug#42366: UTF8 QR codes vs. emacs

[Robert Pluim]

>>>>> On Wed, 15 Jul 2020 22:20:45 +0800, 積丹尼 Dan Jacobson <jidanni@jidanni.org> said:

    積丹尼> Compare how
    積丹尼> $ qrencode -t UTF8 x
    積丹尼> looks in
    積丹尼> $ emacs -nw -f shell # vs.
    積丹尼> $ emacs     -f shell

    積丹尼> As you can see in the latter it is quite mangled,
    積丹尼> perhaps beyond machine recognition.

Using which font on which platform?

Robert

bug#42366: UTF8 QR codes vs. emacs

[積丹尼 Dan Jacobson]

>>>>> "RP" == Robert Pluim <rpluim@gmail.com> writes:

RP> Using which font on which platform?

Mmmm... the majority.
How about yours. Can you take a screenshot of it, scan it with e.g.,
zbarimg, and have it still be decoded successfully?

bug#42366: UTF8 QR codes vs. emacs

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

積丹尼 Dan Jacobson <jidanni@jidanni.org> writes:

> Compare how
> $ qrencode -t UTF8 x
> looks in
> $ emacs -nw -f shell # vs.
> $ emacs     -f shell
>
> As you can see in the latter it is quite mangled,
> perhaps beyond machine recognition.

It's not good in any case, really.  Here's the X version:


[-- Attachment #2: Type: image/png, Size: 22874 bytes --]

[-- Attachment #3: Type: text/plain, Size: 106 bytes --]


Note the horizontal lines -- it should be a solid white field.

And with -nw it's just a complete mess:


[-- Attachment #4: Type: image/png, Size: 80744 bytes --]

[-- Attachment #5: Type: text/plain, Size: 405 bytes --]


I guess...  it's translating all the Unicode characters instead of just
letting the terminal render them?  The terminal does have these
characters; when running the program directly in the shell everything
works fine.

Does anybody know how to make shell-mode (under -nw) just output the
characters?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42366: UTF8 QR codes vs. emacs

[Andreas Schwab]

On Aug 05 2020, Lars Ingebrigtsen wrote:

> Note the horizontal lines -- it should be a solid white field.

That's just a matter of using a proper font.

> Does anybody know how to make shell-mode (under -nw) just output the
> characters?

Worksforme.  Try using a UTF-8 environment.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#42366: UTF8 QR codes vs. emacs

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Aug 05 2020, Lars Ingebrigtsen wrote:
>
>> Note the horizontal lines -- it should be a solid white field.
>
> That's just a matter of using a proper font.

Emacs doesn't select the proper font automatically here, which it should?

>> Does anybody know how to make shell-mode (under -nw) just output the
>> characters?
>
> Worksforme.  Try using a UTF-8 environment.

Oh, yeah.  With emacs -Q this works fine for me under Emacs -nw.  So I
don't think there's any bugs here, and I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 05 Aug 2020 12:26:55 +0200
> Cc: 42366@debbugs.gnu.org
> 
> > Compare how
> > $ qrencode -t UTF8 x
> > looks in
> > $ emacs -nw -f shell # vs.
> > $ emacs     -f shell
> >
> > As you can see in the latter it is quite mangled,
> > perhaps beyond machine recognition.
> 
> It's not good in any case, really.  Here's the X version:

Can you send the file itself?

> Note the horizontal lines -- it should be a solid white field.

How do you know that?

> And with -nw it's just a complete mess:

No, it isn't...

> I guess...  it's translating all the Unicode characters instead of just
> letting the terminal render them?  The terminal does have these
> characters; when running the program directly in the shell everything
> works fine.

That's what happens when Emacs thinks your terminal cannot display
these characters.  What does 'terminal-coding-system' return on that
display?

> Does anybody know how to make shell-mode (under -nw) just output the
> characters?

Hmm... shell-mode?  Now I'm beginning to think I don't really
understand what are the steps to reproduce the problem, and what is
their meaning (jidanni needs to learn how to report bugs clearly).
Would you please describe the steps, starting from "emacs -Q"?

bug#42366: UTF8 QR codes vs. emacs

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Wed, 05 Aug 2020 12:26:55 +0200
>> Cc: 42366@debbugs.gnu.org
>> 
>> > Compare how
>> > $ qrencode -t UTF8 x
>> > looks in
>> > $ emacs -nw -f shell # vs.
>> > $ emacs     -f shell
>> >
>> > As you can see in the latter it is quite mangled,
>> > perhaps beyond machine recognition.
>> 
>> It's not good in any case, really.  Here's the X version:
>
> Can you send the file itself?

There's no file -- it's just the output from the qrencode command.

>> Note the horizontal lines -- it should be a solid white field.
>
> How do you know that?

Because that's how it displays in the shell:


[-- Attachment #2: Type: image/png, Size: 13584 bytes --]

[-- Attachment #3: Type: text/plain, Size: 522 bytes --]



And besides, a QR Code isn't supposed to have horizontal lines like
that.

> Hmm... shell-mode?  Now I'm beginning to think I don't really
> understand what are the steps to reproduce the problem, and what is
> their meaning (jidanni needs to learn how to report bugs clearly).
> Would you please describe the steps, starting from "emacs -Q"?

There's not much to it: Run the qrencode program:

$ qrencode -t UTF8 x

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: jidanni@jidanni.org,  42366@debbugs.gnu.org
> Date: Wed, 05 Aug 2020 16:22:33 +0200
> 
> > Can you send the file itself?
> 
> There's no file -- it's just the output from the qrencode command.

Can you redirect the output to a file?

> And besides, a QR Code isn't supposed to have horizontal lines like
> that.

As Andreas says, this depends on the font.  Some fonts leave a pixel
between this line and the next when they design these "characters".

> > Hmm... shell-mode?  Now I'm beginning to think I don't really
> > understand what are the steps to reproduce the problem, and what is
> > their meaning (jidanni needs to learn how to report bugs clearly).
> > Would you please describe the steps, starting from "emacs -Q"?
> 
> There's not much to it: Run the qrencode program:
> 
> $ qrencode -t UTF8 x

So you are saying that your problem with \uNNNN was because you used
an incorrect locale?  IOW, that there's no bug here at all?

bug#42366: UTF8 QR codes vs. emacs

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> There's no file -- it's just the output from the qrencode command.
>
> Can you redirect the output to a file?

Sure, it's just a bunch of UTF-8.  I guess I could also just include the
data here:

█████████████████████████████
█████████████████████████████
████ ▄▄▄▄▄ █▀ █ ▄█ ▄▄▄▄▄ ████
████ █   █ █▄ █▀▄█ █   █ ████
████ █▄▄▄█ █ ██▀ █ █▄▄▄█ ████
████▄▄▄▄▄▄▄█ ▀ ▀ █▄▄▄▄▄▄▄████
████  ▀▄▀▀▄ █ ▄███ ▄▄█  █████
██████ ▄ ▄▄▀▄█▄▀▄▄███▀▀  ████
████▄▄▄▄▄▄▄█  ▀▀▄█▄▀ ▀▄▀▀████
████ ▄▄▄▄▄ █▄▄█▀▀ █ ▀▀▄▄█████
████ █   █ █▀▄▄ ██▄▄▄    ████
████ █▄▄▄█ █▀█ █▄▀▀█ █▀█▄████
████▄▄▄▄▄▄▄█▄█▄▄▄███▄▄▄▄█████
█████████████████████████████
█████████████████████████████

I've also included as an attachment, in case that doesn't work, for some
reason.

> So you are saying that your problem with \uNNNN was because you used
> an incorrect locale?  IOW, that there's no bug here at all?

Yes.  Except for the horizontal lines in the X version of Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Attachment #2: qr.bin --]
[-- Type: text/plain, Size: 1219 bytes --]

█████████████████████████████
█████████████████████████████
████ ▄▄▄▄▄ █▀ █ ▄█ ▄▄▄▄▄ ████
████ █   █ █▄ █▀▄█ █   █ ████
████ █▄▄▄█ █ ██▀ █ █▄▄▄█ ████
████▄▄▄▄▄▄▄█ ▀ ▀ █▄▄▄▄▄▄▄████
████  ▀▄▀▀▄ █ ▄███ ▄▄█  █████
██████ ▄ ▄▄▀▄█▄▀▄▄███▀▀  ████
████▄▄▄▄▄▄▄█  ▀▀▄█▄▀ ▀▄▀▀████
████ ▄▄▄▄▄ █▄▄█▀▀ █ ▀▀▄▄█████
████ █   █ █▀▄▄ ██▄▄▄    ████
████ █▄▄▄█ █▀█ █▄▀▀█ █▀█▄████
████▄▄▄▄▄▄▄█▄█▄▄▄███▄▄▄▄█████
█████████████████████████████
█████████████████████████████

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: jidanni@jidanni.org,  42366@debbugs.gnu.org
> Date: Wed, 05 Aug 2020 16:54:32 +0200
> 
> > Can you redirect the output to a file?
> 
> Sure, it's just a bunch of UTF-8.  I guess I could also just include the
> data here:

Thanks.  FWIW, with the default font my Emacs uses, I see no lines.

> > So you are saying that your problem with \uNNNN was because you used
> > an incorrect locale?  IOW, that there's no bug here at all?
> 
> Yes.  Except for the horizontal lines in the X version of Emacs.

Can you try other fonts, and see if some of them eliminate the
problem?  I think this is simply a matter of the shell window and the
terminal emulator using different fonts.

bug#42366: UTF8 QR codes vs. emacs

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

>>>>> On Wed, 05 Aug 2020 18:03:22 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> Yes.  Except for the horizontal lines in the X version of Emacs.

    Eli> Can you try other fonts, and see if some of them eliminate the
    Eli> problem?  I think this is simply a matter of the shell window and the
    Eli> terminal emulator using different fonts.

Iʼm not sure about that. On macOS, this is

emacs -Q -font Consolas


[-- Attachment #2: emacs-consolas.png --]
[-- Type: image/png, Size: 3976 bytes --]

[-- Attachment #3: Type: text/plain, Size: 70 bytes --]


And this is the same build, but in a terminal window using Consolas


[-- Attachment #4: emacs-nw-consolas.png --]
[-- Type: image/png, Size: 2919 bytes --]

[-- Attachment #5: Type: text/plain, Size: 298 bytes --]


In the gui version, the cursor looks like it extends 1 pixel below
each component character, whilst in the '-nw' on the bottom of the
cursor lines up exactly.

If I use something else like Menlo, there are horizontal white lines,
but the cursor lines up with the bottom of the characters.

Robert

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  42366@debbugs.gnu.org,
>   jidanni@jidanni.org
> Date: Wed, 05 Aug 2020 17:33:10 +0200
> 
> 
> [1:text/plain Hide]
> 
> And this is the same build, but in a terminal window using Consolas

What exactly is surprising here?  In a TTY frame, Emacs doesn't
control the layout: the terminal emulator does.  Emacs just sends
codes to the terminal device, and that's it.

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> Date: Wed, 05 Aug 2020 18:38:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 42366@debbugs.gnu.org, jidanni@jidanni.org
> 
> What exactly is surprising here?  In a TTY frame, Emacs doesn't
> control the layout: the terminal emulator does.  Emacs just sends
> codes to the terminal device, and that's it.

To clarify: the original complaint, AFAIU, was about the display in a
GUI frame, in which case the layout is entirely ours.  You are now
saying that the problem is on TTY frames, but there we don't control
the layout at all.

bug#42366: UTF8 QR codes vs. emacs

[Andreas Schwab]

On Aug 05 2020, Robert Pluim wrote:

> Iʼm not sure about that. On macOS, this is
>
> emacs -Q -font Consolas
>
>
>
>
> And this is the same build, but in a terminal window using Consolas

And as you can see, each program has its own warts.

FWIW, with Liberation Mono, I see perfect lineup.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#42366: UTF8 QR codes vs. emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> > So you are saying that your problem with \uNNNN was because you used
>> > an incorrect locale?  IOW, that there's no bug here at all?
>> 
>> Yes.  Except for the horizontal lines in the X version of Emacs.
>
> Can you try other fonts, and see if some of them eliminate the
> problem?  I think this is simply a matter of the shell window and the
> terminal emulator using different fonts.

Yeah, it's probably just a font issue, so I don't think it's worth
investigating any further.  QR codes in Emacs is a rather obscure use
case.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42366: UTF8 QR codes vs. emacs

[Robert Pluim]

>>>>> On Wed, 05 Aug 2020 18:42:11 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> Date: Wed, 05 Aug 2020 18:38:38 +0300
    >> From: Eli Zaretskii <eliz@gnu.org>
    >> Cc: larsi@gnus.org, 42366@debbugs.gnu.org, jidanni@jidanni.org
    >> 
    >> What exactly is surprising here?  In a TTY frame, Emacs doesn't
    >> control the layout: the terminal emulator does.  Emacs just sends
    >> codes to the terminal device, and that's it.

    Eli> To clarify: the original complaint, AFAIU, was about the display in a
    Eli> GUI frame, in which case the layout is entirely ours.  You are now
    Eli> saying that the problem is on TTY frames, but there we don't control
    Eli> the layout at all.

I donʼt think I said that. The TTY frame layout looks ok, the gui
frame one has what looks like extra pixels between the characters.

Robert

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  42366@debbugs.gnu.org,  jidanni@jidanni.org
> Date: Thu, 06 Aug 2020 09:57:04 +0200
> 
>     Eli> To clarify: the original complaint, AFAIU, was about the display in a
>     Eli> GUI frame, in which case the layout is entirely ours.  You are now
>     Eli> saying that the problem is on TTY frames, but there we don't control
>     Eli> the layout at all.
> 
> I donʼt think I said that. The TTY frame layout looks ok, the gui
> frame one has what looks like extra pixels between the characters.

Then I don't think I understand what you are saying when I look at the
screenshots you posted.  When you say "between the characters", do you
mean between two successive lines (i.e., vertically), or do you mean
between two adjacent characters on the same line?

bug#42366: UTF8 QR codes vs. emacs

[Robert Pluim]

>>>>> On Thu, 06 Aug 2020 16:44:59 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: larsi@gnus.org,  42366@debbugs.gnu.org,  jidanni@jidanni.org
    >> Date: Thu, 06 Aug 2020 09:57:04 +0200
    >> 
    Eli> To clarify: the original complaint, AFAIU, was about the display in a
    Eli> GUI frame, in which case the layout is entirely ours.  You are now
    Eli> saying that the problem is on TTY frames, but there we don't control
    Eli> the layout at all.
    >> 
    >> I donʼt think I said that. The TTY frame layout looks ok, the gui
    >> frame one has what looks like extra pixels between the characters.

    Eli> Then I don't think I understand what you are saying when I look at the
    Eli> screenshots you posted.  When you say "between the characters", do you
    Eli> mean between two successive lines (i.e., vertically), or do you mean
    Eli> between two adjacent characters on the same line?

I meant between two adjacent characters (did I mix up the screenshots?
entirely possible).

With Liberation Mono, I see thin horizontal white lines (ie between
successive lines)

I guess we could put this all down to font differences.

Robert

bug#42366: UTF8 QR codes vs. emacs

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  42366@debbugs.gnu.org,  jidanni@jidanni.org
> Date: Thu, 06 Aug 2020 16:35:53 +0200
> 
> I meant between two adjacent characters (did I mix up the screenshots?
> entirely possible).
> 
> With Liberation Mono, I see thin horizontal white lines (ie between
> successive lines)

Ah, yes: I think you mixed up the images.