bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

[Alexander Miller]

In my config I have the following function that is used to prettify my
mode-line (the actual version is pure and side-effect-free and inlined,
but that does not seem to have any effect here):

(defun f (n)
   (pcase n
     (1 " ➊") (2 " ➋") (3 " ➌") (4 " ➍") (5 " ➎") (6 " ➏")
     (7 " ➐") (8 " ➑") (9 " ➒") (10 " ➓") (_ "")))

When native compiled it appears to return raw bytes instead of the
unicode symbols, for example I am seeing \342\236\212 instead of ➊.

I can circumvent that by using (1 (decode-coding-string " ➊" 'utf-8)),
but of course that should not be necessary, since there's no such
problem with the byte-compiled version.

And on a probably unrelated note: is it normal for such a simple
function to be compiled to 100 LOC of assembly? That seems surprisingly
to my amateur eyes.

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.23, cairo version 1.17.3)
of 2020-12-20 built on am-laptop
Repository revision: ab985f41db5fdaeada513d28a065332fd8838cf4
Repository branch: makepkg
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Manjaro Linux

Configured using:
'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
--localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
--with-sound=alsa --with-modules --without-gconf --without-gsettings
--with-nativecomp --with-x-toolkit=gtk3 --without-xaw3d
--without-m17n-flt --with-cairo --without-compress-install
'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -g
-fuse-ld=gold' CPPFLAGS=-D_FORTIFY_SOURCE=2
LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3
X11 XDBE XIM MODULES NATIVE_COMP THREADS LIBSYSTEMD JSON PDUMPER LCMS2

Important settings:
value of $LC_MONETARY: de_DE.UTF-8
value of $LC_NUMERIC: de_DE.UTF-8
value of $LC_TIME: de_DE.UTF-8
value of $LANG: en_GB.utf8
locale-coding-system: utf-8

Major mode: ELisp/l

Minor modes in effect:
rainbow-delimiters-mode: t
rainbow-mode: t
display-line-numbers-mode: t
hl-todo-mode: t
org-super-agenda-mode: t
treemacs-icons-dired-mode: t
treemacs-filewatch-mode: t
treemacs-follow-mode: t
treemacs-git-mode: deferred
treemacs-fringe-indicator-mode: t
helm-mode: t
helm--remap-mouse-mode: t
async-bytecomp-package-mode: t
gcmh-mode: t
projectile-mode: t
company-prescient-mode: t
global-company-mode: t
company-mode: t
global-evil-vimish-fold-mode: t
evil-vimish-fold-mode: t
vimish-fold-mode: t
show-smartparens-global-mode: t
show-smartparens-mode: t
smartparens-global-mode: t
smartparens-mode: t
framey-mode: t
purpose-mode: t
shackle-mode: t
winum-mode: t
eyebrowse-mode: t
global-subword-mode: t
subword-mode: t
global-evil-surround-mode: t
evil-surround-mode: t
evil-lion-mode: t
evil-goggles-mode: t
shell-dirtrack-mode: t
evil-mode: t
evil-local-mode: t
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
prettify-symbols-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
line-number-mode: t
transient-mark-mode: t

Load-path shadows:
/usr/share/emacs/28.0.50/lisp/emacs-lisp/let-alist hides
/home/am/.emacs.d/straight/build/let-alist/let-alist
~/Documents/git/treemacs/src/elisp/treemacs-interface hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-interface
~/Documents/git/treemacs/src/elisp/treemacs-rendering hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-rendering
~/Documents/git/treemacs/src/elisp/treemacs-dom hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-dom
~/Documents/git/treemacs/src/elisp/treemacs-tags hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-tags
~/Documents/git/treemacs/src/elisp/treemacs hides
/home/am/.emacs.d/straight/build/treemacs/treemacs
~/Documents/git/treemacs/src/elisp/treemacs-workspaces hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-workspaces
~/Documents/git/treemacs/src/elisp/treemacs-customization hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-customization
~/Documents/git/treemacs/src/elisp/treemacs-faces hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-faces
~/Documents/git/treemacs/src/elisp/treemacs-themes hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-themes
~/Documents/git/treemacs/src/elisp/treemacs-mouse-interface hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-mouse-interface
~/Documents/git/treemacs/src/elisp/treemacs-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-mode
~/Documents/git/treemacs/src/elisp/treemacs-icons hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-icons
~/Documents/git/treemacs/src/elisp/treemacs-compatibility hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-compatibility
~/Documents/git/treemacs/src/elisp/treemacs-follow-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-follow-mode
~/Documents/git/treemacs/src/elisp/treemacs-visuals hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-visuals
~/Documents/git/treemacs/src/elisp/treemacs-core-utils hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-core-utils
~/Documents/git/treemacs/src/elisp/treemacs-extensions hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-extensions
~/Documents/git/treemacs/src/elisp/treemacs-filewatch-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-filewatch-mode
~/Documents/git/treemacs/src/elisp/treemacs-persistence hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-persistence
~/Documents/git/treemacs/src/elisp/treemacs-async hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-async
~/Documents/git/treemacs/src/elisp/treemacs-bookmarks hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-bookmarks
~/Documents/git/treemacs/src/elisp/treemacs-tag-follow-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-tag-follow-mode
~/Documents/git/treemacs/src/elisp/treemacs-logging hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-logging
~/Documents/git/treemacs/src/elisp/treemacs-header-line hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-header-line
~/Documents/git/treemacs/src/elisp/treemacs-fringe-indicator hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-fringe-indicator
~/Documents/git/treemacs/src/elisp/treemacs-diagnostics hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-diagnostics
~/Documents/git/treemacs/src/elisp/treemacs-macros hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-macros
~/Documents/git/treemacs/src/elisp/treemacs-scope hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-scope

Features:
(shadow sort ispell prose-complete mail-extr emacsbug winner face-remap
helm-command helm-elisp helm-eval edebug backtrace helm-info info vc-mtn
vc-hg vc-git diff-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc
vc-dispatcher rainbow-delimiters rainbow-mode xterm-color color
display-line-numbers hl-todo pp view mu4e-alert time alert log4e
notifications dbus xml gntp org-mu4e mu4e desktop frameset mu4e-org
smartparens-org german-holidays org-super-agenda ts org-habit
org-element avl-tree generator org-agenda org-refile org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table ol org-keys
org-compat org-macs org-loaddefs find-func mu4e-main mu4e-view cal-menu
calendar cal-loaddefs browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
url-vars mailcap mu4e-headers mu4e-compose mu4e-context mu4e-draft
mu4e-actions ido rfc2368 smtpmail sendmail mu4e-mark mu4e-message
flow-fill mu4e-proc mu4e-utils doc-view jka-compr mu4e-lists mule-util
mu4e-vars message rmc puny treemacs-icons-dired treemacs-projectile
treemacs-evil treemacs treemacs-header-line treemacs-compatibility
treemacs-mode treemacs-interface treemacs-extensions
treemacs-persistence treemacs-mouse-interface treemacs-tag-follow-mode
treemacs-filewatch-mode treemacs-tags imenu xref project
treemacs-follow-mode treemacs-rendering treemacs-async
treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope treemacs-faces treemacs-icons
treemacs-themes treemacs-core-utils pfuture ace-window avy hl-line
treemacs-logging treemacs-customization treemacs-macros dired+
image-dired image-mode exif image-file image-converter dired-x dired-aux
dired dired-loaddefs rfc822 mml mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader mu4e-meta helm-mode helm-projectile
helm-files filenotify docker-tramp tramp-cache tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat parse-time iso8601
time-date ls-lisp auth-source password-cache json map helm-tags
helm-buffers helm-occur helm-grep helm-regexp helm-utils helm-locate
helm-help helm-types framey-helm helm-config helm async-bytecomp
helm-global-bindings helm-source helm-multi-match helm-lib async gcmh
projectile grep compile text-property-search ibuf-ext ibuffer
ibuffer-loaddefs company-keywords company-dabbrev-code company-dabbrev
company-yasnippet company-files company-capf company-prescient prescient
company server evil-vimish-fold vimish-fold smartparens-config
smartparens-text paren smartparens doom-modeline doom-modeline-segments
doom-modeline-env doom-modeline-core shrink-path f all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons memoize framey inline ht s
window-purpose window-purpose-fixes window-purpose-prefix-overload
window-purpose-switch let-alist window-purpose-layout
window-purpose-core window-purpose-configuration eieio-compat eieio
window-purpose-utils shackle trace winum eyebrowse format-spec
morning-star-theme cap-words superword subword evil-surround evil-lion
evil-goggles pulse evil evil-keybindings evil-integration evil-maps
evil-commands reveal evil-jumps evil-command-window evil-types
evil-search evil-ex shell pcomplete comint ansi-color evil-macros
evil-repeat evil-states evil-core comp comp-cstr warnings subr-x cl-seq
cl-extra help-mode seq byte-opt bytecomp byte-compile cconv advice
evil-common windmove thingatpt rect evil-digraphs evil-vars ring edmacro
kmacro dash yequake-autoloads yasnippet-autoloads yaml-mode-autoloads
xterm-color-autoloads wttrin-autoloads writeroom-mode-autoloads
with-editor-autoloads winum-autoloads window-purpose-autoloads
eieio-core cl-macs eieio-loaddefs cl-loaddefs cl-lib vterm-autoloads
visual-fill-column-autoloads vimish-fold-autoloads ts-autoloads
tridactylrc-mode-autoloads treepy-autoloads treemacs-autoloads
transient-autoloads toml-mode-autoloads toc-org-autoloads
tablist-autoloads straight-autoloads spinner-autoloads
smartparens-autoloads shrink-path-autoloads shackle-autoloads
s-autoloads rust-mode-autoloads restart-emacs-autoloads
rainbow-mode-autoloads rainbow-delimiters-autoloads projectile-autoloads
pretty-hydra-autoloads prescient-autoloads posframe-autoloads
pos-tip-autoloads popup-autoloads pkg-info-autoloads pfuture-autoloads
perspective-autoloads persp-mode-autoloads peep-dired-autoloads
pdf-tools-autoloads pcre2el-autoloads package-lint-autoloads
org-superstar-autoloads org-super-agenda-autoloads org-autoloads
multi-compile-autoloads mu4e-alert-autoloads morning-star-autoloads
memoize-autoloads markdown-mode-autoloads magit-todos-autoloads
magit-autoloads macrostep-autoloads lv-autoloads lsp-ui-autoloads
lsp-treemacs-autoloads lsp-mode-autoloads log4e-autoloads
link-hint-autoloads let-alist-autoloads ledger-mode-autoloads
json-snatcher-autoloads json-reformat-autoloads json-mode-autoloads rx
imenu-list-autoloads i3wm-config-mode-autoloads hydra-autoloads
ht-autoloads hl-todo-autoloads helpful-autoloads
helm-projectile-autoloads helm-org-autoloads helm-easymenu easymenu
helm-core-autoloads helm-ag-autoloads helm-autoloads goto-chg-autoloads
gntp-autoloads git-modes-autoloads git-gutter-fringe-autoloads
git-gutter-autoloads git-commit-autoloads ghub-autoloads
german-holidays-autoloads gcmh-autoloads frog-menu-autoloads
fringe-helper-autoloads framey-autoloads frame-local-autoloads
forge-autoloads flyspell-correct-autoloads flycheck-pos-tip-autoloads
flycheck-autoloads fish-mode-autoloads fill-column-indicator-autoloads
f-autoloads eyebrowse-autoloads expand-region-autoloads
evil-vimish-fold-autoloads evil-surround-autoloads
evil-numbers-autoloads evil-nerd-commenter-autoloads
evil-magit-autoloads evil-lion-autoloads evil-ledger-autoloads
evil-goggles-autoloads evil-exchange-autoloads evil-collection-autoloads
evil-autoloads eros-autoloads epl-autoloads epkg-autoloads
emacsql-sqlite-autoloads emacsql-autoloads elisp-refs-autoloads
elfeed-org-autoloads elfeed-autoloads doom-modeline-autoloads
doct-autoloads dockerfile-mode-autoloads docker-tramp-autoloads
docker-compose-mode-autoloads docker-autoloads dired+-autoloads
dash-functional-autoloads dash-autoloads ctrlf-autoloads easy-mmode
company-shell-autoloads company-quickhelp-autoloads
company-prescient-autoloads company-box-autoloads company-autoloads
closql-autoloads buttercup-autoloads avy-autoloads async-autoloads
anzu-autoloads annalist-autoloads all-the-icons-autoloads
alert-autoloads ace-window-autoloads gv iso-transl tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face pcase macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting font-render-setting cairo move-toolbar gtk x-toolkit x
multi-tty make-network-process nativecomp emacs)

Memory information:
((conses 16 522069 654871)
(symbols 48 39178 0)
(strings 32 130168 48392)
(string-bytes 1 4717070)
(vectors 16 60791)
(vector-slots 8 888360 435946)
(floats 8 754 2012)
(intervals 56 1687 512)
(buffers 984 18))

bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

[Eli Zaretskii]

> From: Alexander Miller <alexanderm@web.de>
> Date: Sun, 20 Dec 2020 18:36:46 +0100
> 
> In my config I have the following function that is used to prettify my
> mode-line (the actual version is pure and side-effect-free and inlined,
> but that does not seem to have any effect here):
> 
> (defun f (n)
>    (pcase n
>      (1 " ➊") (2 " ➋") (3 " ➌") (4 " ➍") (5 " ➎") (6 " ➏")
>      (7 " ➐") (8 " ➑") (9 " ➒") (10 " ➓") (_ "")))
> 
> When native compiled it appears to return raw bytes instead of the
> unicode symbols, for example I am seeing \342\236\212 instead of ➊.

\342\236\212 is the UTF-8 encoding of ➊.  Which probably means the
native-compiled function returns a unibyte string instead of a
multibyte string, for some reason.

bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alexander Miller <alexanderm@web.de> writes:

> In my config I have the following function that is used to prettify my
> mode-line (the actual version is pure and side-effect-free and inlined,
> but that does not seem to have any effect here):
>
> (defun f (n)
>   (pcase n
>     (1 " ➊") (2 " ➋") (3 " ➌") (4 " ➍") (5 " ➎") (6 " ➏")
>     (7 " ➐") (8 " ➑") (9 " ➒") (10 " ➓") (_ "")))
>
> When native compiled it appears to return raw bytes instead of the
> unicode symbols, for example I am seeing \342\236\212 instead of ➊.
>
> I can circumvent that by using (1 (decode-coding-string " ➊" 'utf-8)),
> but of course that should not be necessary, since there's no such
> problem with the byte-compiled version.

Hi Alexander,

72c1a41573 fix this for me, would you like to confirm?

> And on a probably unrelated note: is it normal for such a simple
> function to be compiled to 100 LOC of assembly? That seems surprisingly
> to my amateur eyes.

Yes, big switch cases is ATM a case we do not generate optimal code for.

Thanks for the report!

  Andrea

bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

[Alexander Miller]

Hi Andrea,

I can confirm that 72c1a41573 fixes the issue.

However there's still another problem that I thought was related, but
apparently it isn't. (I can open another bug for it if you like).

It looks like eval-when-compile is not working as expected. I am also
using small xpms for my modeline that are defined like this:

(defconst selected-window-xpm
    (eval-when-compile (doom-modeline--make-xpm 'modeline-xpm-face 5 30)))

The generating code looks like this:

(defun doom-modeline--make-xpm (face width height)
   "Create an XPM bitmap via FACE, WIDTH and HEIGHT. Inspired by 
`powerline''s `pl/make-xpm'."
   (when (and (display-graphic-p)
              (image-type-available-p 'xpm))
     (propertize
      " " 'display
      (let ((data (make-list height (make-list width 1)))
            (color (or (face-background face nil t) "None")))
        (ignore-errors
          (create-image
           (concat
            (format
             "/* XPM */\nstatic char * percent[] = {\n\"%i %i 2 
1\",\n\". c %s\",\n\"  c %s\","
             (length (car data)) (length data) color color)
            (apply #'concat
                   (cl-loop with idx = 0
                            with len = (length data)
                            for dl in data
                            do (cl-incf idx)
                            collect
                            (concat
                             "\""
                             (cl-loop for d in dl
                                      if (= d 0) collect (string-to-char 
" ")
                                      else collect (string-to-char "."))
                             (if (eq idx len) "\"};" "\",\n")))))
   'xpm t :ascent 'center))))))

When native-compiling I only get nil, with byte-compiling it works
without problems. The correct value should look like this:

#(" " 0 1
(display
(image :type xpm :data "/* XPM */
static char * percent[] = {
\"5 30 2 1\",
\". c #559955\",
\" c #559955\",\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\",
\".....\"};" :scale 1 :ascent center)))

bug#45342: 28.0.50; Native compiled function returns raw bytes, not string

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alexander Miller <alexanderm@web.de> writes:

> Hi Andrea,
>
> I can confirm that 72c1a41573 fixes the issue.
>
> However there's still another problem that I thought was related, but
> apparently it isn't. (I can open another bug for it if you like).
>
> It looks like eval-when-compile is not working as expected. I am also
> using small xpms for my modeline that are defined like this:

Hi Alex,

I tried reproducing but I get nil both executing interpreted byte-compiled
and native-compiled, so probably I'm not doing exactly what you do.

Could you open a dedicated bug attaching the full reproducer and how you
exercise it?

I'm closing this one.

Thanks!

  Andrea

bug#45342: closed (Re: bug#45342: 28.0.50; Native compiled function returns raw bytes, not string)

[Alexander Miller]

Hi Andrea,

never mind about the second thing, the function is guarded by a
(and (display-graphic-p) (image-type-available-p 'xpm)) check. I compile
my config in a batch script, so getting nil here is the correct
behaviour.
That does open up the question why it does work with byte-compilation,
but that's probably not a bug, just something I need to figure out.

bug#45342: closed (Re: bug#45342: 28.0.50; Native compiled function returns raw bytes, not string)

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alexander Miller <alexanderm@web.de> writes:

> Hi Andrea,
>
> never mind about the second thing, the function is guarded by a
> (and (display-graphic-p) (image-type-available-p 'xpm)) check. I compile
> my config in a batch script, so getting nil here is the correct
> behaviour.
> That does open up the question why it does work with byte-compilation,
> but that's probably not a bug, just something I need to figure out.

Hi Alexander,

okay, you know where to complain in case :)

  Andrea