bug#55278: 29.0.50; OSM package

bug#55278: 29.0.50; OSM package

[Richard Stallman]

I did M-x osm-search RET London RET
just after starting the package, and I got an error
that function json-parse-string is undefined.

None pf the Lisp sources in my checkout defines that function.
Is it in some ELPA package?



In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.32, cairo version 1.16.0)
 of 2022-04-13 built on freetop
Repository revision: ed6237e039e6fb51229054d0fce74df074903f19
Repository branch: master
System Description: Trisquel GNU/Linux Nabia (10.0)

Configured using:
 'configure 'CFLAGS=-O0 -g' --with-gnutls=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GPM GSETTINGS HARFBUZZ JPEG LIBOTF
LIBSELINUX LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK2
ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: linux
  auto-encryption-mode: t
  auto-compression-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug org-element avl-tree ol-eww eww xdg url-queue mm-url
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-art mm-uu mml2015
mm-view mml-smime smime gnutls dig gnus-sum gnus-group gnus-undo
gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail mail-source utf7
netrc nnoo parse-time gnus-spec gnus-int gnus-range gnus-win gnus
nnheader range wid-edit ol-docview doc-view image-mode exif ol-bibtex
ol-bbdb ol-w3m ol-doi org-link-doi org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
org-list org-faces org-entities noutline outline easy-mmode
org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic bibtex
iso8601 ol rx org-keys oc org-compat org-macs org-loaddefs format-spec
ispell cl-extra cal-move cal-menu calendar cal-loaddefs help-fns
radix-tree cl-print debug backtrace find-func etags fileloop generator
xref project mule-util unrmail epa-mail rmailsum jka-compr textsec
uni-scripts idna-mapping ucs-normalize uni-confusable textsec-check
shr pixel-fill kinsoku url-file url-dired svg xml dom rmailkwd
rmailout qp mailalias shell pcomplete thingatpt files-x grep compile
comint ansi-color ring misearch multi-isearch quail help-mode dabbrev
rmailmm message sendmail yank-media rmc puny rfc822 mml mml-sec epa
epg rfc6068 epg-config gnus-util text-property-search time-date
mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader
mail-parse rfc2231 rmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils dired-aux dired dired-loaddefs t-mouse term/linux view
derived advice finder-inf package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json map url-vars seq gv subr-x
byte-opt bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win
x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice simple cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 810373 84371)
 (symbols 48 35124 37)
 (strings 32 115214 8073)
 (string-bytes 1 3031048)
 (vectors 16 74948)
 (vector-slots 8 1703386 84678)
 (floats 8 278 443)
 (intervals 56 100719 2724)
 (buffers 992 83))
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: 29.0.50; OSM package

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 05 May 2022 14:38:18 -0400
> 
> 
> I did M-x osm-search RET London RET
> just after starting the package, and I got an error
> that function json-parse-string is undefined.
> 
> None pf the Lisp sources in my checkout defines that function.
> Is it in some ELPA package?

No, it's implemented in json.c in the Emacs sources.  But to have that
function defined, you need to build Emacs with libjansson, which is an
optional dependency of Emacs.  The OSM package says in its README:

  The package is available on GNU ELPA and can be installed with =package-install=.
  Note that osm.el requires Emacs 27 and depends on the external =curl= program.
  Emacs must be built with libxml, libjansson, librsvg, libjpeg and libpng
  support. The following is an example configuration which relies on =use-package=.
  Please take a look at the [[https://github.com/minad/osm/wiki][wiki]] for additional configuration.

So it looks like you must to have those optional libraries installed
and you must build Emacs with them, in order to use OSM.

bug#55278: 29.0.50; OSM package

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So it looks like you must to have those optional libraries installed
> and you must build Emacs with them, in order to use OSM.

So this doesn't seem to be a bug, and I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55278: 29.0.50; OSM package

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, it's implemented in json.c in the Emacs sources.  But to have that
  > function defined, you need to build Emacs with libjansson, which is an
  > optional dependency of Emacs.

That is strange.  If the function json-parse-string is defined in the
Emacs sources, why compile it only if something else is enabled?

Does OSM depend on anything about jansson other than json-parse-string?
What does jansson do?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: 29.0.50; OSM package

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Richard Stallman <rms@gnu.org> writes:

> That is strange.  If the function json-parse-string is defined in the
> Emacs sources, why compile it only if something else is enabled?

Jansson is a library for parsing JSON.  Emacs provides two mechanisms
for parsing JSON into Lisp data, one of which is based on the Jansson
library, and the other is implemented in Lisp inside json.el.

OSM uses the former, but it can trivially be made to use the latter:

--- a/osm.el
+++ b/osm.el
@@ -39,6 +39,7 @@
 
 (require 'bookmark)
 (require 'dom)
+(require 'json)
 (eval-when-compile
   (require 'cl-lib)
   (require 'subr-x))
@@ -1376,14 +1377,12 @@
 
 (defun osm--fetch-json (url)
   "Get json from URL."
-  (json-parse-string
+  (json-read-from-string
    (let ((default-process-coding-system '(utf-8-unix . utf-8-unix)))
      (shell-command-to-string
       (concat
        "curl " osm-curl-options " "
-       (shell-quote-argument url))))
-   :array-type 'list
-   :object-type 'alist))
+       (shell-quote-argument url))))))
 
 ;;;###autoload
 (defun osm-search (search &optional lucky)

bug#55278: 29.0.50; OSM package

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: 55278@debbugs.gnu.org
> Date: Fri, 06 May 2022 19:19:32 -0400
> 
>   > No, it's implemented in json.c in the Emacs sources.  But to have that
>   > function defined, you need to build Emacs with libjansson, which is an
>   > optional dependency of Emacs.
> 
> That is strange.  If the function json-parse-string is defined in the
> Emacs sources, why compile it only if something else is enabled?

Because the actual implementation is in the libjansson library; the
Emacs sources only wrap that by a glue to allow use to use JSON with
Emacs Lisp objects.

> Does OSM depend on anything about jansson other than json-parse-string?
> What does jansson do?

The library implements most of JSON handling.

bug#55278: 29.0.50; OSM package

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Because the actual implementation is in the libjansson library; the
  > Emacs sources only wrap that by a glue to allow use to use JSON with
  > Emacs Lisp objects.

That makes sense, by itself.  But the end result is that the proper
functioning of a user-level feature depends on compiling Emacs with a
special library that most users have never heard of.

If Emacs is compiled without libpng and therefore can't display png
files, that will make sense to the user, because the user can see the
relationship between one and the other.  Users will accept that.

But users won't see any relationship between Open Street Map and
libjansson.  They will say, "How was I supposed to know that???"

I think we had better fix this somehow.

Po Lu wrote:

  > Jansson is a library for parsing JSON.  Emacs provides two mechanisms
  > for parsing JSON into Lisp data, one of which is based on the Jansson
  > library, and the other is implemented in Lisp inside json.el.

  > OSM uses the former, but it can trivially be made to use the latter:

  > --- a/osm.el
  > +++ b/osm.el
  > @@ -39,6 +39,7 @@
 
  >  (require 'bookmark)
  >  (require 'dom)
  > +(require 'json)
  >  (eval-when-compile
  >    (require 'cl-lib)
  >    (require 'subr-x))
  > @@ -1376,14 +1377,12 @@
 
  >  (defun osm--fetch-json (url)
  >    "Get json from URL."
  > -  (json-parse-string
  > +  (json-read-from-string
  >     (let ((default-process-coding-system '(utf-8-unix . utf-8-unix)))
  >       (shell-command-to-string
  >        (concat
  >         "curl " osm-curl-options " "
  > -       (shell-quote-argument url))))
  > -   :array-type 'list
  > -   :object-type 'alist))
  > +       (shell-quote-argument url))))))

How about making osm.el determine whether the C-level one is available,
and use the fastest available one?

Or perhaps define a common interface for the two, so we only need to
document that interface.  That would simplify Emacs overall.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Daniel Mendler]

Regarding `json-parse-string` - the `compat` package, which is available
on GNU ELPA ensures that `json-parse-string` is always available. If you
install that, the problem should be fixed. I've talked to Philip (compat
maintainer) about including the `json-parse-string` fallback directly
into Emacs. This should simply Emacs itself slightly, since multiple
packages in Emacs, among them python.el and jsonrpc.el, currently have
to check for the availability of the libjansson-APIs and implement their
own fallback.

Daniel

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This should simply Emacs itself slightly, since multiple
  > packages in Emacs, among them python.el and jsonrpc.el, currently have
  > to check for the availability of the libjansson-APIs and implement their
  > own fallback.

I agree that we should implement some built-in fallback mechanism
so that it works for all packages that use json parsing.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I agree that we should implement some built-in fallback mechanism
> so that it works for all packages that use json parsing.

The two libraries aren't really compatible without reparsing the entire
parse tree returned by them.  For instance, the C library returns:

(json-parse-string "null")
=> :null

While json.el says:

(json-read-from-string "null")
=> nil

Libraries like osm.el depend on the former type, and it parses a lot of
data, making compat transformations prohibitively slow.  We'd be doing
our users a disservice if we were to just be slow here instead of
telling them the real story: Build your Emacs with jansson support,
already.  (This happens by default if you have the library installed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55278: json-parse-string

[Daniel Mendler]

On 5/9/22 12:27, Lars Ingebrigtsen wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
>> I agree that we should implement some built-in fallback mechanism
>> so that it works for all packages that use json parsing.
> 
> The two libraries aren't really compatible without reparsing the entire
> parse tree returned by them.  For instance, the C library returns:

The json-parse-string implementation in compat.el should be compatible
as far as I understood Philip and it relies on json.el.

> Libraries like osm.el depend on the former type, and it parses a lot of
> data, making compat transformations prohibitively slow.  We'd be doing
> our users a disservice if we were to just be slow here instead of
> telling them the real story: Build your Emacs with jansson support,
> already.  (This happens by default if you have the library installed.)

Yes, that's a good point. Should we rather bail out or try to do our
best by falling back to a slower implementation? For example eww bails
out if libxml is missing, probably rightly so.

In osm-mode I have a list of requirements which must be satisfied before
it can be used reasonably. These requirements are also listed in the
README and in the Commentary of the package.

  (unless (display-graphic-p)
    (warn "osm: Graphical display is required"))
  (dolist (type '(svg jpeg png))
    (unless (image-type-available-p type)
      (warn "osm: Support for %s images is missing" type)))
  (unless (libxml-available-p)
    (warn "osm: libxml is not available"))
  (unless (ignore-errors (equal [] (json-parse-string "[]")))
    (warn "osm: libjansson is not available"))

Daniel

bug#55278: json-parse-string

[Lars Ingebrigtsen]

Daniel Mendler <mail@daniel-mendler.de> writes:

> In osm-mode I have a list of requirements which must be satisfied before
> it can be used reasonably. These requirements are also listed in the
> README and in the Commentary of the package.
>
>   (unless (display-graphic-p)
>     (warn "osm: Graphical display is required"))
>   (dolist (type '(svg jpeg png))
>     (unless (image-type-available-p type)
>       (warn "osm: Support for %s images is missing" type)))
>   (unless (libxml-available-p)
>     (warn "osm: libxml is not available"))
>   (unless (ignore-errors (equal [] (json-parse-string "[]")))
>     (warn "osm: libjansson is not available"))

Perhaps it should be signalling errors here instead of warnings?  (And
it would probably be nice if it collected all the requirements and said
"Error: Need libjansson, libxml and" etc in one go.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55278: json-parse-string

[Daniel Mendler]

On 5/9/22 12:45, Lars Ingebrigtsen wrote:
> Daniel Mendler <mail@daniel-mendler.de> writes:
> 
>> In osm-mode I have a list of requirements which must be satisfied before
>> it can be used reasonably. These requirements are also listed in the
>> README and in the Commentary of the package.
>>
>>   (unless (display-graphic-p)
>>     (warn "osm: Graphical display is required"))
>>   (dolist (type '(svg jpeg png))
>>     (unless (image-type-available-p type)
>>       (warn "osm: Support for %s images is missing" type)))
>>   (unless (libxml-available-p)
>>     (warn "osm: libxml is not available"))
>>   (unless (ignore-errors (equal [] (json-parse-string "[]")))
>>     (warn "osm: libjansson is not available"))
> 
> Perhaps it should be signalling errors here instead of warnings?  (And
> it would probably be nice if it collected all the requirements and said
> "Error: Need libjansson, libxml and" etc in one go.)

Okay, I will do that.

Daniel

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > (unless (ignore-errors (equal [] (json-parse-string "[]")))
    >   (warn "osm: libjansson is not available"))

I don't think I saw that warning.  Perhaps it was displayed in a way
that was difficult to see.

I just ran osm-home in a fresh Emacs, and that warning did not appear.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > (unless (ignore-errors (equal [] (json-parse-string "[]")))
    >   (warn "osm: libjansson is not available"))

Assuming that now this warning will actually appear,
it still has two problems.

The text of the warning is not clear.  It fails to say what the
practical consequences are: that OSM simply will not work.

It should be an error, not a warning.  And I suggest text
saying "OSM can't run, because Emacs was built without libjansson."

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     > (unless (ignore-errors (equal [] (json-parse-string "[]")))
>     >   (warn "osm: libjansson is not available"))
>
> Assuming that now this warning will actually appear,
> it still has two problems.
>
> The text of the warning is not clear.  It fails to say what the
> practical consequences are: that OSM simply will not work.

> It should be an error, not a warning.  And I suggest text
> saying "OSM can't run, because Emacs was built without libjansson."

This has already been done, the current error message looks like

     Osm: Please compile Emacs with the required libraries, [list of libraries] needed to proceed

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The text of the warning is not clear.  It fails to say what the
  > > practical consequences are: that OSM simply will not work.

  > > It should be an error, not a warning.  And I suggest text
  > > saying "OSM can't run, because Emacs was built without libjansson."

  > This has already been done, the current error message looks like

That's good.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The two libraries aren't really compatible without reparsing the entire
  > parse tree returned by them.  For instance, the C library returns:

  > (json-parse-string "null")
  > => :null

  > While json.el says:

  > (json-read-from-string "null")
  > => nil

No deep difference between them is visible in this example.
What is the a deep difference? 

Is there a reason why they _must_ give different output formats
or is that difference just for historical reasons?
Could we make a version of the Lisp code
that produces the same output as jansson?
It would be slower but that is better than not running.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#55278: json-parse-string

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   We'd be doing
  > our users a disservice if we were to just be slow here instead of
  > telling them the real story: Build your Emacs with jansson support,
  > already.

We do not tell users that, not now.  What we tell them is that the function
`json-parse-string' is undefined.  Even when they investigate why,
it is hard to find out that this is because of a library called janssen.

The current situation shows we have failed to think in terms of
reliable clear results for the user.

If it is normal for user packages to fail to work unless Emacs is
linked with libjannson, Emacs should give an error unless libjannson
is loaded.  (Or unless the option --nojansson is explicitly given.)

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)