bug#72358: 29.4; oauth2.el improvements

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 72931 bytes --]

Hi,

I have been trying out using oauth2.el to enable OAuth2-based
authentication for email service providers and had some success for
Gmail.  During this process, I have made a few changes to oauth2.el that
enables it to use with Gmail OAuth2 as well as some usability and
debugging improvements, which I'm sharing below.

This is a series of five patches, which are attached.

The first patch shows the authentication URL in the minibuffer window
alongside the prompt accepting the authorization code.  This helps when
a user has multiple accounts from the same provider but is logged into a
different account than the one that the user is trying to set up.  If
the user use the link (or through `browse-url') it will use the active
account instead of the one intended.  By showing the URL in the
minibuffer, the user can choose other ways to get the authorization code
(e.g. using another browser, using private/encognito mode, etc.)

The second patch adds the parameters `access_type=offline' and
`prompt=consent' to the authorization URL, which is required for Gmail
OAuth2 to get the refresh token.  Without these 2 parameters, Gmail
response will only contain the access token which expires in one hour.
They should also be compatible with other OAuth2 authentication process.
(Though I am currently having trouble to get outlook.com to work
regardless of these parameters, which I'll ask in a separate thread.)

Note that the second patch depends on the first patch as they modify the same
part of the code.

The third patch encodes the parameters for requesting refreshing access
token, which is recommended because the client secret and other
parameters may contain characters that may break parameter parsing.

The fourth patch may need a bit of background: oauth2.el (optionally)
uses plstore to save authentication data for future reuse, and the
plstore id for an account is computed using a combination of `auth-url',
`token-url', and `scope'.  However, this combination of data doesn't
guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
the three parameters are the same for different accounts, and hence
storing a second account information will override the first one.

This fourth patch adds `client-id' to the calculation of plstore id to
ensure its uniqueness.  This may cause a few concerns:

- This will invalidate all existing entries and a user will have to redo
  the authorization process again to get a new refresh token.  However,
  I think it's more important to ensure that oauth2.el works correctly
  for multiple accounts of the same provider, or a user may suffer from
  confusion when adding a new account invalidates a previous account.

- Adding `client-id' to the calculation of plstore id may provoke
  suspicion of leaking it as the hash calculation uses md5.  In most
  cases, requesting a refresh token requires both `client-id' and
  `client-secret', so without including the latter it should be safe.
  There are cases when requesting only the access token may work with
  `client-id' along.  Still, I think this should not be a big concern as
  the data is combined with `auth-url', `token-url', and `scope' which
  provides sufficient salt.  Alternatively, we can also choose to use a
  more secure hash function, e.g. SHA2 or better, given that existing
  entries will be invalidated anyway.

The fifth patch adds debug messages when doing a URL query which records
the request URL, the request data, and the response data, and provide a
custom variable to enable this.  This provides a way to help debugging
the requests, and I find it handy when testing oauth2 against different
providers.

Please review and comment.  TIA!


In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
 cairo version 1.16.0) of 2024-07-02, modified by Debian built on
 debian-hx90
System Description: Debian GNU/Linux 12 (bookworm)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/emacs-UNWIcy/emacs-29.4+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2
XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: VTerm

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  mu4e-modeline-mode: t
  windmove-mode: t
  rcirc-track-minor-mode: t
  server-mode: t
  xclip-mode: t
  global-treesit-auto-mode: t
  treemacs-project-follow-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: t
  treemacs-fringe-indicator-mode: t
  corfu-terminal-mode: t
  corfu-popupinfo-mode: t
  corfu-echo-mode: t
  global-corfu-mode: t
  corfu-mode: t
  activities-tabs-mode: t
  activities-mode: t
  fido-vertical-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  fido-mode: t
  override-global-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  global-auto-revert-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-autoloads hides /usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/debian-autoloads
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-section hides /usr/share/emacs/site-lisp/elpa/magit-section-3.3.0/magit-section
/usr/share/emacs/site-lisp/elpa/ace-window-0.10.0/ace-window-autoloads hides /usr/share/emacs/site-lisp/elpa-src/ace-window-0.10.0/ace-window-autoloads
/usr/share/emacs/site-lisp/elpa/ace-window-0.10.0/ace-window-pkg hides /usr/share/emacs/site-lisp/elpa-src/ace-window-0.10.0/ace-window-pkg
/usr/share/emacs/site-lisp/elpa/ace-window-0.10.0/ace-window hides /usr/share/emacs/site-lisp/elpa-src/ace-window-0.10.0/ace-window
/usr/share/emacs/site-lisp/elpa/activities-0.7/activities-autoloads hides /usr/share/emacs/site-lisp/elpa-src/activities-0.7/activities-autoloads
/usr/share/emacs/site-lisp/elpa/activities-0.7/activities-pkg hides /usr/share/emacs/site-lisp/elpa-src/activities-0.7/activities-pkg
/usr/share/emacs/site-lisp/elpa/activities-0.7/activities-list hides /usr/share/emacs/site-lisp/elpa-src/activities-0.7/activities-list
/usr/share/emacs/site-lisp/elpa/activities-0.7/activities hides /usr/share/emacs/site-lisp/elpa-src/activities-0.7/activities
/usr/share/emacs/site-lisp/elpa/activities-0.7/activities-tabs hides /usr/share/emacs/site-lisp/elpa-src/activities-0.7/activities-tabs
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode-autoloads
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode-pkg
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode hides /usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-info hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-info
/usr/share/emacs/site-lisp/elpa/auctex-13.3/latex-flymake hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/latex-flymake
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-site hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-site
/usr/share/emacs/site-lisp/elpa/auctex-13.3/texmathp hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/texmathp
/usr/share/emacs/site-lisp/elpa/auctex-13.3/toolbar-x hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/toolbar-x
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-style hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-style
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-font hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-font
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-jp hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-jp
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-mik hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-mik
/usr/share/emacs/site-lisp/elpa/auctex-13.3/plain-tex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/plain-tex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-fold hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-fold
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-ispell hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-ispell
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-bar hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-bar
/usr/share/emacs/site-lisp/elpa/auctex-13.3/preview-latex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/preview-latex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/bib-cite hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/bib-cite
/usr/share/emacs/site-lisp/elpa/auctex-13.3/preview hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/preview
/usr/share/emacs/site-lisp/elpa/auctex-13.3/context-nl hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/context-nl
/usr/share/emacs/site-lisp/elpa/auctex-13.3/auto-loads hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/auto-loads
/usr/share/emacs/site-lisp/elpa/auctex-13.3/multi-prompt hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/multi-prompt
/usr/share/emacs/site-lisp/elpa/auctex-13.3/context-en hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/context-en
/usr/share/emacs/site-lisp/elpa/auctex-13.3/lpath hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/lpath
/usr/share/emacs/site-lisp/elpa/auctex-13.3/auctex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/auctex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/auctex-autoloads hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/auctex-autoloads
/usr/share/emacs/site-lisp/elpa/auctex-13.3/font-latex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/font-latex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/auctex-pkg hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/auctex-pkg
/usr/share/emacs/site-lisp/elpa/auctex-13.3/context hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/context
/usr/share/emacs/site-lisp/elpa/auctex-13.3/latex hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/latex
/usr/share/emacs/site-lisp/elpa/auctex-13.3/tex-wizard hides /usr/share/emacs/site-lisp/elpa-src/auctex-13.3/tex-wizard
/usr/share/emacs/site-lisp/elpa/avy-0.5.0/avy hides /usr/share/emacs/site-lisp/elpa-src/avy-0.5.0/avy
/usr/share/emacs/site-lisp/elpa/avy-0.5.0/avy-autoloads hides /usr/share/emacs/site-lisp/elpa-src/avy-0.5.0/avy-autoloads
/usr/share/emacs/site-lisp/elpa/avy-0.5.0/avy-pkg hides /usr/share/emacs/site-lisp/elpa-src/avy-0.5.0/avy-pkg
/usr/share/emacs/site-lisp/elpa/bazel-0/bazel-autoloads hides /usr/share/emacs/site-lisp/elpa-src/bazel-0/bazel-autoloads
/usr/share/emacs/site-lisp/elpa/bazel-0/bazel hides /usr/share/emacs/site-lisp/elpa-src/bazel-0/bazel
/usr/share/emacs/site-lisp/elpa/bazel-0/test hides /usr/share/emacs/site-lisp/elpa-src/bazel-0/test
/usr/share/emacs/site-lisp/elpa/bazel-0/bazel-pkg hides /usr/share/emacs/site-lisp/elpa-src/bazel-0/bazel-pkg
/usr/share/emacs/site-lisp/elpa/bison-mode-0.3/bison-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/bison-mode-0.3/bison-mode-pkg
/usr/share/emacs/site-lisp/elpa/bison-mode-0.3/bison-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/bison-mode-0.3/bison-mode-autoloads
/usr/share/emacs/site-lisp/elpa/bison-mode-0.3/bison-mode hides /usr/share/emacs/site-lisp/elpa-src/bison-mode-0.3/bison-mode
/usr/share/emacs/site-lisp/elpa/boxquote-2.3/boxquote hides /usr/share/emacs/site-lisp/elpa-src/boxquote-2.3/boxquote
/usr/share/emacs/site-lisp/elpa/boxquote-2.3/boxquote-autoloads hides /usr/share/emacs/site-lisp/elpa-src/boxquote-2.3/boxquote-autoloads
/usr/share/emacs/site-lisp/elpa/boxquote-2.3/boxquote-pkg hides /usr/share/emacs/site-lisp/elpa-src/boxquote-2.3/boxquote-pkg
/usr/share/emacs/site-lisp/elpa/buttercup-1.26/buttercup-pkg hides /usr/share/emacs/site-lisp/elpa-src/buttercup-1.26/buttercup-pkg
/usr/share/emacs/site-lisp/elpa/buttercup-1.26/buttercup hides /usr/share/emacs/site-lisp/elpa-src/buttercup-1.26/buttercup
/usr/share/emacs/site-lisp/elpa/buttercup-1.26/buttercup-autoloads hides /usr/share/emacs/site-lisp/elpa-src/buttercup-1.26/buttercup-autoloads
/usr/share/emacs/site-lisp/elpa/buttercup-1.26/buttercup-compat hides /usr/share/emacs/site-lisp/elpa-src/buttercup-1.26/buttercup-compat
/usr/share/emacs/site-lisp/elpa/cfrs-1.6.0/cfrs-pkg hides /usr/share/emacs/site-lisp/elpa-src/cfrs-1.6.0/cfrs-pkg
/usr/share/emacs/site-lisp/elpa/cfrs-1.6.0/cfrs hides /usr/share/emacs/site-lisp/elpa-src/cfrs-1.6.0/cfrs
/usr/share/emacs/site-lisp/elpa/cfrs-1.6.0/cfrs-autoloads hides /usr/share/emacs/site-lisp/elpa-src/cfrs-1.6.0/cfrs-autoloads
/usr/share/emacs/site-lisp/elpa/clojure-mode-5.19.0/clojure-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-5.19.0/clojure-mode-pkg
/usr/share/emacs/site-lisp/elpa/clojure-mode-5.19.0/clojure-mode hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-5.19.0/clojure-mode
/usr/share/emacs/site-lisp/elpa/clojure-mode-5.19.0/clojure-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-5.19.0/clojure-mode-autoloads
/usr/share/emacs/site-lisp/elpa/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking-pkg hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking-pkg
/usr/share/emacs/site-lisp/elpa/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking-autoloads hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking-autoloads
/usr/share/emacs/site-lisp/elpa/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking hides /usr/share/emacs/site-lisp/elpa-src/clojure-mode-extra-font-locking-3.0.0/clojure-mode-extra-font-locking
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.29.0/cmake-mode hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.29.0/cmake-mode
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.29.0/cmake-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.29.0/cmake-mode-autoloads
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.29.0/cmake-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.29.0/cmake-mode-pkg
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-dabbrev hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-dabbrev
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-capf hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-capf
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-yasnippet hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-yasnippet
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-ispell hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-ispell
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-etags hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-etags
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-template hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-template
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-abbrev hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-abbrev
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-files hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-files
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-css hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-css
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-tests hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-tests
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-dabbrev-code hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-dabbrev-code
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-pkg hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-pkg
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-oddmuse hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-oddmuse
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-bbdb hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-bbdb
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-clang hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-clang
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-nxml hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-nxml
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-gtags hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-gtags
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-tempo hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-tempo
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-autoloads hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-autoloads
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-cmake hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-cmake
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-tng hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-tng
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-elisp hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-elisp
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-semantic hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-semantic
/usr/share/emacs/site-lisp/elpa/company-0.10.2/company-keywords hides /usr/share/emacs/site-lisp/elpa-src/company-0.10.2/company-keywords
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-26 hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-26
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-28 hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-28
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-macs hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-macs
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-autoloads hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-autoloads
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-pkg hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-pkg
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-25 hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-25
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-29 hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-29
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat-27 hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat-27
/usr/share/emacs/site-lisp/elpa/compat-29.1.4.5/compat hides /usr/share/emacs/site-lisp/elpa-src/compat-29.1.4.5/compat
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-quick hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-quick
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-info hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-info
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-history hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-history
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-popupinfo hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-popupinfo
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-indexed hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-indexed
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-pkg hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-pkg
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-echo hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-echo
/usr/share/emacs/site-lisp/elpa/corfu-1.4/corfu-autoloads hides /usr/share/emacs/site-lisp/elpa-src/corfu-1.4/corfu-autoloads
/usr/share/emacs/site-lisp/elpa/corfu-terminal-0.7/corfu-terminal-autoloads hides /usr/share/emacs/site-lisp/elpa-src/corfu-terminal-0.7/corfu-terminal-autoloads
/usr/share/emacs/site-lisp/elpa/corfu-terminal-0.7/corfu-terminal-pkg hides /usr/share/emacs/site-lisp/elpa-src/corfu-terminal-0.7/corfu-terminal-pkg
/usr/share/emacs/site-lisp/elpa/corfu-terminal-0.7/corfu-terminal hides /usr/share/emacs/site-lisp/elpa-src/corfu-terminal-0.7/corfu-terminal
/usr/share/emacs/site-lisp/elpa/csv-mode-1.23/csv-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.23/csv-mode-autoloads
/usr/share/emacs/site-lisp/elpa/csv-mode-1.23/csv-mode hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.23/csv-mode
/usr/share/emacs/site-lisp/elpa/csv-mode-1.23/csv-mode-tests hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.23/csv-mode-tests
/usr/share/emacs/site-lisp/elpa/csv-mode-1.23/csv-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.23/csv-mode-pkg
/usr/share/emacs/site-lisp/elpa/dart-mode-1.0.7/dart-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/dart-mode-1.0.7/dart-mode-autoloads
/usr/share/emacs/site-lisp/elpa/dart-mode-1.0.7/dart-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/dart-mode-1.0.7/dart-mode-pkg
/usr/share/emacs/site-lisp/elpa/dart-mode-1.0.7/dart-mode hides /usr/share/emacs/site-lisp/elpa-src/dart-mode-1.0.7/dart-mode
/usr/share/emacs/site-lisp/elpa/dash-2.19.1/dash hides /usr/share/emacs/site-lisp/elpa-src/dash-2.19.1/dash
/usr/share/emacs/site-lisp/elpa/dash-2.19.1/dash-pkg hides /usr/share/emacs/site-lisp/elpa-src/dash-2.19.1/dash-pkg
/usr/share/emacs/site-lisp/elpa/dash-2.19.1/dash-autoloads hides /usr/share/emacs/site-lisp/elpa-src/dash-2.19.1/dash-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-el-autoloads hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/debian-el-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/apt-sources hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/apt-sources
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-bug hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/debian-bug
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/apt-utils hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/apt-utils
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-el-pkg hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/debian-el-pkg
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-autoloads hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/debian-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/gnus-BTS hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/gnus-BTS
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/deb-view hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/deb-view
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-el hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/debian-el
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/preseed hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37.16/preseed
/usr/share/emacs/site-lisp/elpa/debpaste-0.1.5/debpaste hides /usr/share/emacs/site-lisp/elpa-src/debpaste-0.1.5/debpaste
/usr/share/emacs/site-lisp/elpa/debpaste-0.1.5/debpaste-pkg hides /usr/share/emacs/site-lisp/elpa-src/debpaste-0.1.5/debpaste-pkg
/usr/share/emacs/site-lisp/elpa/debpaste-0.1.5/debpaste-autoloads hides /usr/share/emacs/site-lisp/elpa-src/debpaste-0.1.5/debpaste-autoloads
/usr/share/emacs/site-lisp/elpa/devscripts-40/devscripts hides /usr/share/emacs/site-lisp/elpa-src/devscripts-40/devscripts
/usr/share/emacs/site-lisp/elpa/devscripts-40/devscripts-autoloads hides /usr/share/emacs/site-lisp/elpa-src/devscripts-40/devscripts-autoloads
/usr/share/emacs/site-lisp/elpa/devscripts-40/pbuilder-mode hides /usr/share/emacs/site-lisp/elpa-src/devscripts-40/pbuilder-mode
/usr/share/emacs/site-lisp/elpa/devscripts-40/devscripts-pkg hides /usr/share/emacs/site-lisp/elpa-src/devscripts-40/devscripts-pkg
/usr/share/emacs/site-lisp/elpa/devscripts-40/pbuilder-log-view-mode hides /usr/share/emacs/site-lisp/elpa-src/devscripts-40/pbuilder-log-view-mode
/usr/share/emacs/site-lisp/elpa/dockerfile-mode-1.7/dockerfile-mode hides /usr/share/emacs/site-lisp/elpa-src/dockerfile-mode-1.7/dockerfile-mode
/usr/share/emacs/site-lisp/elpa/dockerfile-mode-1.7/dockerfile-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/dockerfile-mode-1.7/dockerfile-mode-autoloads
/usr/share/emacs/site-lisp/elpa/dockerfile-mode-1.7/dockerfile-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/dockerfile-mode-1.7/dockerfile-mode-pkg
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/debian-bts-control hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/debian-bts-control
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/debian-changelog-mode hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/debian-changelog-mode
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/debian-autopkgtest-control-mode hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/debian-autopkgtest-control-mode
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/dpkg-dev-el-autoloads hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/dpkg-dev-el-autoloads
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/dpkg-dev-el-pkg hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/dpkg-dev-el-pkg
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/dpkg-dev-el hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/dpkg-dev-el
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/debian-control-mode hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/debian-control-mode
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/debian-copyright hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/debian-copyright
/usr/share/emacs/site-lisp/elpa/dpkg-dev-el-37.16/readme-debian hides /usr/share/emacs/site-lisp/elpa-src/dpkg-dev-el-37.16/readme-debian
/usr/share/emacs/site-lisp/elpa/exec-path-from-shell-2.1/exec-path-from-shell hides /usr/share/emacs/site-lisp/elpa-src/exec-path-from-shell-2.1/exec-path-from-shell
/usr/share/emacs/site-lisp/elpa/exec-path-from-shell-2.1/exec-path-from-shell-autoloads hides /usr/share/emacs/site-lisp/elpa-src/exec-path-from-shell-2.1/exec-path-from-shell-autoloads
/usr/share/emacs/site-lisp/elpa/exec-path-from-shell-2.1/exec-path-from-shell-pkg hides /usr/share/emacs/site-lisp/elpa-src/exec-path-from-shell-2.1/exec-path-from-shell-pkg
/usr/share/emacs/site-lisp/elpa/format-all-0.6.0/format-all hides /usr/share/emacs/site-lisp/elpa-src/format-all-0.6.0/format-all
/usr/share/emacs/site-lisp/elpa/format-all-0.6.0/format-all-pkg hides /usr/share/emacs/site-lisp/elpa-src/format-all-0.6.0/format-all-pkg
/usr/share/emacs/site-lisp/elpa/format-all-0.6.0/format-all-autoloads hides /usr/share/emacs/site-lisp/elpa-src/format-all-0.6.0/format-all-autoloads
/usr/share/emacs/site-lisp/elpa/git-commit-3.3.0/git-commit hides /usr/share/emacs/site-lisp/elpa-src/git-commit-3.3.0/git-commit
/usr/share/emacs/site-lisp/elpa/git-commit-3.3.0/git-commit-autoloads hides /usr/share/emacs/site-lisp/elpa-src/git-commit-3.3.0/git-commit-autoloads
/usr/share/emacs/site-lisp/elpa/git-commit-3.3.0/git-commit-pkg hides /usr/share/emacs/site-lisp/elpa-src/git-commit-3.3.0/git-commit-pkg
/usr/share/emacs/site-lisp/elpa/git-modes-1.4.2/git-modes hides /usr/share/emacs/site-lisp/elpa-src/git-modes-1.4.2/git-modes
/usr/share/emacs/site-lisp/elpa/git-modes-1.4.2/git-modes-pkg hides /usr/share/emacs/site-lisp/elpa-src/git-modes-1.4.2/git-modes-pkg
/usr/share/emacs/site-lisp/elpa/git-modes-1.4.2/git-modes-autoloads hides /usr/share/emacs/site-lisp/elpa-src/git-modes-1.4.2/git-modes-autoloads
/usr/share/emacs/site-lisp/elpa/gitattributes-mode-1.4.2/gitattributes-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/gitattributes-mode-1.4.2/gitattributes-mode-pkg
/usr/share/emacs/site-lisp/elpa/gitattributes-mode-1.4.2/gitattributes-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gitattributes-mode-1.4.2/gitattributes-mode-autoloads
/usr/share/emacs/site-lisp/elpa/gitattributes-mode-1.4.2/gitattributes-mode hides /usr/share/emacs/site-lisp/elpa-src/gitattributes-mode-1.4.2/gitattributes-mode
/usr/share/emacs/site-lisp/elpa/gitconfig-mode-1.4.2/gitconfig-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gitconfig-mode-1.4.2/gitconfig-mode-autoloads
/usr/share/emacs/site-lisp/elpa/gitconfig-mode-1.4.2/gitconfig-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/gitconfig-mode-1.4.2/gitconfig-mode-pkg
/usr/share/emacs/site-lisp/elpa/gitconfig-mode-1.4.2/gitconfig-mode hides /usr/share/emacs/site-lisp/elpa-src/gitconfig-mode-1.4.2/gitconfig-mode
/usr/share/emacs/site-lisp/elpa/gitignore-mode-1.4.2/gitignore-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/gitignore-mode-1.4.2/gitignore-mode-pkg
/usr/share/emacs/site-lisp/elpa/gitignore-mode-1.4.2/gitignore-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gitignore-mode-1.4.2/gitignore-mode-autoloads
/usr/share/emacs/site-lisp/elpa/gitignore-mode-1.4.2/gitignore-mode hides /usr/share/emacs/site-lisp/elpa-src/gitignore-mode-1.4.2/gitignore-mode
/usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/gnuplot hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/gnuplot
/usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/gnuplot-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/gnuplot-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37.16/debian-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/debian-autoloads
/usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/gnuplot-pkg hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/gnuplot-pkg
/usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/gnuplot-context hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/gnuplot-context
/usr/share/emacs/site-lisp/elpa/gnuplot-0.8.1/gnuplot-gui hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-0.8.1/gnuplot-gui
/usr/share/emacs/site-lisp/elpa/go-mode-1.6.0/go-mode hides /usr/share/emacs/site-lisp/elpa-src/go-mode-1.6.0/go-mode
/usr/share/emacs/site-lisp/elpa/go-mode-1.6.0/go-guru hides /usr/share/emacs/site-lisp/elpa-src/go-mode-1.6.0/go-guru
/usr/share/emacs/site-lisp/elpa/go-mode-1.6.0/go-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/go-mode-1.6.0/go-mode-pkg
/usr/share/emacs/site-lisp/elpa/go-mode-1.6.0/go-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/go-mode-1.6.0/go-mode-autoloads
/usr/share/emacs/site-lisp/elpa/go-mode-1.6.0/go-rename hides /usr/share/emacs/site-lisp/elpa-src/go-mode-1.6.0/go-rename
/usr/share/emacs/site-lisp/elpa/graphviz-dot-mode-0.4.2/graphviz-dot-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/graphviz-dot-mode-0.4.2/graphviz-dot-mode-autoloads
/usr/share/emacs/site-lisp/elpa/graphviz-dot-mode-0.4.2/graphviz-dot-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/graphviz-dot-mode-0.4.2/graphviz-dot-mode-pkg
/usr/share/emacs/site-lisp/elpa/graphviz-dot-mode-0.4.2/graphviz-dot-mode hides /usr/share/emacs/site-lisp/elpa-src/graphviz-dot-mode-0.4.2/graphviz-dot-mode
/usr/share/emacs/site-lisp/elpa/ht-2.3/ht-pkg hides /usr/share/emacs/site-lisp/elpa-src/ht-2.3/ht-pkg
/usr/share/emacs/site-lisp/elpa/ht-2.3/ht-autoloads hides /usr/share/emacs/site-lisp/elpa-src/ht-2.3/ht-autoloads
/usr/share/emacs/site-lisp/elpa/ht-2.3/ht hides /usr/share/emacs/site-lisp/elpa-src/ht-2.3/ht
/usr/share/emacs/site-lisp/elpa/hydra-0.15.0/hydra-ox hides /usr/share/emacs/site-lisp/elpa-src/hydra-0.15.0/hydra-ox
/usr/share/emacs/site-lisp/elpa/hydra-0.15.0/hydra-autoloads hides /usr/share/emacs/site-lisp/elpa-src/hydra-0.15.0/hydra-autoloads
/usr/share/emacs/site-lisp/elpa/hydra-0.15.0/hydra-pkg hides /usr/share/emacs/site-lisp/elpa-src/hydra-0.15.0/hydra-pkg
/usr/share/emacs/site-lisp/elpa/hydra-0.15.0/hydra-examples hides /usr/share/emacs/site-lisp/elpa-src/hydra-0.15.0/hydra-examples
/usr/share/emacs/site-lisp/elpa/hydra-0.15.0/hydra hides /usr/share/emacs/site-lisp/elpa-src/hydra-0.15.0/hydra
/usr/share/emacs/site-lisp/elpa/inheritenv-0.2/inheritenv hides /usr/share/emacs/site-lisp/elpa-src/inheritenv-0.2/inheritenv
/usr/share/emacs/site-lisp/elpa/inheritenv-0.2/inheritenv-autoloads hides /usr/share/emacs/site-lisp/elpa-src/inheritenv-0.2/inheritenv-autoloads
/usr/share/emacs/site-lisp/elpa/inheritenv-0.2/inheritenv-pkg hides /usr/share/emacs/site-lisp/elpa-src/inheritenv-0.2/inheritenv-pkg
/usr/share/emacs/site-lisp/elpa/inheritenv-0.2/inheritenv-tests hides /usr/share/emacs/site-lisp/elpa-src/inheritenv-0.2/inheritenv-tests
/usr/share/emacs/site-lisp/elpa/language-id-0.20/language-id hides /usr/share/emacs/site-lisp/elpa-src/language-id-0.20/language-id
/usr/share/emacs/site-lisp/elpa/language-id-0.20/language-id-pkg hides /usr/share/emacs/site-lisp/elpa-src/language-id-0.20/language-id-pkg
/usr/share/emacs/site-lisp/elpa/language-id-0.20/language-id-autoloads hides /usr/share/emacs/site-lisp/elpa-src/language-id-0.20/language-id-autoloads
/usr/share/emacs/site-lisp/elpa/lintian-0.1/lintian-pkg hides /usr/share/emacs/site-lisp/elpa-src/lintian-0.1/lintian-pkg
/usr/share/emacs/site-lisp/elpa/lintian-0.1/lintian-autoloads hides /usr/share/emacs/site-lisp/elpa-src/lintian-0.1/lintian-autoloads
/usr/share/emacs/site-lisp/elpa/lintian-0.1/lintian hides /usr/share/emacs/site-lisp/elpa-src/lintian-0.1/lintian
/usr/share/emacs/site-lisp/elpa/lv-0.15.0/lv-autoloads hides /usr/share/emacs/site-lisp/elpa-src/lv-0.15.0/lv-autoloads
/usr/share/emacs/site-lisp/elpa/lv-0.15.0/lv hides /usr/share/emacs/site-lisp/elpa-src/lv-0.15.0/lv
/usr/share/emacs/site-lisp/elpa/lv-0.15.0/lv-pkg hides /usr/share/emacs/site-lisp/elpa-src/lv-0.15.0/lv-pkg
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-remote hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-remote
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/git-rebase hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/git-rebase
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-bisect hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-bisect
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-margin hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-margin
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-merge hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-merge
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-section hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-section
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-patch hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-patch
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-commit hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-commit
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-autoloads hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-autoloads
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-files hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-files
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-stash hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-stash
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-bookmark hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-bookmark
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-submodule hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-submodule
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-apply hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-apply
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-repos hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-repos
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-core hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-core
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-subtree hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-subtree
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-autorevert hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-autorevert
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-gitignore hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-gitignore
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-transient hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-transient
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-extras hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-extras
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-git hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-git
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-notes hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-notes
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-reflog hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-reflog
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-mode hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-mode
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-push hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-push
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-tag hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-tag
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-process hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-process
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-ediff hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-ediff
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-imenu hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-imenu
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-diff hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-diff
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-clone hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-clone
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-log hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-log
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-utils hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-utils
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-wip hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-wip
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-branch hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-branch
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-pull hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-pull
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-reset hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-reset
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-sequence hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-sequence
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-status hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-status
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-refs hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-refs
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-obsolete hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-obsolete
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-fetch hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-fetch
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-worktree hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-worktree
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-blame hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-blame
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-pkg hides /usr/share/emacs/site-lisp/elpa-src/magit-3.3.0/magit-pkg
/usr/share/emacs/site-lisp/elpa/magit-section-3.3.0/magit-section-autoloads hides /usr/share/emacs/site-lisp/elpa-src/magit-section-3.3.0/magit-section-autoloads
/usr/share/emacs/site-lisp/elpa/magit-3.3.0/magit-section hides /usr/share/emacs/site-lisp/elpa-src/magit-section-3.3.0/magit-section
/usr/share/emacs/site-lisp/elpa/magit-section-3.3.0/magit-section-pkg hides /usr/share/emacs/site-lisp/elpa-src/magit-section-3.3.0/magit-section-pkg
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-cgen hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-cgen
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-scan hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-scan
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-publish hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-publish
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-complete hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-complete
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-syntax hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-syntax
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/tlc hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/tlc
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-netshell hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-netshell
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/semantic-matlab hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/semantic-matlab
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/mlint hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/mlint
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/company-matlab-shell hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/company-matlab-shell
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-shell-gud hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-shell-gud
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/mlgud hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/mlgud
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/srecode-matlab hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/srecode-matlab
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-compat hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-compat
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-mode-pkg
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-maint hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-maint
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/cedet-matlab hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/cedet-matlab
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-topic hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-topic
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/linemark hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/linemark
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-shell hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-shell
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/semanticdb-matlab hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/semanticdb-matlab
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-load hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-load
/usr/share/emacs/site-lisp/elpa/matlab-mode-4.0/matlab-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/matlab-mode-4.0/matlab-mode-autoloads
/usr/share/emacs/site-lisp/elpa/meson-mode-0.2/meson-mode hides /usr/share/emacs/site-lisp/elpa-src/meson-mode-0.2/meson-mode
/usr/share/emacs/site-lisp/elpa/meson-mode-0.2/utils hides /usr/share/emacs/site-lisp/elpa-src/meson-mode-0.2/utils
/usr/share/emacs/site-lisp/elpa/meson-mode-0.2/meson-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/meson-mode-0.2/meson-mode-autoloads
/usr/share/emacs/site-lisp/elpa/meson-mode-0.2/meson-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/meson-mode-0.2/meson-mode-pkg
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-draft hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-draft
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-modeline hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-modeline
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-view hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-view
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-message hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-message
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-helpers hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-helpers
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-pkg hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-pkg
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-bookmarks hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-bookmarks
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-thread hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-thread
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-mime-parts hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-mime-parts
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-server hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-server
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-query-items hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-query-items
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-contrib hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-contrib
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-window hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-window
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-config hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-config
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-autoloads hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-autoloads
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-icalendar hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-icalendar
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-mark hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-mark
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-headers hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-headers
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-org hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-org
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-contacts hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-contacts
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-speedbar hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-speedbar
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-obsolete hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-obsolete
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-vars hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-vars
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-actions hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-actions
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-main hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-main
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-search hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-search
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-notification hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-notification
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-context hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-context
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-compose hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-compose
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-lists hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-lists
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-folders hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-folders
/usr/share/emacs/site-lisp/elpa/mu4e-1.12.5/mu4e-update hides /usr/share/emacs/site-lisp/elpa-src/mu4e-1.12.5/mu4e-update
/usr/share/emacs/site-lisp/elpa/nginx-mode-1.1.9/nginx-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/nginx-mode-1.1.9/nginx-mode-autoloads
/usr/share/emacs/site-lisp/elpa/nginx-mode-1.1.9/nginx-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/nginx-mode-1.1.9/nginx-mode-pkg
/usr/share/emacs/site-lisp/elpa/nginx-mode-1.1.9/nginx-mode hides /usr/share/emacs/site-lisp/elpa-src/nginx-mode-1.1.9/nginx-mode
/usr/share/emacs/site-lisp/elpa/paredit-26/paredit-autoloads hides /usr/share/emacs/site-lisp/elpa-src/paredit-26/paredit-autoloads
/usr/share/emacs/site-lisp/elpa/paredit-26/paredit-pkg hides /usr/share/emacs/site-lisp/elpa-src/paredit-26/paredit-pkg
/usr/share/emacs/site-lisp/elpa/paredit-26/paredit hides /usr/share/emacs/site-lisp/elpa-src/paredit-26/paredit
/usr/share/emacs/site-lisp/elpa/persist-0.6.1/persist hides /usr/share/emacs/site-lisp/elpa-src/persist-0.6.1/persist
/usr/share/emacs/site-lisp/elpa/persist-0.6.1/persist-pkg hides /usr/share/emacs/site-lisp/elpa-src/persist-0.6.1/persist-pkg
/usr/share/emacs/site-lisp/elpa/persist-0.6.1/persist-autoloads hides /usr/share/emacs/site-lisp/elpa-src/persist-0.6.1/persist-autoloads
/usr/share/emacs/site-lisp/elpa/pfuture-1.9/pfuture hides /usr/share/emacs/site-lisp/elpa-src/pfuture-1.9/pfuture
/usr/share/emacs/site-lisp/elpa/pfuture-1.9/pfuture-autoloads hides /usr/share/emacs/site-lisp/elpa-src/pfuture-1.9/pfuture-autoloads
/usr/share/emacs/site-lisp/elpa/pfuture-1.9/pfuture-pkg hides /usr/share/emacs/site-lisp/elpa-src/pfuture-1.9/pfuture-pkg
/usr/share/emacs/site-lisp/elpa/po-mode-0.21/po-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/po-mode-0.21/po-mode-pkg
/usr/share/emacs/site-lisp/elpa/po-mode-0.21/po-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/po-mode-0.21/po-mode-autoloads
/usr/share/emacs/site-lisp/elpa/po-mode-0.21/po-mode hides /usr/share/emacs/site-lisp/elpa-src/po-mode-0.21/po-mode
/usr/share/emacs/site-lisp/elpa/popon-0.13/popon hides /usr/share/emacs/site-lisp/elpa-src/popon-0.13/popon
/usr/share/emacs/site-lisp/elpa/popon-0.13/popon-autoloads hides /usr/share/emacs/site-lisp/elpa-src/popon-0.13/popon-autoloads
/usr/share/emacs/site-lisp/elpa/popon-0.13/popon-pkg hides /usr/share/emacs/site-lisp/elpa-src/popon-0.13/popon-pkg
/usr/share/emacs/site-lisp/elpa/posframe-1.1.7/posframe-pkg hides /usr/share/emacs/site-lisp/elpa-src/posframe-1.1.7/posframe-pkg
/usr/share/emacs/site-lisp/elpa/posframe-1.1.7/posframe-autoloads hides /usr/share/emacs/site-lisp/elpa-src/posframe-1.1.7/posframe-autoloads
/usr/share/emacs/site-lisp/elpa/posframe-1.1.7/posframe hides /usr/share/emacs/site-lisp/elpa-src/posframe-1.1.7/posframe
/usr/share/emacs/site-lisp/elpa/projectile-2.8.0/projectile-pkg hides /usr/share/emacs/site-lisp/elpa-src/projectile-2.8.0/projectile-pkg
/usr/share/emacs/site-lisp/elpa/projectile-2.8.0/projectile-autoloads hides /usr/share/emacs/site-lisp/elpa-src/projectile-2.8.0/projectile-autoloads
/usr/share/emacs/site-lisp/elpa/projectile-2.8.0/projectile hides /usr/share/emacs/site-lisp/elpa-src/projectile-2.8.0/projectile
/usr/share/emacs/site-lisp/elpa/py-isort-2016.1/py-isort hides /usr/share/emacs/site-lisp/elpa-src/py-isort-2016.1/py-isort
/usr/share/emacs/site-lisp/elpa/py-isort-2016.1/py-isort-autoloads hides /usr/share/emacs/site-lisp/elpa-src/py-isort-2016.1/py-isort-autoloads
/usr/share/emacs/site-lisp/elpa/py-isort-2016.1/py-isort-pkg hides /usr/share/emacs/site-lisp/elpa-src/py-isort-2016.1/py-isort-pkg
/usr/share/emacs/site-lisp/elpa/pyvenv-1.21/pyvenv hides /usr/share/emacs/site-lisp/elpa-src/pyvenv-1.21/pyvenv
/usr/share/emacs/site-lisp/elpa/pyvenv-1.21/pyvenv-pkg hides /usr/share/emacs/site-lisp/elpa-src/pyvenv-1.21/pyvenv-pkg
/usr/share/emacs/site-lisp/elpa/pyvenv-1.21/pyvenv-autoloads hides /usr/share/emacs/site-lisp/elpa-src/pyvenv-1.21/pyvenv-autoloads
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-common hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-common
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-mode-tests hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-mode-tests
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-mode-treesitter hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-mode-treesitter
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-cargo hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-cargo
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-mode-autoloads
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-utils hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-utils
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-rustfmt hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-rustfmt
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-mode hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-mode
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-playpen hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-playpen
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-prog-mode hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-prog-mode
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-mode-pkg
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-cargo-tests hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-cargo-tests
/usr/share/emacs/site-lisp/elpa/rust-mode-1.0.5/rust-compile hides /usr/share/emacs/site-lisp/elpa-src/rust-mode-1.0.5/rust-compile
/usr/share/emacs/site-lisp/elpa/s-1.12.0/s-autoloads hides /usr/share/emacs/site-lisp/elpa-src/s-1.12.0/s-autoloads
/usr/share/emacs/site-lisp/elpa/s-1.12.0/s-pkg hides /usr/share/emacs/site-lisp/elpa-src/s-1.12.0/s-pkg
/usr/share/emacs/site-lisp/elpa/s-1.12.0/s hides /usr/share/emacs/site-lisp/elpa-src/s-1.12.0/s
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-map hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-map
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-indent hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-indent
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-syntax hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-syntax
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-lib hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-lib
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-pkg
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-prettify-symbols hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-prettify-symbols
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-compile hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-compile
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-organise hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-organise
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-autoloads
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-paragraph hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-paragraph
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-imenu hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-imenu
/usr/share/emacs/site-lisp/elpa/scala-mode-1.1.0/scala-mode-fontlock hides /usr/share/emacs/site-lisp/elpa-src/scala-mode-1.1.0/scala-mode-fontlock
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq-24 hides /usr/share/emacs/site-lisp/elpa-src/seq-2.24/seq-24
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq-autoloads hides /usr/share/emacs/site-lisp/elpa-src/seq-2.24/seq-autoloads
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq hides /usr/share/emacs/site-lisp/elpa-src/seq-2.24/seq
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq-pkg hides /usr/share/emacs/site-lisp/elpa-src/seq-2.24/seq-pkg
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq-25 hides /usr/share/emacs/site-lisp/elpa-src/seq-2.24/seq-25
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-project-follow-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-project-follow-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-dom hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-dom
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-follow-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-follow-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-logging hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-logging
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-autoloads hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-autoloads
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-compatibility hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-compatibility
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-async hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-async
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-themes hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-themes
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-mouse-interface hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-mouse-interface
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-tag-follow-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-tag-follow-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-bookmarks hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-bookmarks
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-fringe-indicator hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-fringe-indicator
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-filewatch-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-filewatch-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-interface hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-interface
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-tags hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-tags
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-visuals hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-visuals
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-core-utils hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-core-utils
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-peek-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-peek-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-annotations hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-annotations
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-icons hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-icons
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-persistence hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-persistence
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-diagnostics hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-diagnostics
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-scope hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-scope
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-extensions hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-extensions
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-rendering hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-rendering
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-customization hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-customization
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-file-management hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-file-management
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-faces hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-faces
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-macros hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-macros
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-git-commit-diff-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-git-commit-diff-mode
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-treelib hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-treelib
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-hydras hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-hydras
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-pkg hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-pkg
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-header-line hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-header-line
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-icons-dired hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-icons-dired
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-workspaces hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-workspaces
/usr/share/emacs/site-lisp/elpa/treemacs-3.1/treemacs-mode hides /usr/share/emacs/site-lisp/elpa-src/treemacs-3.1/treemacs-mode
/usr/share/emacs/site-lisp/elpa/treemacs-magit-3.1/treemacs-magit hides /usr/share/emacs/site-lisp/elpa-src/treemacs-magit-3.1/treemacs-magit
/usr/share/emacs/site-lisp/elpa/treemacs-magit-3.1/treemacs-magit-pkg hides /usr/share/emacs/site-lisp/elpa-src/treemacs-magit-3.1/treemacs-magit-pkg
/usr/share/emacs/site-lisp/elpa/treemacs-magit-3.1/treemacs-magit-autoloads hides /usr/share/emacs/site-lisp/elpa-src/treemacs-magit-3.1/treemacs-magit-autoloads
/usr/share/emacs/site-lisp/elpa/treemacs-projectile-3.1/treemacs-projectile-pkg hides /usr/share/emacs/site-lisp/elpa-src/treemacs-projectile-3.1/treemacs-projectile-pkg
/usr/share/emacs/site-lisp/elpa/treemacs-projectile-3.1/treemacs-projectile-autoloads hides /usr/share/emacs/site-lisp/elpa-src/treemacs-projectile-3.1/treemacs-projectile-autoloads
/usr/share/emacs/site-lisp/elpa/treemacs-projectile-3.1/treemacs-projectile hides /usr/share/emacs/site-lisp/elpa-src/treemacs-projectile-3.1/treemacs-projectile
/usr/share/emacs/site-lisp/elpa/vterm-0.0.2/vterm-load-path hides /usr/share/emacs/site-lisp/elpa-src/vterm-0.0.2/vterm-load-path
/usr/share/emacs/site-lisp/elpa/vterm-0.0.2/vterm hides /usr/share/emacs/site-lisp/elpa-src/vterm-0.0.2/vterm
/usr/share/emacs/site-lisp/elpa/vterm-0.0.2/vterm-pkg hides /usr/share/emacs/site-lisp/elpa-src/vterm-0.0.2/vterm-pkg
/usr/share/emacs/site-lisp/elpa/vterm-0.0.2/vterm-autoloads hides /usr/share/emacs/site-lisp/elpa-src/vterm-0.0.2/vterm-autoloads
/usr/share/emacs/site-lisp/elpa/web-mode-17.3.13/web-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/web-mode-17.3.13/web-mode-pkg
/usr/share/emacs/site-lisp/elpa/web-mode-17.3.13/web-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/web-mode-17.3.13/web-mode-autoloads
/usr/share/emacs/site-lisp/elpa/web-mode-17.3.13/web-mode hides /usr/share/emacs/site-lisp/elpa-src/web-mode-17.3.13/web-mode
/usr/share/emacs/site-lisp/elpa/with-editor-3.3.2/with-editor-pkg hides /usr/share/emacs/site-lisp/elpa-src/with-editor-3.3.2/with-editor-pkg
/usr/share/emacs/site-lisp/elpa/with-editor-3.3.2/with-editor-autoloads hides /usr/share/emacs/site-lisp/elpa-src/with-editor-3.3.2/with-editor-autoloads
/usr/share/emacs/site-lisp/elpa/with-editor-3.3.2/with-editor hides /usr/share/emacs/site-lisp/elpa-src/with-editor-3.3.2/with-editor
/usr/share/emacs/site-lisp/elpa/xml-rpc-1.6.17/xml-rpc hides /usr/share/emacs/site-lisp/elpa-src/xml-rpc-1.6.17/xml-rpc
/usr/share/emacs/site-lisp/elpa/xml-rpc-1.6.17/xml-rpc-pkg hides /usr/share/emacs/site-lisp/elpa-src/xml-rpc-1.6.17/xml-rpc-pkg
/usr/share/emacs/site-lisp/elpa/xml-rpc-1.6.17/xml-rpc-autoloads hides /usr/share/emacs/site-lisp/elpa-src/xml-rpc-1.6.17/xml-rpc-autoloads
/usr/share/emacs/site-lisp/elpa/yaml-mode-0.0.16/yaml-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/yaml-mode-0.0.16/yaml-mode-pkg
/usr/share/emacs/site-lisp/elpa/yaml-mode-0.0.16/yaml-mode hides /usr/share/emacs/site-lisp/elpa-src/yaml-mode-0.0.16/yaml-mode
/usr/share/emacs/site-lisp/elpa/yaml-mode-0.0.16/yaml-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/yaml-mode-0.0.16/yaml-mode-autoloads
/usr/share/emacs/site-lisp/elpa/yasnippet-0.14.1/yasnippet-autoloads hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-0.14.1/yasnippet-autoloads
/usr/share/emacs/site-lisp/elpa/yasnippet-0.14.1/yasnippet-pkg hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-0.14.1/yasnippet-pkg
/usr/share/emacs/site-lisp/elpa/yasnippet-0.14.1/yasnippet hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-0.14.1/yasnippet
/usr/share/emacs/site-lisp/elpa/yasnippet-snippets-20220713/yasnippet-snippets hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-snippets-20220713/yasnippet-snippets
/usr/share/emacs/site-lisp/elpa/yasnippet-snippets-20220713/yasnippet-snippets-pkg hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-snippets-20220713/yasnippet-snippets-pkg
/usr/share/emacs/site-lisp/elpa/yasnippet-snippets-20220713/yasnippet-snippets-autoloads hides /usr/share/emacs/site-lisp/elpa-src/yasnippet-snippets-20220713/yasnippet-snippets-autoloads
/usr/share/emacs/site-lisp/elpa/zenburn-theme-2.8.0/zenburn-theme hides /usr/share/emacs/site-lisp/elpa-src/zenburn-theme-2.8.0/zenburn-theme
/usr/share/emacs/site-lisp/elpa/zenburn-theme-2.8.0/zenburn-theme-pkg hides /usr/share/emacs/site-lisp/elpa-src/zenburn-theme-2.8.0/zenburn-theme-pkg
/usr/share/emacs/site-lisp/elpa/zenburn-theme-2.8.0/zenburn-theme-autoloads hides /usr/share/emacs/site-lisp/elpa-src/zenburn-theme-2.8.0/zenburn-theme-autoloads
/usr/share/emacs/site-lisp/elpa/seq-2.24/seq hides /usr/share/emacs/29.4/lisp/emacs-lisp/seq

Features:
(shadow emacsbug novice debian-copyright shr-color json-ts-mode
git-rebase reporter debian-bts-control vterm tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat term ehelp vterm-module
debian-changelog-mode debian-bug conf-mode make-mode goto-addr misearch
multi-isearch url-queue magit-extras magit-bookmark magit-submodule
magit-obsolete magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func magit-diff
git-commit log-edit add-log magit-core magit-autorevert magit-margin
magit-transient magit-process with-editor shell magit-mode transient
magit-git magit-section magit-utils crm smerge-mode diff mailalias
face-remap dired-aux mm-archive qp sort gnus-cite mail-extr textsec
uni-scripts idna-mapping ucs-normalize uni-confusable textsec-check
gnus-async gnus-bcklg gnus-ml gnus-topic cursor-sensor timezone utf-7
url-cache nnfolder gnus-demon nnml ezgnus gnus-delay gnus-draft
gnus-agent gnus-srvr gnus-score score-mode nnvirtual nntp gnus-cache
nndraft nnmh auth-source-xoauth2-plugin oauth2 url-http url-auth url-gw
plstore eglot external-completion array jsonrpc ert ewoc debug backtrace
xref pcase imenu matlab matlab-scan matlab-syntax matlab-compat
jka-compr mu4e mu4e-org org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-src ob-comint org-pcomplete pcomplete org-list
org-footnote org-faces org-entities noutline outline ob-emacs-lisp
ob-core ob-eval org-cycle org-table ol org-fold org-fold-core org-keys
oc org-loaddefs find-func org-version org-compat org-macs format-spec
mu4e-notification notifications mu4e-main smtpmail mu4e-view
mu4e-mime-parts cal-menu calendar cal-loaddefs mu4e-headers mu4e-thread
mu4e-actions mu4e-compose mu4e-draft gnus-msg gnus-art mm-uu mml2015
mm-view mml-smime smime dig gnus-sum gnus-group gnus-undo gnus-start
gnus-dbus dbus gnus-cloud nnimap nnmail mail-source utf7 nnoo gnus-spec
gnus-int gnus-range gnus-win gnus nnheader range mu4e-search mu4e-lists
mu4e-bookmarks mu4e-mark mu4e-message shr pixel-fill kinsoku url-file
svg xml dom flow-fill mule-util mu4e-contacts mu4e-update mu4e-folders
mu4e-context mu4e-query-items mu4e-server mu4e-modeline mu4e-vars
mu4e-helpers mu4e-config mu4e-window ido message sendmail yank-media
rfc822 mml mml-sec gnus-util mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader mu4e-obsolete windmove flyspell ispell
gnutls network-stream puny nsm epa-file epa derived epg rfc6068
epg-config rcirc parse-time iso8601 time-date term/xterm xterm comp
comp-cstr server cap-words superword subword vc-hg vc-git diff-mode
vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view pcvs-util vc
vc-dispatcher bug-reference disp-table whitespace yasnippet-snippets
yasnippet cus-edit cus-start wid-edit init mu4e-debian-hx90
zenburn-theme xclip treesit-auto treesit treemacs-project-follow-mode
treemacs-follow-mode treemacs-rendering treemacs-annotations
treemacs-async treemacs-visuals treemacs-fringe-indicator pulse color
treemacs-workspaces treemacs-dom treemacs-icons treemacs-themes
treemacs-scope treemacs-core-utils treemacs-logging
treemacs-customization pfuture inline ht s hl-line dash
keychain-environment exec-path-from-shell corfu-terminal popon
corfu-popupinfo corfu-echo corfu compat activities-tabs activities
persist bookmark pp edmacro kmacro advice icomplete cus-load
flymake-proc flymake project compile text-property-search comint
ansi-osc ansi-color ring warnings icons thingatpt cl-extra help-mode
use-package use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core
display-line-numbers autorevert filenotify
keychain-environment-autoloads treesit-auto-autoloads xclip-autoloads rx
info debian-el dired dired-loaddefs finder-inf package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 1385704 157406)
 (symbols 48 42601 43)
 (strings 32 181071 16554)
 (string-bytes 1 5513742)
 (vectors 16 115417)
 (vector-slots 8 2872186 223315)
 (floats 8 1025 2197)
 (intervals 56 42184 12669)
 (buffers 984 116))

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2055 bytes --]

From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/5] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-parameters-required-by-Google-OAuth2-to-get-refr.patch --]
[-- Type: text/x-diff, Size: 1264 bytes --]

From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:52:02 -0700
Subject: [PATCH 2/5] Add parameters required by Google OAuth2 to get
 refresh_token

* packages/oauth2/oauth2.el (oauth2-request-authorization): add
`access_type=offline' and `prompt=consent' when requesting token to
receive refresh_token.
---
 oauth2.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 3a3e50ad2b..9780ac3a1d 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -63,7 +63,9 @@ It returns the code provided by the service."
                      "&response_type=code"
                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+                     (if state (concat "&state=" (url-hexify-string state)) "")
+                     "&access_type=offline"
+                     "&prompt=consent")))
     (browse-url url)
     (read-string (concat "Follow the instruction on your default browser, or "
                          "visit:\n" url
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Encode-parameters-when-requesting-access.patch --]
[-- Type: text/x-diff, Size: 1194 bytes --]

From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:54:08 -0700
Subject: [PATCH 3/5] Encode parameters when requesting access

* packages/oauth2/oauth2.el (oauth2-request-access): encode all
parameters which may contain characters that breaks URL.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 9780ac3a1d..b035742fc1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
            (oauth2-make-access-request
             token-url
             (concat
-             "client_id=" client-id
+             "client_id=" (url-hexify-string client-id)
 	     (when client-secret
-               (concat  "&client_secret=" client-secret))
-             "&code=" code
+               (concat  "&client_secret=" (url-hexify-string client-secret)))
+             "&code=" (url-hexify-string code)
              "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
              "&grant_type=authorization_code"))))
       (make-oauth2-token :client-id client-id
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Support-storing-data-for-multiple-accounts-of-the-sa.patch --]
[-- Type: text/x-diff, Size: 1987 bytes --]

From eb4afbc2f21c65417de5ca8e06309fd4c53cc1ab Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:00:04 -0700
Subject: [PATCH 4/5] Support storing data for multiple accounts of the same
 provider

Currently the plstore id computed by `oauth2-compute-id' only takes
`auth-url', `token-url', and `scope' into account, which could be the
same for the same provider (e.g. Gmail).  This prevents storing
information for multiple accounts of the same service for some
providers.

This patch adds `client-id' to the calculation of plstore id to make
sure that it is unique for different accounts of the same provider.

* packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
parameter of `oauth2-compute-id' to ensure unique id amount multiple
accounts of the same provider.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index b035742fc1..ea012d4d38 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
   :group 'oauth2
   :type 'file)
 
-(defun oauth2-compute-id (auth-url token-url scope)
+(defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
 This allows to store the token in an unique way."
-  (secure-hash 'md5 (concat auth-url token-url scope)))
+  (secure-hash 'md5 (concat auth-url token-url scope client-id)))
 
 ;;;###autoload
 (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
   "Request access to a resource and store it using `plstore'."
   ;; We store a MD5 sum of all URL
   (let* ((plstore (plstore-open oauth2-token-file))
-         (id (oauth2-compute-id auth-url token-url scope))
+         (id (oauth2-compute-id auth-url token-url scope client-id))
          (plist (cdr (plstore-get plstore id))))
     ;; Check if we found something matching this access
     (if plist
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-debug-messages-and-provide-a-switch-for-enabling.patch --]
[-- Type: text/x-diff, Size: 2174 bytes --]

From 6cf0ab190d24b6fc2a7335d02a8aef29ab24c622 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/5] Add debug messages and provide a switch for enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add debug message and option for
enabling.
---
 oauth2.el | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/oauth2.el b/oauth2.el
index ea012d4d38..06a880532e 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -40,6 +40,7 @@
 (require 'plstore)
 (require 'json)
 (require 'url-http)
+(require 'pp)
 
 (defvar url-http-data)
 (defvar url-http-method)
@@ -53,6 +54,15 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defcustom oauth2-debug nil
+  "Enable debug messages."
+  :type 'boolean)
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (if oauth2-debug
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -79,6 +89,8 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
+  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
+  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
   (let ((url-request-method "POST")
         (url-request-data data)
         (url-request-extra-headers
@@ -86,6 +98,8 @@ It returns the code provided by the service."
     (with-current-buffer (url-retrieve-synchronously url)
       (let ((data (oauth2-request-access-parse)))
         (kill-buffer (current-buffer))
+        (oauth2--do-debug "oauth2-make-access-request: response: %s"
+                          (pp-to-string data))
         data))))
 
 (cl-defstruct oauth2-token
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Robert Pluim]

>>>>> On Mon, 29 Jul 2024 14:25:01 -0700, Xiyue Deng <manphiz@gmail.com> said:

    Xiyue> Hi,
    Xiyue> I have been trying out using oauth2.el to enable OAuth2-based
    Xiyue> authentication for email service providers and had some success for
    Xiyue> Gmail.  During this process, I have made a few changes to oauth2.el that
    Xiyue> enables it to use with Gmail OAuth2 as well as some usability and
    Xiyue> debugging improvements, which I'm sharing below.

Thank you for this. This support is becoming more necessary as time
goes on. I even wonder if we should bring oauth2.el into emacs instead
of it being a package.

    Xiyue> This is a series of five patches, which are attached.

    Xiyue> The first patch shows the authentication URL in the minibuffer window
    Xiyue> alongside the prompt accepting the authorization code.  This helps when
    Xiyue> a user has multiple accounts from the same provider but is logged into a
    Xiyue> different account than the one that the user is trying to set up.  If
    Xiyue> the user use the link (or through `browse-url') it will use the active
    Xiyue> account instead of the one intended.  By showing the URL in the
    Xiyue> minibuffer, the user can choose other ways to get the authorization code
    Xiyue> (e.g. using another browser, using private/encognito mode, etc.)

OK. This fixes one of my irritations with oauth2.el 🙂

    Xiyue> The second patch adds the parameters `access_type=offline' and
    Xiyue> `prompt=consent' to the authorization URL, which is required for Gmail
    Xiyue> OAuth2 to get the refresh token.  Without these 2 parameters, Gmail
    Xiyue> response will only contain the access token which expires in one hour.
    Xiyue> They should also be compatible with other OAuth2 authentication process.
    Xiyue> (Though I am currently having trouble to get outlook.com to work
    Xiyue> regardless of these parameters, which I'll ask in a separate thread.)

    Xiyue> Note that the second patch depends on the first patch as they modify the same
    Xiyue> part of the code.

OK. Iʼm assuming oauth2.el can use the refresh token next time it
needs to authorize? (Iʼve been avoiding actually using oauth2.el in
anger, since app passwords still work)

    Xiyue> The third patch encodes the parameters for requesting refreshing access
    Xiyue> token, which is recommended because the client secret and other
    Xiyue> parameters may contain characters that may break parameter parsing.

OK

    Xiyue> The fourth patch may need a bit of background: oauth2.el (optionally)
    Xiyue> uses plstore to save authentication data for future reuse, and the
    Xiyue> plstore id for an account is computed using a combination of `auth-url',
    Xiyue> `token-url', and `scope'.  However, this combination of data doesn't
    Xiyue> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
    Xiyue> the three parameters are the same for different accounts, and hence
    Xiyue> storing a second account information will override the first one.

    Xiyue> This fourth patch adds `client-id' to the calculation of plstore id to
    Xiyue> ensure its uniqueness.  This may cause a few concerns:

    Xiyue> - This will invalidate all existing entries and a user will have to redo
    Xiyue>   the authorization process again to get a new refresh token.  However,
    Xiyue>   I think it's more important to ensure that oauth2.el works correctly
    Xiyue>   for multiple accounts of the same provider, or a user may suffer from
    Xiyue>   confusion when adding a new account invalidates a previous account.

I donʼt think thatʼs too big a concern. 'modern' authentication flows
regularly re-prompt, so this will not be too surprising (although
maybe call it out in the packageʼs NEWS or README).

    Xiyue> - Adding `client-id' to the calculation of plstore id may provoke
    Xiyue>   suspicion of leaking it as the hash calculation uses md5.  In most
    Xiyue>   cases, requesting a refresh token requires both `client-id' and
    Xiyue>   `client-secret', so without including the latter it should be safe.
    Xiyue>   There are cases when requesting only the access token may work with
    Xiyue>   `client-id' along.  Still, I think this should not be a big concern as
    Xiyue>   the data is combined with `auth-url', `token-url', and `scope' which
    Xiyue>   provides sufficient salt.  Alternatively, we can also choose to use a
    Xiyue>   more secure hash function, e.g. SHA2 or better, given that existing
    Xiyue>   entries will be invalidated anyway.

If the existing entries are going to become invalid anyway, you might
as well take the opportunity to move away from md5 at the same
time. git picked SHA-256, but that was a while ago, so maybe SHA-512?

    Xiyue> The fifth patch adds debug messages when doing a URL query which records
    Xiyue> the request URL, the request data, and the response data, and provide a
    Xiyue> custom variable to enable this.  This provides a way to help debugging
    Xiyue> the requests, and I find it handy when testing oauth2 against different
    Xiyue> providers.

OK (although perhaps make it a defvar rather than a defcustom, to
avoid people accidentally enabling it).

Robert
-- 

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Robert Pluim <rpluim@gmail.com> writes:

>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>     Xiyue>   the authorization process again to get a new refresh token.  However,
>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>     Xiyue>   confusion when adding a new account invalidates a previous account.
>
> I donʼt think thatʼs too big a concern. 'modern' authentication flows
> regularly re-prompt, so this will not be too surprising (although
> maybe call it out in the packageʼs NEWS or README).

In many cases the refreshing of tokens is transparent to the user there
doesn't have to be a re-prompt to refresh the token if the OAuth
provider support it.
Micrsofts OAuth workflow is quite good in this regard as there's a
non-standard error to indicate when the user has to re-authorize the
application.

I assume all implementation of OAuth have their quirks.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 6343 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Mon, 29 Jul 2024 14:25:01 -0700, Xiyue Deng <manphiz@gmail.com> said:
>
>     Xiyue> Hi,
>     Xiyue> I have been trying out using oauth2.el to enable OAuth2-based
>     Xiyue> authentication for email service providers and had some success for
>     Xiyue> Gmail.  During this process, I have made a few changes to oauth2.el that
>     Xiyue> enables it to use with Gmail OAuth2 as well as some usability and
>     Xiyue> debugging improvements, which I'm sharing below.
>
> Thank you for this. This support is becoming more necessary as time
> goes on. I even wonder if we should bring oauth2.el into emacs instead
> of it being a package.
>

I would also love that to happen.  In fact, I have another addon
inspired by auth-source-xoauth2 and makes use of oauth2.el and
auth-source backend, though it requires advising the
auth-source-search-backends and kind of awkward.  I'll post that through
another bug report for comments after this one gets submitted.

>     Xiyue> This is a series of five patches, which are attached.
>
>     Xiyue> The first patch shows the authentication URL in the minibuffer window
>     Xiyue> alongside the prompt accepting the authorization code.  This helps when
>     Xiyue> a user has multiple accounts from the same provider but is logged into a
>     Xiyue> different account than the one that the user is trying to set up.  If
>     Xiyue> the user use the link (or through `browse-url') it will use the active
>     Xiyue> account instead of the one intended.  By showing the URL in the
>     Xiyue> minibuffer, the user can choose other ways to get the authorization code
>     Xiyue> (e.g. using another browser, using private/encognito mode, etc.)
>
> OK. This fixes one of my irritations with oauth2.el 🙂
>

Glad to hear I'm not the only one.

>     Xiyue> The second patch adds the parameters `access_type=offline' and
>     Xiyue> `prompt=consent' to the authorization URL, which is required for Gmail
>     Xiyue> OAuth2 to get the refresh token.  Without these 2 parameters, Gmail
>     Xiyue> response will only contain the access token which expires in one hour.
>     Xiyue> They should also be compatible with other OAuth2 authentication process.
>     Xiyue> (Though I am currently having trouble to get outlook.com to work
>     Xiyue> regardless of these parameters, which I'll ask in a separate thread.)
>
>     Xiyue> Note that the second patch depends on the first patch as they modify the same
>     Xiyue> part of the code.
>
> OK. Iʼm assuming oauth2.el can use the refresh token next time it
> needs to authorize? (Iʼve been avoiding actually using oauth2.el in
> anger, since app passwords still work)
>

Yes, with a valid refresh token you can login without any manual steps.
Unless the refresh_token itself is expired, in which case you'll have to
re-authorize.

>     Xiyue> The third patch encodes the parameters for requesting refreshing access
>     Xiyue> token, which is recommended because the client secret and other
>     Xiyue> parameters may contain characters that may break parameter parsing.
>
> OK
>
>     Xiyue> The fourth patch may need a bit of background: oauth2.el (optionally)
>     Xiyue> uses plstore to save authentication data for future reuse, and the
>     Xiyue> plstore id for an account is computed using a combination of `auth-url',
>     Xiyue> `token-url', and `scope'.  However, this combination of data doesn't
>     Xiyue> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>     Xiyue> the three parameters are the same for different accounts, and hence
>     Xiyue> storing a second account information will override the first one.
>
>     Xiyue> This fourth patch adds `client-id' to the calculation of plstore id to
>     Xiyue> ensure its uniqueness.  This may cause a few concerns:
>
>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>     Xiyue>   the authorization process again to get a new refresh token.  However,
>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>     Xiyue>   confusion when adding a new account invalidates a previous account.
>
> I donʼt think thatʼs too big a concern. 'modern' authentication flows
> regularly re-prompt, so this will not be too surprising (although
> maybe call it out in the packageʼs NEWS or README).
>

I have added a NEWS file in patch 6 documenting this for 0.17 (which
should be the version of the next release.)

>     Xiyue> - Adding `client-id' to the calculation of plstore id may provoke
>     Xiyue>   suspicion of leaking it as the hash calculation uses md5.  In most
>     Xiyue>   cases, requesting a refresh token requires both `client-id' and
>     Xiyue>   `client-secret', so without including the latter it should be safe.
>     Xiyue>   There are cases when requesting only the access token may work with
>     Xiyue>   `client-id' along.  Still, I think this should not be a big concern as
>     Xiyue>   the data is combined with `auth-url', `token-url', and `scope' which
>     Xiyue>   provides sufficient salt.  Alternatively, we can also choose to use a
>     Xiyue>   more secure hash function, e.g. SHA2 or better, given that existing
>     Xiyue>   entries will be invalidated anyway.
>
> If the existing entries are going to become invalid anyway, you might
> as well take the opportunity to move away from md5 at the same
> time. git picked SHA-256, but that was a while ago, so maybe SHA-512?
>

Makes sense.  I have updated the hash function to use SHA-512 in patch 5.

>     Xiyue> The fifth patch adds debug messages when doing a URL query which records
>     Xiyue> the request URL, the request data, and the response data, and provide a
>     Xiyue> custom variable to enable this.  This provides a way to help debugging
>     Xiyue> the requests, and I find it handy when testing oauth2 against different
>     Xiyue> providers.
>
> OK (although perhaps make it a defvar rather than a defcustom, to
> avoid people accidentally enabling it).
>

Done also in patch 5.

> Robert

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2055 bytes --]

From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-parameters-required-by-Google-OAuth2-to-get-refr.patch --]
[-- Type: text/x-diff, Size: 1264 bytes --]

From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:52:02 -0700
Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
 refresh_token

* packages/oauth2/oauth2.el (oauth2-request-authorization): add
`access_type=offline' and `prompt=consent' when requesting token to
receive refresh_token.
---
 oauth2.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 3a3e50ad2b..9780ac3a1d 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -63,7 +63,9 @@ It returns the code provided by the service."
                      "&response_type=code"
                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+                     (if state (concat "&state=" (url-hexify-string state)) "")
+                     "&access_type=offline"
+                     "&prompt=consent")))
     (browse-url url)
     (read-string (concat "Follow the instruction on your default browser, or "
                          "visit:\n" url
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Encode-parameters-when-requesting-access.patch --]
[-- Type: text/x-diff, Size: 1194 bytes --]

From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:54:08 -0700
Subject: [PATCH 3/6] Encode parameters when requesting access

* packages/oauth2/oauth2.el (oauth2-request-access): encode all
parameters which may contain characters that breaks URL.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 9780ac3a1d..b035742fc1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
            (oauth2-make-access-request
             token-url
             (concat
-             "client_id=" client-id
+             "client_id=" (url-hexify-string client-id)
 	     (when client-secret
-               (concat  "&client_secret=" client-secret))
-             "&code=" code
+               (concat  "&client_secret=" (url-hexify-string client-secret)))
+             "&code=" (url-hexify-string code)
              "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
              "&grant_type=authorization_code"))))
       (make-oauth2-token :client-id client-id
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Support-storing-data-for-multiple-accounts-of-the-sa.patch --]
[-- Type: text/x-diff, Size: 2090 bytes --]

From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:00:04 -0700
Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
 provider

Currently the plstore id computed by `oauth2-compute-id' only takes
`auth-url', `token-url', and `scope' into account, which could be the
same for the same provider (e.g. Gmail).  This prevents storing
information for multiple accounts of the same service for some
providers.

This patch adds `client-id' to the calculation of plstore id to make
sure that it is unique for different accounts of the same provider.

It also changes the hash function to sha512 to be more secure.

* packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
parameter of `oauth2-compute-id' to ensure unique id amount multiple
accounts of the same provider, and change hash function to sha512.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index b035742fc1..035971ac85 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
   :group 'oauth2
   :type 'file)
 
-(defun oauth2-compute-id (auth-url token-url scope)
+(defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
 This allows to store the token in an unique way."
-  (secure-hash 'md5 (concat auth-url token-url scope)))
+  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
 
 ;;;###autoload
 (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
   "Request access to a resource and store it using `plstore'."
   ;; We store a MD5 sum of all URL
   (let* ((plstore (plstore-open oauth2-token-file))
-         (id (oauth2-compute-id auth-url token-url scope))
+         (id (oauth2-compute-id auth-url token-url scope client-id))
          (plist (cdr (plstore-get plstore id))))
     ;; Check if we found something matching this access
     (if plist
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-debug-messages-and-provide-a-switch-variable-for.patch --]
[-- Type: text/x-diff, Size: 2751 bytes --]

From 08b04a07aa399bc756f1a8a00ae17cf333be02d9 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
 enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add support for debug messages and a
switch variable for enabling.
---
 oauth2.el | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 035971ac85..aca48f9ce1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -40,6 +40,7 @@
 (require 'plstore)
 (require 'json)
 (require 'url-http)
+(require 'pp)
 
 (defvar url-http-data)
 (defvar url-http-method)
@@ -53,6 +54,15 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defcustom oauth2-debug nil
+  "Enable debug messages."
+  :type 'boolean)
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (if oauth2-debug
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -79,6 +89,8 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
+  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
+  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
   (let ((url-request-method "POST")
         (url-request-data data)
         (url-request-extra-headers
@@ -86,6 +98,8 @@ It returns the code provided by the service."
     (with-current-buffer (url-retrieve-synchronously url)
       (let ((data (oauth2-request-access-parse)))
         (kill-buffer (current-buffer))
+        (oauth2--do-debug "oauth2-make-access-request: response: %s"
+                          (pp-to-string data))
         data))))
 
 (cl-defstruct oauth2-token
@@ -158,10 +172,8 @@ TOKEN should be obtained with `oauth2-request-access'."
     auth-url client-id scope state redirect-uri)
    redirect-uri))
 
-(defcustom oauth2-token-file (concat user-emacs-directory "oauth2.plstore")
-  "File path where store OAuth tokens."
-  :group 'oauth2
-  :type 'file)
+(defvar oauth2-token-file (concat user-emacs-directory "oauth2.plstore")
+  "File path where store OAuth tokens.")
 
 (defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-Add-NEWS-file-to-document-the-changes-to-plstore-id-.patch --]
[-- Type: text/x-diff, Size: 1350 bytes --]

From 2fbc85ee128cfdf7a1521bbc9554424e9ba510da Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Tue, 30 Jul 2024 03:46:57 -0700
Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
 generation

---
 NEWS | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100644 NEWS

diff --git a/NEWS b/NEWS
new file mode 100644
index 0000000000..6715a1914a
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,23 @@
+Summary of changes to oauth2.el
+-------------------------------
+
+For changes of 0.16 and older or full changes please check the git
+history of the repository of oauth2.el.
+
+* 0.17
+
+** Changes to plstore id generation and needs to reacquire refresh_token
+
+The generation of plstore id used to include `auth-url', `token-url',
+and `scope'.  Now `client-id' is also included.  This is required to
+support multiple accounts of some providers which use the same
+`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
+generated plstore id is not unique amount accounts.  Adding
+`client-id' solves this problem.
+
+The hash function of calculating the plstore id has also changed from
+MD5 to SHA512 to be more secure.
+
+As a result, users of oauth2.el will need to redo the authentication
+process to get a new refresh_token when upgrading from older version
+to 0.17.
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Robert Pluim]

>>>>> On Tue, 30 Jul 2024 12:37:05 -0700, Xiyue Deng <manphiz@gmail.com> said:
    Xiyue> The fifth patch adds debug messages when doing a URL query which records
    Xiyue> the request URL, the request data, and the response data, and provide a
    Xiyue> custom variable to enable this.  This provides a way to help debugging
    Xiyue> the requests, and I find it handy when testing oauth2 against different
    Xiyue> providers.
    >> 
    >> OK (although perhaps make it a defvar rather than a defcustom, to
    >> avoid people accidentally enabling it).
    >> 

    Xiyue> Done also in patch 5.

I see you changed `oauth2-token-file' to a `defvar', which I donʼt
think you should do. I was talking about making `oauth2-debug' a
defvar.

Robert
-- 

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 30 Jul 2024 12:37:05 -0700, Xiyue Deng <manphiz@gmail.com> said:
>     Xiyue> The fifth patch adds debug messages when doing a URL query which records
>     Xiyue> the request URL, the request data, and the response data, and provide a
>     Xiyue> custom variable to enable this.  This provides a way to help debugging
>     Xiyue> the requests, and I find it handy when testing oauth2 against different
>     Xiyue> providers.
>     >> 
>     >> OK (although perhaps make it a defvar rather than a defcustom, to
>     >> avoid people accidentally enabling it).
>     >> 
>
>     Xiyue> Done also in patch 5.
>
> I see you changed `oauth2-token-file' to a `defvar', which I donʼt
> think you should do. I was talking about making `oauth2-debug' a
> defvar.
>
> Robert

Oops.  Another reminder not to write code after 2am.

The fixed patch 5 is attached.

-- 
Xiyue Deng


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0005-Add-debug-messages-and-provide-a-switch-variable-for.patch --]
[-- Type: text/x-diff, Size: 2186 bytes --]

From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
 enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add support for debug messages and a
switch variable for enabling.
---
 oauth2.el | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/oauth2.el b/oauth2.el
index 035971ac85..ce7a835100 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -40,6 +40,7 @@
 (require 'plstore)
 (require 'json)
 (require 'url-http)
+(require 'pp)
 
 (defvar url-http-data)
 (defvar url-http-method)
@@ -53,6 +54,14 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defvar oauth2-debug nil
+  "Enable debug messages.")
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (if oauth2-debug
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -79,6 +88,8 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
+  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
+  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
   (let ((url-request-method "POST")
         (url-request-data data)
         (url-request-extra-headers
@@ -86,6 +97,8 @@ It returns the code provided by the service."
     (with-current-buffer (url-retrieve-synchronously url)
       (let ((data (oauth2-request-access-parse)))
         (kill-buffer (current-buffer))
+        (oauth2--do-debug "oauth2-make-access-request: response: %s"
+                          (pp-to-string data))
         data))))
 
 (cl-defstruct oauth2-token
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>>>>>>> On Tue, 30 Jul 2024 12:37:05 -0700, Xiyue Deng <manphiz@gmail.com> said:
>>     Xiyue> The fifth patch adds debug messages when doing a URL query which records
>>     Xiyue> the request URL, the request data, and the response data, and provide a
>>     Xiyue> custom variable to enable this.  This provides a way to help debugging
>>     Xiyue> the requests, and I find it handy when testing oauth2 against different
>>     Xiyue> providers.
>>     >> 
>>     >> OK (although perhaps make it a defvar rather than a defcustom, to
>>     >> avoid people accidentally enabling it).
>>     >> 
>>
>>     Xiyue> Done also in patch 5.
>>
>> I see you changed `oauth2-token-file' to a `defvar', which I donʼt
>> think you should do. I was talking about making `oauth2-debug' a
>> defvar.
>>
>> Robert
>
> Oops.  Another reminder not to write code after 2am.
>
> The fixed patch 5 is attached.

More reviews welcome!

BTW, is there any dev available to commit the changes once it's in a
good shape?

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Robert Pluim]

>>>>> On Fri, 02 Aug 2024 01:15:22 -0700, Xiyue Deng <manphiz@gmail.com> said:

    Xiyue> BTW, is there any dev available to commit the changes once it's in a
    Xiyue> good shape?

You could request commit access. I see you have a number of
contributions to Emacs already.

Robert
-- 

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Hi Robert,

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Fri, 02 Aug 2024 01:15:22 -0700, Xiyue Deng <manphiz@gmail.com> said:
>
>     Xiyue> BTW, is there any dev available to commit the changes once it's in a
>     Xiyue> good shape?
>
> You could request commit access. I see you have a number of
> contributions to Emacs already.
>
> Robert

That would be great to have.  Do you know how I can apply for it (I have
already done the paperwork)?

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Eli Zaretskii]

> Cc: 72358@debbugs.gnu.org
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Fri, 02 Aug 2024 01:15:22 -0700
> 
> BTW, is there any dev available to commit the changes once it's in a
> good shape?

All the 3 co-maintainers are tracking these discussions and install
changes that are ready to be installed.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 72358@debbugs.gnu.org
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Fri, 02 Aug 2024 01:15:22 -0700
>> 
>> BTW, is there any dev available to commit the changes once it's in a
>> good shape?
>
> All the 3 co-maintainers are tracking these discussions and install
> changes that are ready to be installed.

That's good to know!  Thanks Eli!
-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 72358@debbugs.gnu.org
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Fri, 02 Aug 2024 01:15:22 -0700
>> 
>> BTW, is there any dev available to commit the changes once it's in a
>> good shape?
>
> All the 3 co-maintainers are tracking these discussions and install
> changes that are ready to be installed.

It's been a few days since the last time I received feedback for
improvements regarding my patches.  Is there any other feedbacks/reviews
I am expecting from the co-maintainers?  Please also let me know when
it's time to ask for merging and requesting a new tagged release.

Thanks in advance.
-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Eli Zaretskii]

> From: Xiyue Deng <manphiz@gmail.com>
> Cc: rpluim@gmail.com,  72358@debbugs.gnu.org
> Date: Tue, 13 Aug 2024 15:03:30 -0700
> 
> Hi Eli,
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Cc: 72358@debbugs.gnu.org
> >> From: Xiyue Deng <manphiz@gmail.com>
> >> Date: Fri, 02 Aug 2024 01:15:22 -0700
> >> 
> >> BTW, is there any dev available to commit the changes once it's in a
> >> good shape?
> >
> > All the 3 co-maintainers are tracking these discussions and install
> > changes that are ready to be installed.
> 
> It's been a few days since the last time I received feedback for
> improvements regarding my patches.  Is there any other feedbacks/reviews
> I am expecting from the co-maintainers?  Please also let me know when
> it's time to ask for merging and requesting a new tagged release.

?? The last message in this discussion was just yesterday evening, and
my understanding is that you are still discussing the issues and did
not reach the final conclusion.  If I'm mistaken, my apologies; please
describe your conclusion and post the patch that you-all agree would
solve the issues, and let's take it from there.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Xiyue Deng <manphiz@gmail.com>
>> Cc: rpluim@gmail.com,  72358@debbugs.gnu.org
>> Date: Tue, 13 Aug 2024 15:03:30 -0700
>> 
>> Hi Eli,
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Cc: 72358@debbugs.gnu.org
>> >> From: Xiyue Deng <manphiz@gmail.com>
>> >> Date: Fri, 02 Aug 2024 01:15:22 -0700
>> >> 
>> >> BTW, is there any dev available to commit the changes once it's in a
>> >> good shape?
>> >
>> > All the 3 co-maintainers are tracking these discussions and install
>> > changes that are ready to be installed.
>> 
>> It's been a few days since the last time I received feedback for
>> improvements regarding my patches.  Is there any other feedbacks/reviews
>> I am expecting from the co-maintainers?  Please also let me know when
>> it's time to ask for merging and requesting a new tagged release.
>
> ?? The last message in this discussion was just yesterday evening, and
> my understanding is that you are still discussing the issues and did
> not reach the final conclusion.  If I'm mistaken, my apologies;

The recent communication was not related to my patches but to check
whether it is possible to support outlook.com OAuth2 login (and the
conclusion was no because refreshing access token was disabled as
confirmed by MS representative during an online chat.)

> please describe your conclusion and post the patch that you-all agree
> would solve the issues, and let's take it from there.

I actually only received comments from Robert and I have updated my
patches according in [1][2] (also attached in EOM).

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2055 bytes --]

From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 2595 bytes --]

Xiyue Deng <manphiz@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Cc: rpluim@gmail.com,  72358@debbugs.gnu.org
>>> Date: Tue, 13 Aug 2024 15:03:30 -0700
>>> 
>>> Hi Eli,
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> >> Cc: 72358@debbugs.gnu.org
>>> >> From: Xiyue Deng <manphiz@gmail.com>
>>> >> Date: Fri, 02 Aug 2024 01:15:22 -0700
>>> >> 
>>> >> BTW, is there any dev available to commit the changes once it's in a
>>> >> good shape?
>>> >
>>> > All the 3 co-maintainers are tracking these discussions and install
>>> > changes that are ready to be installed.
>>> 
>>> It's been a few days since the last time I received feedback for
>>> improvements regarding my patches.  Is there any other feedbacks/reviews
>>> I am expecting from the co-maintainers?  Please also let me know when
>>> it's time to ask for merging and requesting a new tagged release.
>>
>> ?? The last message in this discussion was just yesterday evening, and
>> my understanding is that you are still discussing the issues and did
>> not reach the final conclusion.  If I'm mistaken, my apologies;
>
> The recent communication was not related to my patches but to check
> whether it is possible to support outlook.com OAuth2 login (and the
> conclusion was no because refreshing access token was disabled as
> confirmed by MS representative during an online chat.)
>
>> please describe your conclusion and post the patch that you-all agree
>> would solve the issues, and let's take it from there.
>
> I actually only received comments from Robert and I have updated my
> patches according in [1][2] (also attached in EOM).
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44

Sorry I accidentally hit sent for my previous mail (which actually ended
on a full sentence, what are the chances?)

To continue, the description of the patches is in the first message[3]
and not changed much except the added 6th patch which added a NEWS file.
TL;DR the patches include
* adding authorization request parameters required for Gmail OAuth2 to
work,
* updating key calculation in oauth2.plstore to include `client-id' so
that it can store multiple accounts for a given service, and
* usability and debugging improvements.

I haven't received more comments from Robert so I hope the patches now
look good to him.  Still, I would like to invite for more reviews (if
any) and submit the patches when ready.

Thanks!

[3] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#5

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2055 bytes --]

From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-parameters-required-by-Google-OAuth2-to-get-refr.patch --]
[-- Type: text/x-diff, Size: 1264 bytes --]

From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:52:02 -0700
Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
 refresh_token

* packages/oauth2/oauth2.el (oauth2-request-authorization): add
`access_type=offline' and `prompt=consent' when requesting token to
receive refresh_token.
---
 oauth2.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 3a3e50ad2b..9780ac3a1d 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -63,7 +63,9 @@ It returns the code provided by the service."
                      "&response_type=code"
                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+                     (if state (concat "&state=" (url-hexify-string state)) "")
+                     "&access_type=offline"
+                     "&prompt=consent")))
     (browse-url url)
     (read-string (concat "Follow the instruction on your default browser, or "
                          "visit:\n" url
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Encode-parameters-when-requesting-access.patch --]
[-- Type: text/x-diff, Size: 1194 bytes --]

From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:54:08 -0700
Subject: [PATCH 3/6] Encode parameters when requesting access

* packages/oauth2/oauth2.el (oauth2-request-access): encode all
parameters which may contain characters that breaks URL.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 9780ac3a1d..b035742fc1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
            (oauth2-make-access-request
             token-url
             (concat
-             "client_id=" client-id
+             "client_id=" (url-hexify-string client-id)
 	     (when client-secret
-               (concat  "&client_secret=" client-secret))
-             "&code=" code
+               (concat  "&client_secret=" (url-hexify-string client-secret)))
+             "&code=" (url-hexify-string code)
              "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
              "&grant_type=authorization_code"))))
       (make-oauth2-token :client-id client-id
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Support-storing-data-for-multiple-accounts-of-the-sa.patch --]
[-- Type: text/x-diff, Size: 2090 bytes --]

From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:00:04 -0700
Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
 provider

Currently the plstore id computed by `oauth2-compute-id' only takes
`auth-url', `token-url', and `scope' into account, which could be the
same for the same provider (e.g. Gmail).  This prevents storing
information for multiple accounts of the same service for some
providers.

This patch adds `client-id' to the calculation of plstore id to make
sure that it is unique for different accounts of the same provider.

It also changes the hash function to sha512 to be more secure.

* packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
parameter of `oauth2-compute-id' to ensure unique id amount multiple
accounts of the same provider, and change hash function to sha512.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index b035742fc1..035971ac85 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
   :group 'oauth2
   :type 'file)
 
-(defun oauth2-compute-id (auth-url token-url scope)
+(defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
 This allows to store the token in an unique way."
-  (secure-hash 'md5 (concat auth-url token-url scope)))
+  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
 
 ;;;###autoload
 (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
   "Request access to a resource and store it using `plstore'."
   ;; We store a MD5 sum of all URL
   (let* ((plstore (plstore-open oauth2-token-file))
-         (id (oauth2-compute-id auth-url token-url scope))
+         (id (oauth2-compute-id auth-url token-url scope client-id))
          (plist (cdr (plstore-get plstore id))))
     ;; Check if we found something matching this access
     (if plist
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-debug-messages-and-provide-a-switch-variable-for.patch --]
[-- Type: text/x-diff, Size: 2186 bytes --]

From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
 enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add support for debug messages and a
switch variable for enabling.
---
 oauth2.el | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/oauth2.el b/oauth2.el
index 035971ac85..ce7a835100 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -40,6 +40,7 @@
 (require 'plstore)
 (require 'json)
 (require 'url-http)
+(require 'pp)
 
 (defvar url-http-data)
 (defvar url-http-method)
@@ -53,6 +54,14 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defvar oauth2-debug nil
+  "Enable debug messages.")
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (if oauth2-debug
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -79,6 +88,8 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
+  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
+  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
   (let ((url-request-method "POST")
         (url-request-data data)
         (url-request-extra-headers
@@ -86,6 +97,8 @@ It returns the code provided by the service."
     (with-current-buffer (url-retrieve-synchronously url)
       (let ((data (oauth2-request-access-parse)))
         (kill-buffer (current-buffer))
+        (oauth2--do-debug "oauth2-make-access-request: response: %s"
+                          (pp-to-string data))
         data))))
 
 (cl-defstruct oauth2-token
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-Add-NEWS-file-to-document-the-changes-to-plstore-id-.patch --]
[-- Type: text/x-diff, Size: 1350 bytes --]

From e8735da21ac82b0698edad1796ddf4a1b8eb4bb2 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Tue, 30 Jul 2024 03:46:57 -0700
Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
 generation

---
 NEWS | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100644 NEWS

diff --git a/NEWS b/NEWS
new file mode 100644
index 0000000000..6715a1914a
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,23 @@
+Summary of changes to oauth2.el
+-------------------------------
+
+For changes of 0.16 and older or full changes please check the git
+history of the repository of oauth2.el.
+
+* 0.17
+
+** Changes to plstore id generation and needs to reacquire refresh_token
+
+The generation of plstore id used to include `auth-url', `token-url',
+and `scope'.  Now `client-id' is also included.  This is required to
+support multiple accounts of some providers which use the same
+`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
+generated plstore id is not unique amount accounts.  Adding
+`client-id' solves this problem.
+
+The hash function of calculating the plstore id has also changed from
+MD5 to SHA512 to be more secure.
+
+As a result, users of oauth2.el will need to redo the authentication
+process to get a new refresh_token when upgrading from older version
+to 0.17.
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Eli Zaretskii]

> From: Xiyue Deng <manphiz@gmail.com>
> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
> Date: Wed, 14 Aug 2024 01:23:19 -0700
> 
> >> It's been a few days since the last time I received feedback for
> >> improvements regarding my patches.  Is there any other feedbacks/reviews
> >> I am expecting from the co-maintainers?  Please also let me know when
> >> it's time to ask for merging and requesting a new tagged release.
> >
> > ?? The last message in this discussion was just yesterday evening, and
> > my understanding is that you are still discussing the issues and did
> > not reach the final conclusion.  If I'm mistaken, my apologies;
> 
> The recent communication was not related to my patches but to check
> whether it is possible to support outlook.com OAuth2 login (and the
> conclusion was no because refreshing access token was disabled as
> confirmed by MS representative during an online chat.)
> 
> > please describe your conclusion and post the patch that you-all agree
> > would solve the issues, and let's take it from there.
> 
> I actually only received comments from Robert and I have updated my
> patches according in [1][2] (also attached in EOM).
> 
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44

Thanks.

Philip, could you please DTRT here?  This seems to be an ELPA package.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Xiyue Deng <manphiz@gmail.com>
>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>> 
>> >> It's been a few days since the last time I received feedback for
>> >> improvements regarding my patches.  Is there any other feedbacks/reviews
>> >> I am expecting from the co-maintainers?  Please also let me know when
>> >> it's time to ask for merging and requesting a new tagged release.
>> >
>> > ?? The last message in this discussion was just yesterday evening, and
>> > my understanding is that you are still discussing the issues and did
>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>> 
>> The recent communication was not related to my patches but to check
>> whether it is possible to support outlook.com OAuth2 login (and the
>> conclusion was no because refreshing access token was disabled as
>> confirmed by MS representative during an online chat.)
>> 
>> > please describe your conclusion and post the patch that you-all agree
>> > would solve the issues, and let's take it from there.
>> 
>> I actually only received comments from Robert and I have updated my
>> patches according in [1][2] (also attached in EOM).
>> 
>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>
> Thanks.
>
> Philip, could you please DTRT here?  This seems to be an ELPA package.

Friendly ping.  Please also let me know if there are more review
comments.

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Philip Kaludercic]

Xiyue Deng <manphiz@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>>> 
>>> >> It's been a few days since the last time I received feedback for
>>> >> improvements regarding my patches.  Is there any other feedbacks/reviews
>>> >> I am expecting from the co-maintainers?  Please also let me know when
>>> >> it's time to ask for merging and requesting a new tagged release.
>>> >
>>> > ?? The last message in this discussion was just yesterday evening, and
>>> > my understanding is that you are still discussing the issues and did
>>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>>> 
>>> The recent communication was not related to my patches but to check
>>> whether it is possible to support outlook.com OAuth2 login (and the
>>> conclusion was no because refreshing access token was disabled as
>>> confirmed by MS representative during an online chat.)
>>> 
>>> > please describe your conclusion and post the patch that you-all agree
>>> > would solve the issues, and let's take it from there.
>>> 
>>> I actually only received comments from Robert and I have updated my
>>> patches according in [1][2] (also attached in EOM).
>>> 
>>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>>
>> Thanks.
>>
>> Philip, could you please DTRT here?  This seems to be an ELPA package.
>
> Friendly ping.  Please also let me know if there are more review
> comments.

I'm sorry, this was a rather long thread and I didn't have the time yet
to follow up on it.  Can you confirm that you want me to review the
patches attached to this message:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20

?

-- 
	Philip Kaludercic on peregrine

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

Hi Philip,

Philip Kaludercic <philipk@posteo.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Xiyue Deng <manphiz@gmail.com>
>>>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>>>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>>>> 
>>>> >> It's been a few days since the last time I received feedback for
>>>> >> improvements regarding my patches.  Is there any other feedbacks/reviews
>>>> >> I am expecting from the co-maintainers?  Please also let me know when
>>>> >> it's time to ask for merging and requesting a new tagged release.
>>>> >
>>>> > ?? The last message in this discussion was just yesterday evening, and
>>>> > my understanding is that you are still discussing the issues and did
>>>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>>>> 
>>>> The recent communication was not related to my patches but to check
>>>> whether it is possible to support outlook.com OAuth2 login (and the
>>>> conclusion was no because refreshing access token was disabled as
>>>> confirmed by MS representative during an online chat.)
>>>> 
>>>> > please describe your conclusion and post the patch that you-all agree
>>>> > would solve the issues, and let's take it from there.
>>>> 
>>>> I actually only received comments from Robert and I have updated my
>>>> patches according in [1][2] (also attached in EOM).
>>>> 
>>>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>>>
>>> Thanks.
>>>
>>> Philip, could you please DTRT here?  This seems to be an ELPA package.
>>
>> Friendly ping.  Please also let me know if there are more review
>> comments.
>
> I'm sorry, this was a rather long thread and I didn't have the time yet
> to follow up on it.  Can you confirm that you want me to review the
> patches attached to this message:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>
> ?

Almost with a later update for patch 5.  I am now attaching the latest
patches here to avoid any confusions.  Thanks!

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2055 bytes --]

From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-parameters-required-by-Google-OAuth2-to-get-refr.patch --]
[-- Type: text/x-diff, Size: 1264 bytes --]

From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:52:02 -0700
Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
 refresh_token

* packages/oauth2/oauth2.el (oauth2-request-authorization): add
`access_type=offline' and `prompt=consent' when requesting token to
receive refresh_token.
---
 oauth2.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 3a3e50ad2b..9780ac3a1d 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -63,7 +63,9 @@ It returns the code provided by the service."
                      "&response_type=code"
                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+                     (if state (concat "&state=" (url-hexify-string state)) "")
+                     "&access_type=offline"
+                     "&prompt=consent")))
     (browse-url url)
     (read-string (concat "Follow the instruction on your default browser, or "
                          "visit:\n" url
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Encode-parameters-when-requesting-access.patch --]
[-- Type: text/x-diff, Size: 1194 bytes --]

From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:54:08 -0700
Subject: [PATCH 3/6] Encode parameters when requesting access

* packages/oauth2/oauth2.el (oauth2-request-access): encode all
parameters which may contain characters that breaks URL.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 9780ac3a1d..b035742fc1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
            (oauth2-make-access-request
             token-url
             (concat
-             "client_id=" client-id
+             "client_id=" (url-hexify-string client-id)
 	     (when client-secret
-               (concat  "&client_secret=" client-secret))
-             "&code=" code
+               (concat  "&client_secret=" (url-hexify-string client-secret)))
+             "&code=" (url-hexify-string code)
              "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
              "&grant_type=authorization_code"))))
       (make-oauth2-token :client-id client-id
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Support-storing-data-for-multiple-accounts-of-the-sa.patch --]
[-- Type: text/x-diff, Size: 2090 bytes --]

From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:00:04 -0700
Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
 provider

Currently the plstore id computed by `oauth2-compute-id' only takes
`auth-url', `token-url', and `scope' into account, which could be the
same for the same provider (e.g. Gmail).  This prevents storing
information for multiple accounts of the same service for some
providers.

This patch adds `client-id' to the calculation of plstore id to make
sure that it is unique for different accounts of the same provider.

It also changes the hash function to sha512 to be more secure.

* packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
parameter of `oauth2-compute-id' to ensure unique id amount multiple
accounts of the same provider, and change hash function to sha512.
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index b035742fc1..035971ac85 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
   :group 'oauth2
   :type 'file)
 
-(defun oauth2-compute-id (auth-url token-url scope)
+(defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
 This allows to store the token in an unique way."
-  (secure-hash 'md5 (concat auth-url token-url scope)))
+  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
 
 ;;;###autoload
 (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
   "Request access to a resource and store it using `plstore'."
   ;; We store a MD5 sum of all URL
   (let* ((plstore (plstore-open oauth2-token-file))
-         (id (oauth2-compute-id auth-url token-url scope))
+         (id (oauth2-compute-id auth-url token-url scope client-id))
          (plist (cdr (plstore-get plstore id))))
     ;; Check if we found something matching this access
     (if plist
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-debug-messages-and-provide-a-switch-variable-for.patch --]
[-- Type: text/x-diff, Size: 2186 bytes --]

From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
 enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add support for debug messages and a
switch variable for enabling.
---
 oauth2.el | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/oauth2.el b/oauth2.el
index 035971ac85..ce7a835100 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -40,6 +40,7 @@
 (require 'plstore)
 (require 'json)
 (require 'url-http)
+(require 'pp)
 
 (defvar url-http-data)
 (defvar url-http-method)
@@ -53,6 +54,14 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defvar oauth2-debug nil
+  "Enable debug messages.")
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (if oauth2-debug
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -79,6 +88,8 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
+  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
+  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
   (let ((url-request-method "POST")
         (url-request-data data)
         (url-request-extra-headers
@@ -86,6 +97,8 @@ It returns the code provided by the service."
     (with-current-buffer (url-retrieve-synchronously url)
       (let ((data (oauth2-request-access-parse)))
         (kill-buffer (current-buffer))
+        (oauth2--do-debug "oauth2-make-access-request: response: %s"
+                          (pp-to-string data))
         data))))
 
 (cl-defstruct oauth2-token
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-Add-NEWS-file-to-document-the-changes-to-plstore-id-.patch --]
[-- Type: text/x-diff, Size: 1350 bytes --]

From e8735da21ac82b0698edad1796ddf4a1b8eb4bb2 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Tue, 30 Jul 2024 03:46:57 -0700
Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
 generation

---
 NEWS | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100644 NEWS

diff --git a/NEWS b/NEWS
new file mode 100644
index 0000000000..6715a1914a
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,23 @@
+Summary of changes to oauth2.el
+-------------------------------
+
+For changes of 0.16 and older or full changes please check the git
+history of the repository of oauth2.el.
+
+* 0.17
+
+** Changes to plstore id generation and needs to reacquire refresh_token
+
+The generation of plstore id used to include `auth-url', `token-url',
+and `scope'.  Now `client-id' is also included.  This is required to
+support multiple accounts of some providers which use the same
+`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
+generated plstore id is not unique amount accounts.  Adding
+`client-id' solves this problem.
+
+The hash function of calculating the plstore id has also changed from
+MD5 to SHA512 to be more secure.
+
+As a result, users of oauth2.el will need to redo the authentication
+process to get a new refresh_token when upgrading from older version
+to 0.17.
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Philip,
>
> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Xiyue Deng <manphiz@gmail.com>
>>>>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>>>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>>>>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>>>>> 
>>>>> >> It's been a few days since the last time I received feedback for
>>>>> >> improvements regarding my patches.  Is there any other feedbacks/reviews
>>>>> >> I am expecting from the co-maintainers?  Please also let me know when
>>>>> >> it's time to ask for merging and requesting a new tagged release.
>>>>> >
>>>>> > ?? The last message in this discussion was just yesterday evening, and
>>>>> > my understanding is that you are still discussing the issues and did
>>>>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>>>>> 
>>>>> The recent communication was not related to my patches but to check
>>>>> whether it is possible to support outlook.com OAuth2 login (and the
>>>>> conclusion was no because refreshing access token was disabled as
>>>>> confirmed by MS representative during an online chat.)
>>>>> 
>>>>> > please describe your conclusion and post the patch that you-all agree
>>>>> > would solve the issues, and let's take it from there.
>>>>> 
>>>>> I actually only received comments from Robert and I have updated my
>>>>> patches according in [1][2] (also attached in EOM).
>>>>> 
>>>>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>>>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>>>>
>>>> Thanks.
>>>>
>>>> Philip, could you please DTRT here?  This seems to be an ELPA package.
>>>
>>> Friendly ping.  Please also let me know if there are more review
>>> comments.
>>
>> I'm sorry, this was a rather long thread and I didn't have the time yet
>> to follow up on it.  Can you confirm that you want me to review the
>> patches attached to this message:
>>
>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>
>> ?
>
> Almost with a later update for patch 5.  I am now attaching the latest
> patches here to avoid any confusions.  Thanks!

Friendly ping.

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Philip Kaludercic]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Philip,
>
> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Xiyue Deng <manphiz@gmail.com>
>>>>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>>>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>>>>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>>>>> 
>>>>> >> It's been a few days since the last time I received feedback for
>>>>> >> improvements regarding my patches.  Is there any other
>>>>> >> feedbacks/reviews
>>>>> >> I am expecting from the co-maintainers?  Please also let me know when
>>>>> >> it's time to ask for merging and requesting a new tagged release.
>>>>> >
>>>>> > ?? The last message in this discussion was just yesterday evening, and
>>>>> > my understanding is that you are still discussing the issues and did
>>>>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>>>>> 
>>>>> The recent communication was not related to my patches but to check
>>>>> whether it is possible to support outlook.com OAuth2 login (and the
>>>>> conclusion was no because refreshing access token was disabled as
>>>>> confirmed by MS representative during an online chat.)
>>>>> 
>>>>> > please describe your conclusion and post the patch that you-all agree
>>>>> > would solve the issues, and let's take it from there.
>>>>> 
>>>>> I actually only received comments from Robert and I have updated my
>>>>> patches according in [1][2] (also attached in EOM).
>>>>> 
>>>>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>>>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>>>>
>>>> Thanks.
>>>>
>>>> Philip, could you please DTRT here?  This seems to be an ELPA package.
>>>
>>> Friendly ping.  Please also let me know if there are more review
>>> comments.
>>
>> I'm sorry, this was a rather long thread and I didn't have the time yet
>> to follow up on it.  Can you confirm that you want me to review the
>> patches attached to this message:
>>
>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>
>> ?
>
> Almost with a later update for patch 5.  I am now attaching the latest
> patches here to avoid any confusions.  Thanks!
>
> -- 
> Xiyue Deng
>
> From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Sun, 21 Jul 2024 14:50:56 -0700
> Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
>  visit it
>
> * packages/oauth2/oauth2.el (oauth2-request-authorization): show full
> authentication URL in user prompt.

Generally it would be nice if you could attach a "(Bug#72358)" to the
end of each commit.  Just add it like a new sentence.

    [...]
    authentication URL in user prompt.  (Bug#72358)

> ---
>  oauth2.el | 19 +++++++++++--------
>  1 file changed, 11 insertions(+), 8 deletions(-)
>
> diff --git a/oauth2.el b/oauth2.el
> index 7da9702004..3a3e50ad2b 100644
> --- a/oauth2.el
> +++ b/oauth2.el
> @@ -57,14 +57,17 @@
>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>  CLIENT-ID is the client id provided by the provider.
>  It returns the code provided by the service."
> -  (browse-url (concat auth-url
> -                      (if (string-match-p "\?" auth-url) "&" "?")
> -                      "client_id=" (url-hexify-string client-id)
> -                      "&response_type=code"
> -                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
> -                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
> -                      (if state (concat "&state=" (url-hexify-string state)) "")))
> -  (read-string "Enter the code your browser displayed: "))
> +  (let ((url (concat auth-url
> +                     (if (string-match-p "\?" auth-url) "&" "?")
> +                     "client_id=" (url-hexify-string client-id)
> +                     "&response_type=code"
> +                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
> +                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
> +                     (if state (concat "&state=" (url-hexify-string state)) ""))))
> +    (browse-url url)
> +    (read-string (concat "Follow the instruction on your default browser, or "
> +                         "visit:\n" url
> +                         "\nEnter the code your browser displayed: "))))

LGTM.

>  
>  (defun oauth2-request-access-parse ()
>    "Parse the result of an OAuth request."
> -- 
> 2.39.2
>
>
> From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Sun, 21 Jul 2024 14:52:02 -0700
> Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
>  refresh_token
>
> * packages/oauth2/oauth2.el (oauth2-request-authorization): add
> `access_type=offline' and `prompt=consent' when requesting token to

I wouldn't use `...' syntax in commit ChangeLog messages.

> receive refresh_token.
> ---
>  oauth2.el | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/oauth2.el b/oauth2.el
> index 3a3e50ad2b..9780ac3a1d 100644
> --- a/oauth2.el
> +++ b/oauth2.el
> @@ -63,7 +63,9 @@ It returns the code provided by the service."
>                       "&response_type=code"
>                       "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>                       (if scope (concat "&scope=" (url-hexify-string scope)) "")
> -                     (if state (concat "&state=" (url-hexify-string state)) ""))))
> +                     (if state (concat "&state=" (url-hexify-string state)) "")
> +                     "&access_type=offline"
> +                     "&prompt=consent")))

You do what you say, but perhaps a comment explaining why you have these
queries might be useful?

>      (browse-url url)
>      (read-string (concat "Follow the instruction on your default browser, or "
>                           "visit:\n" url
> -- 
> 2.39.2
>
>
> From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Sun, 21 Jul 2024 14:54:08 -0700
> Subject: [PATCH 3/6] Encode parameters when requesting access
>
> * packages/oauth2/oauth2.el (oauth2-request-access): encode all
> parameters which may contain characters that breaks URL.
> ---
>  oauth2.el | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/oauth2.el b/oauth2.el
> index 9780ac3a1d..b035742fc1 100644
> --- a/oauth2.el
> +++ b/oauth2.el
> @@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
>             (oauth2-make-access-request
>              token-url
>              (concat
> -             "client_id=" client-id
> +             "client_id=" (url-hexify-string client-id)
>  	     (when client-secret
> -               (concat  "&client_secret=" client-secret))
> -             "&code=" code
> +               (concat  "&client_secret=" (url-hexify-string client-secret)))
> +             "&code=" (url-hexify-string code)
>               "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>               "&grant_type=authorization_code"))))
>        (make-oauth2-token :client-id client-id
> -- 
> 2.39.2

1+  Though one might also consider using something like `url-encode-url'
to generate the entire URL.

>
>
> From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Sun, 28 Jul 2024 03:00:04 -0700
> Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
>  provider
>
> Currently the plstore id computed by `oauth2-compute-id' only takes
> `auth-url', `token-url', and `scope' into account, which could be the
> same for the same provider (e.g. Gmail).  This prevents storing
> information for multiple accounts of the same service for some
> providers.
>
> This patch adds `client-id' to the calculation of plstore id to make
> sure that it is unique for different accounts of the same provider.
>
> It also changes the hash function to sha512 to be more secure.
>
> * packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
> parameter of `oauth2-compute-id' to ensure unique id amount multiple
> accounts of the same provider, and change hash function to sha512.
> ---
>  oauth2.el | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/oauth2.el b/oauth2.el
> index b035742fc1..035971ac85 100644
> --- a/oauth2.el
> +++ b/oauth2.el
> @@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
>    :group 'oauth2
>    :type 'file)
>  
> -(defun oauth2-compute-id (auth-url token-url scope)
> +(defun oauth2-compute-id (auth-url token-url scope client-id)
>    "Compute an unique id based on URLs.
>  This allows to store the token in an unique way."
> -  (secure-hash 'md5 (concat auth-url token-url scope)))
> +  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
>  
>  ;;;###autoload
>  (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
>    "Request access to a resource and store it using `plstore'."
>    ;; We store a MD5 sum of all URL
>    (let* ((plstore (plstore-open oauth2-token-file))
> -         (id (oauth2-compute-id auth-url token-url scope))
> +         (id (oauth2-compute-id auth-url token-url scope client-id))
>           (plist (cdr (plstore-get plstore id))))
>      ;; Check if we found something matching this access
>      (if plist
> -- 
> 2.39.2

LGTM

>
> From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Sun, 28 Jul 2024 03:41:20 -0700
> Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
>  enabling
>
> This helps debugging whether the authorization and refresh requests
> were successful and inspecting the responses.
>
> * packages/oauth2/oauth2.el: add support for debug messages and a
> switch variable for enabling.
> ---
>  oauth2.el | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/oauth2.el b/oauth2.el
> index 035971ac85..ce7a835100 100644
> --- a/oauth2.el
> +++ b/oauth2.el
> @@ -40,6 +40,7 @@
>  (require 'plstore)
>  (require 'json)
>  (require 'url-http)
> +(require 'pp)
>  
>  (defvar url-http-data)
>  (defvar url-http-method)
> @@ -53,6 +54,14 @@
>    :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
>    :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
>  
> +(defvar oauth2-debug nil
> +  "Enable debug messages.")

I'd slightly expand this comment in case someone wants to use something
like `apropos-documentation'.  Also, why is this not a user option?

> +
> +(defun oauth2--do-debug (&rest msg)
> +  "Output debug messages when `oauth2-debug' is enabled."
> +  (if oauth2-debug

I'd use `when' here.  

> +    (apply #'message msg)))

... and perhaps prefix the message with a string like "[oauth2] ".

I also notice that you manually mention the function issuing the debug
message below.  It is possible to infer this using `backtrace-frame'
(but that is really something that should be added to the core, instead
of packages having to re-implement it themselves.)

> +
>  (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>  CLIENT-ID is the client id provided by the provider.
> @@ -79,6 +88,8 @@ It returns the code provided by the service."
>  
>  (defun oauth2-make-access-request (url data)
>    "Make an access request to URL using DATA in POST."
> +  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
> +  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
>    (let ((url-request-method "POST")
>          (url-request-data data)
>          (url-request-extra-headers
> @@ -86,6 +97,8 @@ It returns the code provided by the service."
>      (with-current-buffer (url-retrieve-synchronously url)
>        (let ((data (oauth2-request-access-parse)))
>          (kill-buffer (current-buffer))
> +        (oauth2--do-debug "oauth2-make-access-request: response: %s"
> +                          (pp-to-string data))

Is pp-to-string here really much better than prin1-to-string?  Note that
'oauth2--do-debug' is a function, so the arguments are always evaluated.
I have experienced pp being significantly slower than the core print
functions, so this is something that is worth thinking about IMO.

>          data))))
>  
>  (cl-defstruct oauth2-token
> -- 
> 2.39.2
>
>
> From e8735da21ac82b0698edad1796ddf4a1b8eb4bb2 Mon Sep 17 00:00:00 2001
> From: Xiyue Deng <manphiz@gmail.com>
> Date: Tue, 30 Jul 2024 03:46:57 -0700
> Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
>  generation
>
> ---
>  NEWS | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
>  create mode 100644 NEWS
>
> diff --git a/NEWS b/NEWS
> new file mode 100644
> index 0000000000..6715a1914a
> --- /dev/null
> +++ b/NEWS
> @@ -0,0 +1,23 @@
> +Summary of changes to oauth2.el
> +-------------------------------
> +
> +For changes of 0.16 and older or full changes please check the git
> +history of the repository of oauth2.el.
> +
> +* 0.17
> +
> +** Changes to plstore id generation and needs to reacquire refresh_token
> +
> +The generation of plstore id used to include `auth-url', `token-url',
> +and `scope'.  Now `client-id' is also included.  This is required to
> +support multiple accounts of some providers which use the same
> +`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
> +generated plstore id is not unique amount accounts.  Adding
> +`client-id' solves this problem.
> +
> +The hash function of calculating the plstore id has also changed from
> +MD5 to SHA512 to be more secure.
> +
> +As a result, users of oauth2.el will need to redo the authentication
> +process to get a new refresh_token when upgrading from older version
> +to 0.17.

Perhaps you can add a file local variable to enable outline-mode?

All in all the patches look more than fine though, nothing I mentioned
is pressing.  If you don't have the time and the changes are urgent,
then I can also apply them as such.

Xiyue Deng <manphiz@gmail.com> writes:

[...]

> Friendly ping.

Friendly pong.

-- 
	Philip Kaludercic on peregrine

bug#72358: 29.4; oauth2.el improvements

[Robert Pluim]

>>>>> On Thu, 29 Aug 2024 14:14:17 +0000, Philip Kaludercic <philipk@posteo.net> said:
    >> +(defvar oauth2-debug nil
    >> +  "Enable debug messages.")

    Philip> I'd slightly expand this comment in case someone wants to use something
    Philip> like `apropos-documentation'.  Also, why is this not a user option?

Because I asked him to make it a `defvar' :-)  I donʼt think debug
options should be customizable, theyʼre not user-level.

Robert
-- 

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 16503 bytes --]

Hi Philip,

Thanks for the review!  Please see my replies below inline.

Philip Kaludercic <philipk@posteo.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Hi Philip,
>>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>
>>>>>> From: Xiyue Deng <manphiz@gmail.com>
>>>>>> Cc: Thomas Fitzsimmons <fitzsim@fitzsim.org>,  Björn Bidar
>>>>>>  <bjorn.bidar@thaodan.de>,  rpluim@gmail.com,  72358@debbugs.gnu.org
>>>>>> Date: Wed, 14 Aug 2024 01:23:19 -0700
>>>>>> 
>>>>>> >> It's been a few days since the last time I received feedback for
>>>>>> >> improvements regarding my patches.  Is there any other
>>>>>> >> feedbacks/reviews
>>>>>> >> I am expecting from the co-maintainers?  Please also let me know when
>>>>>> >> it's time to ask for merging and requesting a new tagged release.
>>>>>> >
>>>>>> > ?? The last message in this discussion was just yesterday evening, and
>>>>>> > my understanding is that you are still discussing the issues and did
>>>>>> > not reach the final conclusion.  If I'm mistaken, my apologies;
>>>>>> 
>>>>>> The recent communication was not related to my patches but to check
>>>>>> whether it is possible to support outlook.com OAuth2 login (and the
>>>>>> conclusion was no because refreshing access token was disabled as
>>>>>> confirmed by MS representative during an online chat.)
>>>>>> 
>>>>>> > please describe your conclusion and post the patch that you-all agree
>>>>>> > would solve the issues, and let's take it from there.
>>>>>> 
>>>>>> I actually only received comments from Robert and I have updated my
>>>>>> patches according in [1][2] (also attached in EOM).
>>>>>> 
>>>>>> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>>>>> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#44
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Philip, could you please DTRT here?  This seems to be an ELPA package.
>>>>
>>>> Friendly ping.  Please also let me know if there are more review
>>>> comments.
>>>
>>> I'm sorry, this was a rather long thread and I didn't have the time yet
>>> to follow up on it.  Can you confirm that you want me to review the
>>> patches attached to this message:
>>>
>>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72358#20
>>>
>>> ?
>>
>> Almost with a later update for patch 5.  I am now attaching the latest
>> patches here to avoid any confusions.  Thanks!
>>
>> -- 
>> Xiyue Deng
>>
>> From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Sun, 21 Jul 2024 14:50:56 -0700
>> Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
>>  visit it
>>
>> * packages/oauth2/oauth2.el (oauth2-request-authorization): show full
>> authentication URL in user prompt.
>
> Generally it would be nice if you could attach a "(Bug#72358)" to the
> end of each commit.  Just add it like a new sentence.
>
>     [...]
>     authentication URL in user prompt.  (Bug#72358)
>

Done for all patches.

>> ---
>>  oauth2.el | 19 +++++++++++--------
>>  1 file changed, 11 insertions(+), 8 deletions(-)
>>
>> diff --git a/oauth2.el b/oauth2.el
>> index 7da9702004..3a3e50ad2b 100644
>> --- a/oauth2.el
>> +++ b/oauth2.el
>> @@ -57,14 +57,17 @@
>>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>>  CLIENT-ID is the client id provided by the provider.
>>  It returns the code provided by the service."
>> -  (browse-url (concat auth-url
>> -                      (if (string-match-p "\?" auth-url) "&" "?")
>> -                      "client_id=" (url-hexify-string client-id)
>> -                      "&response_type=code"
>> -                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>> -                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
>> -                      (if state (concat "&state=" (url-hexify-string state)) "")))
>> -  (read-string "Enter the code your browser displayed: "))
>> +  (let ((url (concat auth-url
>> +                     (if (string-match-p "\?" auth-url) "&" "?")
>> +                     "client_id=" (url-hexify-string client-id)
>> +                     "&response_type=code"
>> +                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>> +                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
>> +                     (if state (concat "&state=" (url-hexify-string state)) ""))))
>> +    (browse-url url)
>> +    (read-string (concat "Follow the instruction on your default browser, or "
>> +                         "visit:\n" url
>> +                         "\nEnter the code your browser displayed: "))))
>
> LGTM.
>
>>  
>>  (defun oauth2-request-access-parse ()
>>    "Parse the result of an OAuth request."
>> -- 
>> 2.39.2
>>
>>
>> From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Sun, 21 Jul 2024 14:52:02 -0700
>> Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
>>  refresh_token
>>
>> * packages/oauth2/oauth2.el (oauth2-request-authorization): add
>> `access_type=offline' and `prompt=consent' when requesting token to
>
> I wouldn't use `...' syntax in commit ChangeLog messages.
>

Ah I saw this used in the texinfo doc extensively so I thought it was
preferred.  I have changed all these to double quotes, though do let me
know if there is a better preference.

>> receive refresh_token.
>> ---
>>  oauth2.el | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/oauth2.el b/oauth2.el
>> index 3a3e50ad2b..9780ac3a1d 100644
>> --- a/oauth2.el
>> +++ b/oauth2.el
>> @@ -63,7 +63,9 @@ It returns the code provided by the service."
>>                       "&response_type=code"
>>                       "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>                       (if scope (concat "&scope=" (url-hexify-string scope)) "")
>> -                     (if state (concat "&state=" (url-hexify-string state)) ""))))
>> +                     (if state (concat "&state=" (url-hexify-string state)) "")
>> +                     "&access_type=offline"
>> +                     "&prompt=consent")))
>
> You do what you say, but perhaps a comment explaining why you have these
> queries might be useful?
>

Added a comment above to explain the reasons.

>>      (browse-url url)
>>      (read-string (concat "Follow the instruction on your default browser, or "
>>                           "visit:\n" url
>> -- 
>> 2.39.2
>>
>>
>> From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Sun, 21 Jul 2024 14:54:08 -0700
>> Subject: [PATCH 3/6] Encode parameters when requesting access
>>
>> * packages/oauth2/oauth2.el (oauth2-request-access): encode all
>> parameters which may contain characters that breaks URL.
>> ---
>>  oauth2.el | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/oauth2.el b/oauth2.el
>> index 9780ac3a1d..b035742fc1 100644
>> --- a/oauth2.el
>> +++ b/oauth2.el
>> @@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
>>             (oauth2-make-access-request
>>              token-url
>>              (concat
>> -             "client_id=" client-id
>> +             "client_id=" (url-hexify-string client-id)
>>  	     (when client-secret
>> -               (concat  "&client_secret=" client-secret))
>> -             "&code=" code
>> +               (concat  "&client_secret=" (url-hexify-string client-secret)))
>> +             "&code=" (url-hexify-string code)
>>               "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>               "&grant_type=authorization_code"))))
>>        (make-oauth2-token :client-id client-id
>> -- 
>> 2.39.2
>
> 1+  Though one might also consider using something like `url-encode-url'
> to generate the entire URL.
>

It is indeed much cleaner.  Changed accordingly.

>>
>>
>> From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Sun, 28 Jul 2024 03:00:04 -0700
>> Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
>>  provider
>>
>> Currently the plstore id computed by `oauth2-compute-id' only takes
>> `auth-url', `token-url', and `scope' into account, which could be the
>> same for the same provider (e.g. Gmail).  This prevents storing
>> information for multiple accounts of the same service for some
>> providers.
>>
>> This patch adds `client-id' to the calculation of plstore id to make
>> sure that it is unique for different accounts of the same provider.
>>
>> It also changes the hash function to sha512 to be more secure.
>>
>> * packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
>> parameter of `oauth2-compute-id' to ensure unique id amount multiple
>> accounts of the same provider, and change hash function to sha512.
>> ---
>>  oauth2.el | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/oauth2.el b/oauth2.el
>> index b035742fc1..035971ac85 100644
>> --- a/oauth2.el
>> +++ b/oauth2.el
>> @@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
>>    :group 'oauth2
>>    :type 'file)
>>  
>> -(defun oauth2-compute-id (auth-url token-url scope)
>> +(defun oauth2-compute-id (auth-url token-url scope client-id)
>>    "Compute an unique id based on URLs.
>>  This allows to store the token in an unique way."
>> -  (secure-hash 'md5 (concat auth-url token-url scope)))
>> +  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
>>  
>>  ;;;###autoload
>>  (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
>>    "Request access to a resource and store it using `plstore'."
>>    ;; We store a MD5 sum of all URL
>>    (let* ((plstore (plstore-open oauth2-token-file))
>> -         (id (oauth2-compute-id auth-url token-url scope))
>> +         (id (oauth2-compute-id auth-url token-url scope client-id))
>>           (plist (cdr (plstore-get plstore id))))
>>      ;; Check if we found something matching this access
>>      (if plist
>> -- 
>> 2.39.2
>
> LGTM
>
>>
>> From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Sun, 28 Jul 2024 03:41:20 -0700
>> Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
>>  enabling
>>
>> This helps debugging whether the authorization and refresh requests
>> were successful and inspecting the responses.
>>
>> * packages/oauth2/oauth2.el: add support for debug messages and a
>> switch variable for enabling.
>> ---
>>  oauth2.el | 13 +++++++++++++
>>  1 file changed, 13 insertions(+)
>>
>> diff --git a/oauth2.el b/oauth2.el
>> index 035971ac85..ce7a835100 100644
>> --- a/oauth2.el
>> +++ b/oauth2.el
>> @@ -40,6 +40,7 @@
>>  (require 'plstore)
>>  (require 'json)
>>  (require 'url-http)
>> +(require 'pp)
>>  
>>  (defvar url-http-data)
>>  (defvar url-http-method)
>> @@ -53,6 +54,14 @@
>>    :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
>>    :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
>>  
>> +(defvar oauth2-debug nil
>> +  "Enable debug messages.")
>
> I'd slightly expand this comment in case someone wants to use something
> like `apropos-documentation'.

Expanded the doc string to explain the purpose.

> Also, why is this not a user option?
>

As Robert mentioned in the follow-up mail it was his suggestion (thanks
Robert!)  Do let me know if `defcustom' is preferred.

>> +
>> +(defun oauth2--do-debug (&rest msg)
>> +  "Output debug messages when `oauth2-debug' is enabled."
>> +  (if oauth2-debug
>
> I'd use `when' here.  
>

Done

>> +    (apply #'message msg)))
>
> ... and perhaps prefix the message with a string like "[oauth2] ".
>

Done.  I used `setcar' but let me know if there is a more elegant way to
do this.

> I also notice that you manually mention the function issuing the debug
> message below.  It is possible to infer this using `backtrace-frame'
> (but that is really something that should be added to the core, instead
> of packages having to re-implement it themselves.)
>

Added a let-bounded `func-name' by getting function name from
backtrace-frame.  And it would be great if there is a built-in macro
like `__PRETTY_FUNCTION__' and `__LINE__'.

>> +
>>  (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
>>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>>  CLIENT-ID is the client id provided by the provider.
>> @@ -79,6 +88,8 @@ It returns the code provided by the service."
>>  
>>  (defun oauth2-make-access-request (url data)
>>    "Make an access request to URL using DATA in POST."
>> +  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
>> +  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
>>    (let ((url-request-method "POST")
>>          (url-request-data data)
>>          (url-request-extra-headers
>> @@ -86,6 +97,8 @@ It returns the code provided by the service."
>>      (with-current-buffer (url-retrieve-synchronously url)
>>        (let ((data (oauth2-request-access-parse)))
>>          (kill-buffer (current-buffer))
>> +        (oauth2--do-debug "oauth2-make-access-request: response: %s"
>> +                          (pp-to-string data))
>
> Is pp-to-string here really much better than prin1-to-string?  Note that
> 'oauth2--do-debug' is a function, so the arguments are always evaluated.
> I have experienced pp being significantly slower than the core print
> functions, so this is something that is worth thinking about IMO.
>

It's just a personal preference.  And it look like the values are mostly
key-value pairs so they are not that hard to read through
`prin1-to-string' so I changed to it.

>>          data))))
>>  
>>  (cl-defstruct oauth2-token
>> -- 
>> 2.39.2
>>
>>
>> From e8735da21ac82b0698edad1796ddf4a1b8eb4bb2 Mon Sep 17 00:00:00 2001
>> From: Xiyue Deng <manphiz@gmail.com>
>> Date: Tue, 30 Jul 2024 03:46:57 -0700
>> Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
>>  generation
>>
>> ---
>>  NEWS | 23 +++++++++++++++++++++++
>>  1 file changed, 23 insertions(+)
>>  create mode 100644 NEWS
>>
>> diff --git a/NEWS b/NEWS
>> new file mode 100644
>> index 0000000000..6715a1914a
>> --- /dev/null
>> +++ b/NEWS
>> @@ -0,0 +1,23 @@
>> +Summary of changes to oauth2.el
>> +-------------------------------
>> +
>> +For changes of 0.16 and older or full changes please check the git
>> +history of the repository of oauth2.el.
>> +
>> +* 0.17
>> +
>> +** Changes to plstore id generation and needs to reacquire refresh_token
>> +
>> +The generation of plstore id used to include `auth-url', `token-url',
>> +and `scope'.  Now `client-id' is also included.  This is required to
>> +support multiple accounts of some providers which use the same
>> +`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
>> +generated plstore id is not unique amount accounts.  Adding
>> +`client-id' solves this problem.
>> +
>> +The hash function of calculating the plstore id has also changed from
>> +MD5 to SHA512 to be more secure.
>> +
>> +As a result, users of oauth2.el will need to redo the authentication
>> +process to get a new refresh_token when upgrading from older version
>> +to 0.17.
>
> Perhaps you can add a file local variable to enable outline-mode?
>

Done.  Also improved the wording.

> All in all the patches look more than fine though, nothing I mentioned
> is pressing.  If you don't have the time and the changes are urgent,
> then I can also apply them as such.
>
> Xiyue Deng <manphiz@gmail.com> writes:
>
> [...]
>
>> Friendly ping.
>
> Friendly pong.

The updated patches are attached.  Thanks again!

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Show-full-authentication-URL-to-let-user-choose-how-.patch --]
[-- Type: text/x-diff, Size: 2068 bytes --]

From dc1930f5b4828eb285cb28ad0138e67158003c22 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:50:56 -0700
Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
 visit it

* packages/oauth2/oauth2.el (oauth2-request-authorization): show full
authentication URL in user prompt.  (Bug#72358)
---
 oauth2.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 7da9702004..3a3e50ad2b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -57,14 +57,17 @@
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
 It returns the code provided by the service."
-  (browse-url (concat auth-url
-                      (if (string-match-p "\?" auth-url) "&" "?")
-                      "client_id=" (url-hexify-string client-id)
-                      "&response_type=code"
-                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                      (if state (concat "&state=" (url-hexify-string state)) "")))
-  (read-string "Enter the code your browser displayed: "))
+  (let ((url (concat auth-url
+                     (if (string-match-p "\?" auth-url) "&" "?")
+                     "client_id=" (url-hexify-string client-id)
+                     "&response_type=code"
+                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
+                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
+                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+    (browse-url url)
+    (read-string (concat "Follow the instruction on your default browser, or "
+                         "visit:\n" url
+                         "\nEnter the code your browser displayed: "))))
 
 (defun oauth2-request-access-parse ()
   "Parse the result of an OAuth request."
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-parameters-required-by-Google-OAuth2-to-get-refr.patch --]
[-- Type: text/x-diff, Size: 1419 bytes --]

From 4f6628439bdee67025719314bab5b824c86d19e9 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:52:02 -0700
Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
 refresh_token

* packages/oauth2/oauth2.el (oauth2-request-authorization): add
"access_type=offline" and "prompt=consent" when requesting token to
receive refresh_token.  (Bug#72358)
---
 oauth2.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 3a3e50ad2b..8371e73ffb 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -63,7 +63,11 @@ It returns the code provided by the service."
                      "&response_type=code"
                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
-                     (if state (concat "&state=" (url-hexify-string state)) ""))))
+                     (if state (concat "&state=" (url-hexify-string state)) "")
+                     ;; The following two parameters are required for Gmail
+                     ;; OAuth2 to generate the refresh token
+                     "&access_type=offline"
+                     "&prompt=consent")))
     (browse-url url)
     (read-string (concat "Follow the instruction on your default browser, or "
                          "visit:\n" url
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Encode-URL-when-requesting-access.patch --]
[-- Type: text/x-diff, Size: 1524 bytes --]

From 5f2877930ac69bef64b2d53e7ded4913e13c50b6 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 21 Jul 2024 14:54:08 -0700
Subject: [PATCH 3/6] Encode URL when requesting access

* packages/oauth2/oauth2.el (oauth2-request-access): encode the whole
URL to avoid having characters that breaks URL parsing.  (Bug#72358)
---
 oauth2.el | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 8371e73ffb..54ca61e1cf 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -108,13 +108,14 @@ Return an `oauth2-token' structure."
     (let ((result
            (oauth2-make-access-request
             token-url
-            (concat
-             "client_id=" client-id
-	     (when client-secret
-               (concat  "&client_secret=" client-secret))
-             "&code=" code
-             "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
-             "&grant_type=authorization_code"))))
+            (url-encode-url
+             (concat
+              "client_id=" client-id
+              (when client-secret
+                (concat  "&client_secret=" client-secret))
+              "&code=" code
+              "&redirect_uri=" (or redirect-uri "urn:ietf:wg:oauth:2.0:oob")
+              "&grant_type=authorization_code")))))
       (make-oauth2-token :client-id client-id
                          :client-secret client-secret
                          :access-token (cdr (assoc 'access_token result))
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Support-storing-data-for-multiple-accounts-of-the-sa.patch --]
[-- Type: text/x-diff, Size: 2103 bytes --]

From a5241f369f5ea255fcb6f36c0b52829745d71ca8 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:00:04 -0700
Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
 provider

Currently the plstore id computed by "oauth2-compute-id" only takes
"auth-url", "token-url", and "scope" into account, which could be the
same for the same provider (e.g. Gmail).  This prevents storing
information for multiple accounts of the same service for some
providers.

This patch adds "client-id" to the calculation of plstore id to make
sure that it is unique for different accounts of the same provider.

It also changes the hash function to sha512 to be more secure.

* packages/oauth2/oauth2.el (oauth2-compute-id): add "client-id" as a
parameter of "oauth2-compute-id" to ensure unique id amount multiple
accounts of the same provider, and change hash function to
sha512.  (Bug#72358)
---
 oauth2.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index 54ca61e1cf..c8011ebf9b 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -166,17 +166,17 @@ TOKEN should be obtained with `oauth2-request-access'."
   :group 'oauth2
   :type 'file)
 
-(defun oauth2-compute-id (auth-url token-url scope)
+(defun oauth2-compute-id (auth-url token-url scope client-id)
   "Compute an unique id based on URLs.
 This allows to store the token in an unique way."
-  (secure-hash 'md5 (concat auth-url token-url scope)))
+  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
 
 ;;;###autoload
 (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
   "Request access to a resource and store it using `plstore'."
   ;; We store a MD5 sum of all URL
   (let* ((plstore (plstore-open oauth2-token-file))
-         (id (oauth2-compute-id auth-url token-url scope))
+         (id (oauth2-compute-id auth-url token-url scope client-id))
          (plist (cdr (plstore-get plstore id))))
     ;; Check if we found something matching this access
     (if plist
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-debug-messages-and-provide-an-option-variable-fo.patch --]
[-- Type: text/x-diff, Size: 2540 bytes --]

From e2ece9aee7f43ec35789823576dc5390d3384dc6 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Sun, 28 Jul 2024 03:41:20 -0700
Subject: [PATCH 5/6] Add debug messages and provide an option variable for
 enabling

This helps debugging whether the authorization and refresh requests
were successful and inspecting the responses.

* packages/oauth2/oauth2.el: add support for debug messages and an
option variable for enabling.  (Bug#72358)
---
 oauth2.el | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/oauth2.el b/oauth2.el
index c8011ebf9b..9c91bb08a1 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -53,6 +53,15 @@
   :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
   :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
 
+(defvar oauth2-debug nil
+  "Enable verbose logging in oauth2 to help debugging.")
+
+(defun oauth2--do-debug (&rest msg)
+  "Output debug messages when `oauth2-debug' is enabled."
+  (when oauth2-debug
+    (setcar msg (concat "[oauth2] " (car msg)))
+    (apply #'message msg)))
+
 (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
   "Request OAuth authorization at AUTH-URL by launching `browse-url'.
 CLIENT-ID is the client id provided by the provider.
@@ -81,14 +90,18 @@ It returns the code provided by the service."
 
 (defun oauth2-make-access-request (url data)
   "Make an access request to URL using DATA in POST."
-  (let ((url-request-method "POST")
-        (url-request-data data)
-        (url-request-extra-headers
-         '(("Content-Type" . "application/x-www-form-urlencoded"))))
-    (with-current-buffer (url-retrieve-synchronously url)
-      (let ((data (oauth2-request-access-parse)))
-        (kill-buffer (current-buffer))
-        data))))
+  (let ((func-name (nth 1 (backtrace-frame 3))))
+    (oauth2--do-debug "%s: url: %s" func-name url)
+    (oauth2--do-debug "%s: data: %s" func-name data)
+    (let ((url-request-method "POST")
+          (url-request-data data)
+          (url-request-extra-headers
+           '(("Content-Type" . "application/x-www-form-urlencoded"))))
+      (with-current-buffer (url-retrieve-synchronously url)
+        (let ((data (oauth2-request-access-parse)))
+          (kill-buffer (current-buffer))
+          (oauth2--do-debug "%s: response: %s" func-name (prin1-to-string data))
+          data)))))
 
 (cl-defstruct oauth2-token
   plstore
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-Add-NEWS-file-to-document-the-changes-to-plstore-id-.patch --]
[-- Type: text/x-diff, Size: 1444 bytes --]

From e99ec9b3f209312bfa0048c5ae94876e03654c18 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Tue, 30 Jul 2024 03:46:57 -0700
Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
 generation

* Also add local variables to enable outline-mode.  (Bug#72358)
---
 NEWS | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 NEWS

diff --git a/NEWS b/NEWS
new file mode 100644
index 0000000000..72b0606659
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,27 @@
+Summary of changes to oauth2.el
+-------------------------------
+
+(For changes of 0.16 and older please check the git history of the
+repository of oauth2.el.)
+
+* 0.17
+
+** Changes to plstore id generation and needs to reacquire refresh_token
+
+The generation of plstore id used to include `auth-url', `token-url',
+and `scope'.  Now `client-id' is also included.  This is required to
+support multiple accounts of some providers which use the same
+`auth-url', `token-url', and `scope' (e.g. Gmail), or the generated
+plstore id is not unique amount accounts.  Adding `client-id' solves
+this problem.
+
+The hash function of calculating the plstore id has also changed from
+MD5 to SHA512 to be more secure.
+
+As a result, users of oauth2.el will need to redo the authentication
+process to get a new refresh_token when upgrading from older version
+to 0.17.
+
+# Local variables:
+# mode: outline
+# End:
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Philip Kaludercic]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Philip,
>
> Thanks for the review!  Please see my replies below inline.
>

[...]

>>> From 2b9e50cb0948e0b4f28883042109994ffa295d3d Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Sun, 21 Jul 2024 14:50:56 -0700
>>> Subject: [PATCH 1/6] Show full authentication URL to let user choose how to
>>>  visit it
>>>
>>> * packages/oauth2/oauth2.el (oauth2-request-authorization): show full
>>> authentication URL in user prompt.
>>
>> Generally it would be nice if you could attach a "(Bug#72358)" to the
>> end of each commit.  Just add it like a new sentence.
>>
>>     [...]
>>     authentication URL in user prompt.  (Bug#72358)
>>
>
> Done for all patches.

1+

>>> ---
>>>  oauth2.el | 19 +++++++++++--------
>>>  1 file changed, 11 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/oauth2.el b/oauth2.el
>>> index 7da9702004..3a3e50ad2b 100644
>>> --- a/oauth2.el
>>> +++ b/oauth2.el
>>> @@ -57,14 +57,17 @@
>>>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>>>  CLIENT-ID is the client id provided by the provider.
>>>  It returns the code provided by the service."
>>> -  (browse-url (concat auth-url
>>> -                      (if (string-match-p "\?" auth-url) "&" "?")
>>> -                      "client_id=" (url-hexify-string client-id)
>>> -                      "&response_type=code"
>>> -                      "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>> -                      (if scope (concat "&scope=" (url-hexify-string scope)) "")
>>> -                      (if state (concat "&state=" (url-hexify-string state)) "")))
>>> -  (read-string "Enter the code your browser displayed: "))
>>> +  (let ((url (concat auth-url
>>> +                     (if (string-match-p "\?" auth-url) "&" "?")
>>> +                     "client_id=" (url-hexify-string client-id)
>>> +                     "&response_type=code"
>>> +                     "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>> +                     (if scope (concat "&scope=" (url-hexify-string scope)) "")
>>> +                     (if state (concat "&state=" (url-hexify-string state)) ""))))
>>> +    (browse-url url)
>>> +    (read-string (concat "Follow the instruction on your default browser, or "
>>> +                         "visit:\n" url
>>> +                         "\nEnter the code your browser displayed: "))))
>>
>> LGTM.
>>
>>>  
>>>  (defun oauth2-request-access-parse ()
>>>    "Parse the result of an OAuth request."
>>> -- 
>>> 2.39.2
>>>
>>>
>>> From 26ed9886bd9d3970d55cf76e4269cef3998503a7 Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Sun, 21 Jul 2024 14:52:02 -0700
>>> Subject: [PATCH 2/6] Add parameters required by Google OAuth2 to get
>>>  refresh_token
>>>
>>> * packages/oauth2/oauth2.el (oauth2-request-authorization): add
>>> `access_type=offline' and `prompt=consent' when requesting token to
>>
>> I wouldn't use `...' syntax in commit ChangeLog messages.
>>
>
> Ah I saw this used in the texinfo doc extensively so I thought it was
> preferred.  I have changed all these to double quotes, though do let me
> know if there is a better preference.

I think that double or single quotes are both fine.

>>> receive refresh_token.
>>> ---
>>>  oauth2.el | 4 +++-
>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/oauth2.el b/oauth2.el
>>> index 3a3e50ad2b..9780ac3a1d 100644
>>> --- a/oauth2.el
>>> +++ b/oauth2.el
>>> @@ -63,7 +63,9 @@ It returns the code provided by the service."
>>>                       "&response_type=code"
>>>                       "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>>                       (if scope (concat "&scope=" (url-hexify-string scope)) "")
>>> -                     (if state (concat "&state=" (url-hexify-string state)) ""))))
>>> +                     (if state (concat "&state=" (url-hexify-string state)) "")
>>> +                     "&access_type=offline"
>>> +                     "&prompt=consent")))
>>
>> You do what you say, but perhaps a comment explaining why you have these
>> queries might be useful?
>>
>
> Added a comment above to explain the reasons.

1+

>>>      (browse-url url)
>>>      (read-string (concat "Follow the instruction on your default browser, or "
>>>                           "visit:\n" url
>>> -- 
>>> 2.39.2
>>>
>>>
>>> From 59225412e1d06ae9e165cfde6a4a985cee4fc569 Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Sun, 21 Jul 2024 14:54:08 -0700
>>> Subject: [PATCH 3/6] Encode parameters when requesting access
>>>
>>> * packages/oauth2/oauth2.el (oauth2-request-access): encode all
>>> parameters which may contain characters that breaks URL.
>>> ---
>>>  oauth2.el | 6 +++---
>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/oauth2.el b/oauth2.el
>>> index 9780ac3a1d..b035742fc1 100644
>>> --- a/oauth2.el
>>> +++ b/oauth2.el
>>> @@ -107,10 +107,10 @@ Return an `oauth2-token' structure."
>>>             (oauth2-make-access-request
>>>              token-url
>>>              (concat
>>> -             "client_id=" client-id
>>> +             "client_id=" (url-hexify-string client-id)
>>>  	     (when client-secret
>>> -               (concat  "&client_secret=" client-secret))
>>> -             "&code=" code
>>> +               (concat  "&client_secret=" (url-hexify-string client-secret)))
>>> +             "&code=" (url-hexify-string code)
>>>               "&redirect_uri=" (url-hexify-string (or redirect-uri "urn:ietf:wg:oauth:2.0:oob"))
>>>               "&grant_type=authorization_code"))))
>>>        (make-oauth2-token :client-id client-id
>>> -- 
>>> 2.39.2
>>
>> 1+  Though one might also consider using something like `url-encode-url'
>> to generate the entire URL.
>>
>
> It is indeed much cleaner.  Changed accordingly.

I am glad to hear that this was possible.

>>>
>>>
>>> From e801af578e63c7e333e668bdfef05e4cf0802582 Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Sun, 28 Jul 2024 03:00:04 -0700
>>> Subject: [PATCH 4/6] Support storing data for multiple accounts of the same
>>>  provider
>>>
>>> Currently the plstore id computed by `oauth2-compute-id' only takes
>>> `auth-url', `token-url', and `scope' into account, which could be the
>>> same for the same provider (e.g. Gmail).  This prevents storing
>>> information for multiple accounts of the same service for some
>>> providers.
>>>
>>> This patch adds `client-id' to the calculation of plstore id to make
>>> sure that it is unique for different accounts of the same provider.
>>>
>>> It also changes the hash function to sha512 to be more secure.
>>>
>>> * packages/oauth2/oauth2.el (oauth2-compute-id): add `client-id' as a
>>> parameter of `oauth2-compute-id' to ensure unique id amount multiple
>>> accounts of the same provider, and change hash function to sha512.
>>> ---
>>>  oauth2.el | 6 +++---
>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/oauth2.el b/oauth2.el
>>> index b035742fc1..035971ac85 100644
>>> --- a/oauth2.el
>>> +++ b/oauth2.el
>>> @@ -163,17 +163,17 @@ TOKEN should be obtained with `oauth2-request-access'."
>>>    :group 'oauth2
>>>    :type 'file)
>>>  
>>> -(defun oauth2-compute-id (auth-url token-url scope)
>>> +(defun oauth2-compute-id (auth-url token-url scope client-id)
>>>    "Compute an unique id based on URLs.
>>>  This allows to store the token in an unique way."
>>> -  (secure-hash 'md5 (concat auth-url token-url scope)))
>>> +  (secure-hash 'sha512 (concat auth-url token-url scope client-id)))
>>>  
>>>  ;;;###autoload
>>>  (defun oauth2-auth-and-store (auth-url token-url scope client-id client-secret &optional redirect-uri state)
>>>    "Request access to a resource and store it using `plstore'."
>>>    ;; We store a MD5 sum of all URL
>>>    (let* ((plstore (plstore-open oauth2-token-file))
>>> -         (id (oauth2-compute-id auth-url token-url scope))
>>> +         (id (oauth2-compute-id auth-url token-url scope client-id))
>>>           (plist (cdr (plstore-get plstore id))))
>>>      ;; Check if we found something matching this access
>>>      (if plist
>>> -- 
>>> 2.39.2
>>
>> LGTM
>>
>>>
>>> From 55417ec61c91f6b4d8e16a0c9933fb178d7bb657 Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Sun, 28 Jul 2024 03:41:20 -0700
>>> Subject: [PATCH 5/6] Add debug messages and provide a switch variable for
>>>  enabling
>>>
>>> This helps debugging whether the authorization and refresh requests
>>> were successful and inspecting the responses.
>>>
>>> * packages/oauth2/oauth2.el: add support for debug messages and a
>>> switch variable for enabling.
>>> ---
>>>  oauth2.el | 13 +++++++++++++
>>>  1 file changed, 13 insertions(+)
>>>
>>> diff --git a/oauth2.el b/oauth2.el
>>> index 035971ac85..ce7a835100 100644
>>> --- a/oauth2.el
>>> +++ b/oauth2.el
>>> @@ -40,6 +40,7 @@
>>>  (require 'plstore)
>>>  (require 'json)
>>>  (require 'url-http)
>>> +(require 'pp)
>>>  
>>>  (defvar url-http-data)
>>>  (defvar url-http-method)
>>> @@ -53,6 +54,14 @@
>>>    :link '(url-link :tag "Savannah" "https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/?h=externals/oauth2")
>>>    :link '(url-link :tag "ELPA" "https://elpa.gnu.org/packages/oauth2.html"))
>>>  
>>> +(defvar oauth2-debug nil
>>> +  "Enable debug messages.")
>>
>> I'd slightly expand this comment in case someone wants to use something
>> like `apropos-documentation'.
>
> Expanded the doc string to explain the purpose.
>
>> Also, why is this not a user option?
>>
>
> As Robert mentioned in the follow-up mail it was his suggestion (thanks
> Robert!)  Do let me know if `defcustom' is preferred.

No, some packages use user options, others regular variables so I guess
it is up to the person with the strongest opinion on the topic to decide
(which isn't me in this case).

>>> +
>>> +(defun oauth2--do-debug (&rest msg)
>>> +  "Output debug messages when `oauth2-debug' is enabled."
>>> +  (if oauth2-debug
>>
>> I'd use `when' here.  
>>
>
> Done
>
>>> +    (apply #'message msg)))
>>
>> ... and perhaps prefix the message with a string like "[oauth2] ".
>>
>
> Done.  I used `setcar' but let me know if there is a more elegant way to
> do this.

No complains from my side, I find it rather elegant!

>> I also notice that you manually mention the function issuing the debug
>> message below.  It is possible to infer this using `backtrace-frame'
>> (but that is really something that should be added to the core, instead
>> of packages having to re-implement it themselves.)
>>
>
> Added a let-bounded `func-name' by getting function name from
> backtrace-frame.  And it would be great if there is a built-in macro
> like `__PRETTY_FUNCTION__' and `__LINE__'.

I know of nothing.

>>> +
>>>  (defun oauth2-request-authorization (auth-url client-id &optional scope state redirect-uri)
>>>    "Request OAuth authorization at AUTH-URL by launching `browse-url'.
>>>  CLIENT-ID is the client id provided by the provider.
>>> @@ -79,6 +88,8 @@ It returns the code provided by the service."
>>>  
>>>  (defun oauth2-make-access-request (url data)
>>>    "Make an access request to URL using DATA in POST."
>>> +  (oauth2--do-debug "oauth2-make-access-request: url: %s" url)
>>> +  (oauth2--do-debug "oauth2-make-access-request: data: %s" data)
>>>    (let ((url-request-method "POST")
>>>          (url-request-data data)
>>>          (url-request-extra-headers
>>> @@ -86,6 +97,8 @@ It returns the code provided by the service."
>>>      (with-current-buffer (url-retrieve-synchronously url)
>>>        (let ((data (oauth2-request-access-parse)))
>>>          (kill-buffer (current-buffer))
>>> +        (oauth2--do-debug "oauth2-make-access-request: response: %s"
>>> +                          (pp-to-string data))
>>
>> Is pp-to-string here really much better than prin1-to-string?  Note that
>> 'oauth2--do-debug' is a function, so the arguments are always evaluated.
>> I have experienced pp being significantly slower than the core print
>> functions, so this is something that is worth thinking about IMO.
>>
>
> It's just a personal preference.  And it look like the values are mostly
> key-value pairs so they are not that hard to read through
> `prin1-to-string' so I changed to it.

1+

>>>          data))))
>>>  
>>>  (cl-defstruct oauth2-token
>>> -- 
>>> 2.39.2
>>>
>>>
>>> From e8735da21ac82b0698edad1796ddf4a1b8eb4bb2 Mon Sep 17 00:00:00 2001
>>> From: Xiyue Deng <manphiz@gmail.com>
>>> Date: Tue, 30 Jul 2024 03:46:57 -0700
>>> Subject: [PATCH 6/6] Add NEWS file to document the changes to plstore id
>>>  generation
>>>
>>> ---
>>>  NEWS | 23 +++++++++++++++++++++++
>>>  1 file changed, 23 insertions(+)
>>>  create mode 100644 NEWS
>>>
>>> diff --git a/NEWS b/NEWS
>>> new file mode 100644
>>> index 0000000000..6715a1914a
>>> --- /dev/null
>>> +++ b/NEWS
>>> @@ -0,0 +1,23 @@
>>> +Summary of changes to oauth2.el
>>> +-------------------------------
>>> +
>>> +For changes of 0.16 and older or full changes please check the git
>>> +history of the repository of oauth2.el.
>>> +
>>> +* 0.17
>>> +
>>> +** Changes to plstore id generation and needs to reacquire refresh_token
>>> +
>>> +The generation of plstore id used to include `auth-url', `token-url',
>>> +and `scope'.  Now `client-id' is also included.  This is required to
>>> +support multiple accounts of some providers which use the same
>>> +`auth-url', `token-url', and `scope' (e.g. Gmail), and hence the
>>> +generated plstore id is not unique amount accounts.  Adding
>>> +`client-id' solves this problem.
>>> +
>>> +The hash function of calculating the plstore id has also changed from
>>> +MD5 to SHA512 to be more secure.
>>> +
>>> +As a result, users of oauth2.el will need to redo the authentication
>>> +process to get a new refresh_token when upgrading from older version
>>> +to 0.17.
>>
>> Perhaps you can add a file local variable to enable outline-mode?
>>
>
> Done.  Also improved the wording.
>
>> All in all the patches look more than fine though, nothing I mentioned
>> is pressing.  If you don't have the time and the changes are urgent,
>> then I can also apply them as such.
>>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>> [...]
>>
>>> Friendly ping.
>>
>> Friendly pong.
>
> The updated patches are attached.  Thanks again!

OK, this looks very good.  I'll apply the changes and am closing the
report.

Thanks,
-- 
	Philip Kaludercic on peregrine

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> [..snip..]
>>
>> The updated patches are attached.  Thanks again!
>
> OK, this looks very good.  I'll apply the changes and am closing the
> report.
>
> Thanks,

Thanks a lot Philip!  One last thing: I'd like to ask for a new release
(0.17) with these changes if that's OK (patch attached.)  Or let me know
if I should create a new bug for that.

-- 
Xiyue Deng

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-packages-oauth2-oauth2.el-update-version-to-0.17.patch --]
[-- Type: text/x-diff, Size: 636 bytes --]

From e05c2bb372c712b809e31fe317ca993e63384207 Mon Sep 17 00:00:00 2001
From: Xiyue Deng <manphiz@gmail.com>
Date: Fri, 30 Aug 2024 01:28:09 -0700
Subject: [PATCH] * packages/oauth2/oauth2.el: update version to 0.17

---
 oauth2.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/oauth2.el b/oauth2.el
index 9c91bb08a1..87e0c39c5c 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -3,7 +3,7 @@
 ;; Copyright (C) 2011-2021 Free Software Foundation, Inc
 
 ;; Author: Julien Danjou <julien@danjou.info>
-;; Version: 0.16
+;; Version: 0.17
 ;; Keywords: comm
 ;; Package-Requires: ((cl-lib "0.5") (nadvice "0.3"))
 
-- 
2.39.2

bug#72358: 29.4; oauth2.el improvements

[Philip Kaludercic]

Xiyue Deng <manphiz@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> [..snip..]
>>>
>>> The updated patches are attached.  Thanks again!
>>
>> OK, this looks very good.  I'll apply the changes and am closing the
>> report.
>>
>> Thanks,
>
> Thanks a lot Philip!  One last thing: I'd like to ask for a new release
> (0.17) with these changes if that's OK (patch attached.)  

Of course, I forgot to ask if this should trigger a new release or not,
that's my bad.

>                                                           Or let me know
> if I should create a new bug for that.

Nah, not for something as simple as this.  I'll apply it straight away.

-- 
	Philip Kaludercic on peregrine

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Philip Kaludercic <philipk@posteo.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> [..snip..]
>>>>
>>>> The updated patches are attached.  Thanks again!
>>>
>>> OK, this looks very good.  I'll apply the changes and am closing the
>>> report.
>>>
>>> Thanks,
>>
>> Thanks a lot Philip!  One last thing: I'd like to ask for a new release
>> (0.17) with these changes if that's OK (patch attached.)  
>
> Of course, I forgot to ask if this should trigger a new release or not,
> that's my bad.
>
>>                                                           Or let me know
>> if I should create a new bug for that.
>
> Nah, not for something as simple as this.  I'll apply it straight away.

Thanks very much, Philip!

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

In case anyone is interested, I have filed bug#72992[1] where I posted
my code to enable xoauth2 support for nnimap and smtpmail as I promised.
I am also trying to gather feedback on how it can be improved.  Comments
welcome!

[1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2024-09/msg00089.html
-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>
>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>> regularly re-prompt, so this will not be too surprising (although
>> maybe call it out in the packageʼs NEWS or README).
>
> In many cases the refreshing of tokens is transparent to the user there
> doesn't have to be a re-prompt to refresh the token if the OAuth
> provider support it.
> Micrsofts OAuth workflow is quite good in this regard as there's a
> non-standard error to indicate when the user has to re-authorize the
> application.
>

Actually I am currently having trouble for a few weeks to get my
outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
have documented the issues and steps I have tried in this stackoverflow
question[1].  I would great appreciated it if you can shed some lights
there

> I assume all implementation of OAuth have their quirks.

Indeed.


[1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>
>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>> regularly re-prompt, so this will not be too surprising (although
>>> maybe call it out in the packageʼs NEWS or README).
>>
>> In many cases the refreshing of tokens is transparent to the user there
>> doesn't have to be a re-prompt to refresh the token if the OAuth
>> provider support it.
>> Micrsofts OAuth workflow is quite good in this regard as there's a
>> non-standard error to indicate when the user has to re-authorize the
>> application.
>>
>
> Actually I am currently having trouble for a few weeks to get my
> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
> have documented the issues and steps I have tried in this stackoverflow
> question[1].  I would great appreciated it if you can shed some lights
> there
>

I remember when adding OAuth support to Sailfish OS we needed to patch
our signon to work with the non-standard Microsoft flow.

We have this patch on top of the OAuth2 plugin for signond to accept
that they send the authentication as a request for you to fetch upon you
have to another request with the new url to get authentication code.

The patch can be found here:
https://github.com/sailfishos/signon-plugin-oauth2/blob/master/rpm/0005-Support-Microsoft-OAuth2-flow.patch

I'm not the person who wrote the patch but I fixed the plugin later for
Dropbox so that PKCE (RFC7636) isn't used unless the server requests it
(response type must be code to request PKCE).

PKCE is strongly recommended. The patch above already works kinda that
way without requesting PKCE.

Read here for further information:
https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-auth-code-flow
https://datatracker.ietf.org/doc/html/rfc7636

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>
>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>> regularly re-prompt, so this will not be too surprising (although
>>> maybe call it out in the packageʼs NEWS or README).
>>
>> In many cases the refreshing of tokens is transparent to the user there
>> doesn't have to be a re-prompt to refresh the token if the OAuth
>> provider support it.
>> Micrsofts OAuth workflow is quite good in this regard as there's a
>> non-standard error to indicate when the user has to re-authorize the
>> application.
>>
>
> Actually I am currently having trouble for a few weeks to get my
> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
> have documented the issues and steps I have tried in this stackoverflow
> question[1].  I would great appreciated it if you can shed some lights
> there
>
>> I assume all implementation of OAuth have their quirks.
>
> Indeed.
>
>
> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo

Just want to report back that after confirming with an MS representative
through online chat, outlook.com has actually disabled refreshing
access_token through the token endpoint, and users are asked to migrate
to Outlook app or compatibles apps (Thunderbird still works).  I'm not
sure whether this is also the case for organization emails, which may
also be disabled by default (or soonish if not already) but can be
enabled separately by an org admin.  Anyway, I'd suggest people stop
wasting your time here and use Gmail (or maybe Yahoo mail) which has
decent 3rd party OAuth2 support.

Meanwhile I have submitted a request to re-enable this support[1].

[1] https://feedbackportal.microsoft.com/feedback/idea/069f1816-0a55-ef11-b4ad-0022484d3ecc

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Robert Pluim <rpluim@gmail.com> writes:
>>>
>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>
>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>> regularly re-prompt, so this will not be too surprising (although
>>>> maybe call it out in the packageʼs NEWS or README).
>>>
>>> In many cases the refreshing of tokens is transparent to the user there
>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>> provider support it.
>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>> non-standard error to indicate when the user has to re-authorize the
>>> application.
>>>
>>
>> Actually I am currently having trouble for a few weeks to get my
>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>> have documented the issues and steps I have tried in this stackoverflow
>> question[1].  I would great appreciated it if you can shed some lights
>> there
>>
>>> I assume all implementation of OAuth have their quirks.
>>
>> Indeed.
>>
>>
>> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>
> Just want to report back that after confirming with an MS representative
> through online chat, outlook.com has actually disabled refreshing
> access_token through the token endpoint, and users are asked to migrate
> to Outlook app or compatibles apps (Thunderbird still works).

Thank you for notifying me on this I will forward this to my employer.

> I'm not sure whether this is also the case for organization emails, which may
> also be disabled by default (or soonish if not already) but can be
> enabled separately by an org admin.

It does depend some domains use whitelist e.g. Tampere University of
Applies sciences. Without a specific Emacs GNUs/Caldav/whatever AppID
inside Microsoft OAuth2 it will be hard to pass that.


> Anyway, I'd suggest people stop
> wasting your time here and use Gmail (or maybe Yahoo mail) which has
> decent 3rd party OAuth2 support.

I don't think that's an option for most user that complain about working
OAuth2 support, in most cases it's a work or some other organization
account.

Another thing I think is very important is to support Nextcloud as it's
a FOSS app supporting OAuth2 which quite many users and organizations
adopted.


> Meanwhile I have submitted a request to re-enable this support[1].
>
> [1] https://feedbackportal.microsoft.com/feedback/idea/069f1816-0a55-ef11-b4ad-0022484d3ecc

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Robert Pluim <rpluim@gmail.com> writes:
>>>
>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>
>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>> regularly re-prompt, so this will not be too surprising (although
>>>> maybe call it out in the packageʼs NEWS or README).
>>>
>>> In many cases the refreshing of tokens is transparent to the user there
>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>> provider support it.
>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>> non-standard error to indicate when the user has to re-authorize the
>>> application.
>>>
>>
>> Actually I am currently having trouble for a few weeks to get my
>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>> have documented the issues and steps I have tried in this stackoverflow
>> question[1].  I would great appreciated it if you can shed some lights
>> there
>>
>>> I assume all implementation of OAuth have their quirks.
>>
>> Indeed.
>>
>>
>> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>
> Just want to report back that after confirming with an MS representative
> through online chat, outlook.com has actually disabled refreshing
> access_token through the token endpoint, and users are asked to migrate
> to Outlook app or compatibles apps (Thunderbird still works).  I'm not
> sure whether this is also the case for organization emails, which may
> also be disabled by default (or soonish if not already) but can be
> enabled separately by an org admin.  Anyway, I'd suggest people stop
> wasting your time here and use Gmail (or maybe Yahoo mail) which has
> decent 3rd party OAuth2 support.

Can you link a source about that?

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>
>>>> Robert Pluim <rpluim@gmail.com> writes:
>>>>
>>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>>
>>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>>> regularly re-prompt, so this will not be too surprising (although
>>>>> maybe call it out in the packageʼs NEWS or README).
>>>>
>>>> In many cases the refreshing of tokens is transparent to the user there
>>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>>> provider support it.
>>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>>> non-standard error to indicate when the user has to re-authorize the
>>>> application.
>>>>
>>>
>>> Actually I am currently having trouble for a few weeks to get my
>>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>>> have documented the issues and steps I have tried in this stackoverflow
>>> question[1].  I would great appreciated it if you can shed some lights
>>> there
>>>
>>>> I assume all implementation of OAuth have their quirks.
>>>
>>> Indeed.
>>>
>>>
>>> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>>
>> Just want to report back that after confirming with an MS representative
>> through online chat, outlook.com has actually disabled refreshing
>> access_token through the token endpoint, and users are asked to migrate
>> to Outlook app or compatibles apps (Thunderbird still works).
>
> Thank you for notifying me on this I will forward this to my employer.
>
>> I'm not sure whether this is also the case for organization emails, which may
>> also be disabled by default (or soonish if not already) but can be
>> enabled separately by an org admin.
>
> It does depend some domains use whitelist e.g. Tampere University of
> Applies sciences. Without a specific Emacs GNUs/Caldav/whatever AppID
> inside Microsoft OAuth2 it will be hard to pass that.
>
>
>> Anyway, I'd suggest people stop
>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>> decent 3rd party OAuth2 support.
>
> I don't think that's an option for most user that complain about working
> OAuth2 support, in most cases it's a work or some other organization
> account.
>
> Another thing I think is very important is to support Nextcloud as it's
> a FOSS app supporting OAuth2 which quite many users and organizations
> adopted.
>
>

Nextcloud sounds interesting.  Do you know where I can check for the
OAuth2 credentials like client_id and client_secret?

>> Meanwhile I have submitted a request to re-enable this support[1].
>>
>> [1] https://feedbackportal.microsoft.com/feedback/idea/069f1816-0a55-ef11-b4ad-0022484d3ecc

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>> Anyway, I'd suggest people stop
>>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>>> decent 3rd party OAuth2 support.
>>
>> I don't think that's an option for most user that complain about working
>> OAuth2 support, in most cases it's a work or some other organization
>> account.
>>
>> Another thing I think is very important is to support Nextcloud as it's
>> a FOSS app supporting OAuth2 which quite many users and organizations
>> adopted.
>>
>>
>
> Nextcloud sounds interesting.  Do you know where I can check for the
> OAuth2 credentials like client_id and client_secret?
>

You can find them in the administrator settings the Nextcloud server.
Here is there documentation:
https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/oauth2.html

bug#72358: 29.4; oauth2.el improvements

[Thomas Fitzsimmons]

Xiyue Deng <manphiz@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>>
>>>>> Robert Pluim <rpluim@gmail.com> writes:
>>>>>
>>>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>>>
>>>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>>>> regularly re-prompt, so this will not be too surprising (although
>>>>>> maybe call it out in the packageʼs NEWS or README).
>>>>>
>>>>> In many cases the refreshing of tokens is transparent to the user there
>>>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>>>> provider support it.
>>>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>>>> non-standard error to indicate when the user has to re-authorize the
>>>>> application.
>>>>>
>>>>
>>>> Actually I am currently having trouble for a few weeks to get my
>>>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>>>> have documented the issues and steps I have tried in this stackoverflow
>>>> question[1].  I would great appreciated it if you can shed some lights
>>>> there
>>>>
>>>>> I assume all implementation of OAuth have their quirks.
>>>>
>>>> Indeed.
>>>>
>>>>
>>>> [1]
>>>> https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>>>
>>> Just want to report back that after confirming with an MS representative
>>> through online chat, outlook.com has actually disabled refreshing
>>> access_token through the token endpoint, and users are asked to migrate
>>> to Outlook app or compatibles apps (Thunderbird still works).
>>
>> Thank you for notifying me on this I will forward this to my employer.
>>
>>> I'm not sure whether this is also the case for organization emails, which may
>>> also be disabled by default (or soonish if not already) but can be
>>> enabled separately by an org admin.
>>
>> It does depend some domains use whitelist e.g. Tampere University of
>> Applies sciences. Without a specific Emacs GNUs/Caldav/whatever AppID
>> inside Microsoft OAuth2 it will be hard to pass that.
>>
>>
>>> Anyway, I'd suggest people stop
>>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>>> decent 3rd party OAuth2 support.
>>
>> I don't think that's an option for most user that complain about working
>> OAuth2 support, in most cases it's a work or some other organization
>> account.
>>
>> Another thing I think is very important is to support Nextcloud as it's
>> a FOSS app supporting OAuth2 which quite many users and organizations
>> adopted.
>>
>>
>
> Nextcloud sounds interesting.  Do you know where I can check for the
> OAuth2 credentials like client_id and client_secret?

sourcehut [1] provides a Free Software OAuth2 flow, and it has the
benefit of not requiring JavaScript (even FOSS JavaScript) anywhere in
the process.  I wrote url-http-oauth-demo.el [2] as a complete "worked"
example demonstrating its use with url-http-oauth.el.

Thomas

1. https://sourcehut.org/
2. https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/url-http-oauth-demo.el?h=externals/url-http-oauth

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>>
>>>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>>>
>>>>>> Robert Pluim <rpluim@gmail.com> writes:
>>>>>>
>>>>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>>>>
>>>>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>>>>> regularly re-prompt, so this will not be too surprising (although
>>>>>>> maybe call it out in the packageʼs NEWS or README).
>>>>>>
>>>>>> In many cases the refreshing of tokens is transparent to the user there
>>>>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>>>>> provider support it.
>>>>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>>>>> non-standard error to indicate when the user has to re-authorize the
>>>>>> application.
>>>>>>
>>>>>
>>>>> Actually I am currently having trouble for a few weeks to get my
>>>>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>>>>> have documented the issues and steps I have tried in this stackoverflow
>>>>> question[1].  I would great appreciated it if you can shed some lights
>>>>> there
>>>>>
>>>>>> I assume all implementation of OAuth have their quirks.
>>>>>
>>>>> Indeed.
>>>>>
>>>>>
>>>>> [1]
>>>>> https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>>>>
>>>> Just want to report back that after confirming with an MS representative
>>>> through online chat, outlook.com has actually disabled refreshing
>>>> access_token through the token endpoint, and users are asked to migrate
>>>> to Outlook app or compatibles apps (Thunderbird still works).
>>>
>>> Thank you for notifying me on this I will forward this to my employer.
>>>
>>>> I'm not sure whether this is also the case for organization emails, which may
>>>> also be disabled by default (or soonish if not already) but can be
>>>> enabled separately by an org admin.
>>>
>>> It does depend some domains use whitelist e.g. Tampere University of
>>> Applies sciences. Without a specific Emacs GNUs/Caldav/whatever AppID
>>> inside Microsoft OAuth2 it will be hard to pass that.
>>>
>>>
>>>> Anyway, I'd suggest people stop
>>>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>>>> decent 3rd party OAuth2 support.
>>>
>>> I don't think that's an option for most user that complain about working
>>> OAuth2 support, in most cases it's a work or some other organization
>>> account.
>>>
>>> Another thing I think is very important is to support Nextcloud as it's
>>> a FOSS app supporting OAuth2 which quite many users and organizations
>>> adopted.
>>>
>>>
>>
>> Nextcloud sounds interesting.  Do you know where I can check for the
>> OAuth2 credentials like client_id and client_secret?
>
> sourcehut [1] provides a Free Software OAuth2 flow, and it has the
> benefit of not requiring JavaScript (even FOSS JavaScript) anywhere in
> the process.  I wrote url-http-oauth-demo.el [2] as a complete "worked"
> example demonstrating its use with url-http-oauth.el.

Would that provide OAuth2 for providers that require a login through
their webinterface, such as Nextcloud Login, without a browser?
Most platforms such as Android, KDE or Sailfish OS use a browser for
OAuth2 login to login, authorize and then forward the token to the
OS/app.


> Thomas
>
> 1. https://sourcehut.org/
> 2. https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/url-http-oauth-demo.el?h=externals/url-http-oauth

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>>
>>>> Robert Pluim <rpluim@gmail.com> writes:
>>>>
>>>>>     Xiyue> - This will invalidate all existing entries and a user will have to redo
>>>>>     Xiyue>   the authorization process again to get a new refresh token.  However,
>>>>>     Xiyue>   I think it's more important to ensure that oauth2.el works correctly
>>>>>     Xiyue>   for multiple accounts of the same provider, or a user may suffer from
>>>>>     Xiyue>   confusion when adding a new account invalidates a previous account.
>>>>>
>>>>> I donʼt think thatʼs too big a concern. 'modern' authentication flows
>>>>> regularly re-prompt, so this will not be too surprising (although
>>>>> maybe call it out in the packageʼs NEWS or README).
>>>>
>>>> In many cases the refreshing of tokens is transparent to the user there
>>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>>> provider support it.
>>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>>> non-standard error to indicate when the user has to re-authorize the
>>>> application.
>>>>
>>>
>>> Actually I am currently having trouble for a few weeks to get my
>>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>>> have documented the issues and steps I have tried in this stackoverflow
>>> question[1].  I would great appreciated it if you can shed some lights
>>> there
>>>
>>>> I assume all implementation of OAuth have their quirks.
>>>
>>> Indeed.
>>>
>>>
>>> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>>
>> Just want to report back that after confirming with an MS representative
>> through online chat, outlook.com has actually disabled refreshing
>> access_token through the token endpoint, and users are asked to migrate
>> to Outlook app or compatibles apps (Thunderbird still works).  I'm not
>> sure whether this is also the case for organization emails, which may
>> also be disabled by default (or soonish if not already) but can be
>> enabled separately by an org admin.  Anyway, I'd suggest people stop
>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>> decent 3rd party OAuth2 support.
>
> Can you link a source about that?

Unfortunately the MS representative didn't provide any link for this.

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

>>>>> In many cases the refreshing of tokens is transparent to the user there
>>>>> doesn't have to be a re-prompt to refresh the token if the OAuth
>>>>> provider support it.
>>>>> Micrsofts OAuth workflow is quite good in this regard as there's a
>>>>> non-standard error to indicate when the user has to re-authorize the
>>>>> application.
>>>>>
>>>>
>>>> Actually I am currently having trouble for a few weeks to get my
>>>> outlook.com email work with MS OAuth2.  To avoid some repeated typing, I
>>>> have documented the issues and steps I have tried in this stackoverflow
>>>> question[1].  I would great appreciated it if you can shed some lights
>>>> there
>>>>
>>>>> I assume all implementation of OAuth have their quirks.
>>>>
>>>> Indeed.
>>>>
>>>>
>>>> [1] https://stackoverflow.com/questions/78787763/getting-aadsts65001-error-invalid-grant-when-trying-to-refresh-access-token-fo
>>>
>>> Just want to report back that after confirming with an MS representative
>>> through online chat, outlook.com has actually disabled refreshing
>>> access_token through the token endpoint, and users are asked to migrate
>>> to Outlook app or compatibles apps (Thunderbird still works).  I'm not
>>> sure whether this is also the case for organization emails, which may
>>> also be disabled by default (or soonish if not already) but can be
>>> enabled separately by an org admin.  Anyway, I'd suggest people stop
>>> wasting your time here and use Gmail (or maybe Yahoo mail) which has
>>> decent 3rd party OAuth2 support.
>>
>> Can you link a source about that?
>
> Unfortunately the MS representative didn't provide any link for this.

OK that is to bad, if possible I would ask for an official
response. Microsoft counts as a gate keeper, at least officially in the
EU, they should not be able to act like this.

A workaround would be to request a manual token refresh by authorizing
the application all over again.

I will update you on anything new I have if I get new information that I
can post.

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> The fourth patch may need a bit of background: oauth2.el (optionally)
> uses plstore to save authentication data for future reuse, and the
> plstore id for an account is computed using a combination of `auth-url',
> `token-url', and `scope'.  However, this combination of data doesn't
> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
> the three parameters are the same for different accounts, and hence
> storing a second account information will override the first one.

Would it make sense to plug OAuth2.el into auth-source to store the
authentication token safely inside an existing credential storage?

Various applications already do so when using the native credential
storages such as Freedesktop.org or the macOS keyring.

bug#72358: 29.4; oauth2.el improvements

[Robert Pluim]

>>>>> On Tue, 30 Jul 2024 17:08:21 +0300, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> said:

    Björn> Xiyue Deng <manphiz@gmail.com> writes:
    >> The fourth patch may need a bit of background: oauth2.el (optionally)
    >> uses plstore to save authentication data for future reuse, and the
    >> plstore id for an account is computed using a combination of `auth-url',
    >> `token-url', and `scope'.  However, this combination of data doesn't
    >> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
    >> the three parameters are the same for different accounts, and hence
    >> storing a second account information will override the first one.

    Björn> Would it make sense to plug OAuth2.el into auth-source to store the
    Björn> authentication token safely inside an existing credential storage?

    Björn> Various applications already do so when using the native credential
    Björn> storages such as Freedesktop.org or the macOS keyring.

Yes. In fact thereʼs the auth-source-xoauth2 package that does
that. And oauth2 can already store stuff using plstore, so Iʼm sure it
can be extended to use auth-source.


Robert
-- 

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 30 Jul 2024 17:08:21 +0300, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> said:
>
>     Björn> Xiyue Deng <manphiz@gmail.com> writes:
>     >> The fourth patch may need a bit of background: oauth2.el (optionally)
>     >> uses plstore to save authentication data for future reuse, and the
>     >> plstore id for an account is computed using a combination of `auth-url',
>     >> `token-url', and `scope'.  However, this combination of data doesn't
>     >> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>     >> the three parameters are the same for different accounts, and hence
>     >> storing a second account information will override the first one.
>
>     Björn> Would it make sense to plug OAuth2.el into auth-source to store the
>     Björn> authentication token safely inside an existing credential storage?
>
>     Björn> Various applications already do so when using the native credential
>     Björn> storages such as Freedesktop.org or the macOS keyring.
>
> Yes. In fact thereʼs the auth-source-xoauth2 package that does
> that. And oauth2 can already store stuff using plstore, so Iʼm sure it
> can be extended to use auth-source.
>

auth-source-xoauth2 doesn't actually use auth-source
(e.g. ~/.authinfo.gpg) to store the data it needs, but use a custom file
storing an ELisp hash table to store the client-id, client-secret, etc.
It does advice the authentication code to use the calculated token.

>
> Robert

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Thomas Fitzsimmons]

Xiyue Deng <manphiz@gmail.com> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>>>>>>> On Tue, 30 Jul 2024 17:08:21 +0300, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> said:
>>
>>     Björn> Xiyue Deng <manphiz@gmail.com> writes:
>>     >> The fourth patch may need a bit of background: oauth2.el (optionally)
>>     >> uses plstore to save authentication data for future reuse, and the
>>     >> plstore id for an account is computed using a combination of `auth-url',
>>     >> `token-url', and `scope'.  However, this combination of data doesn't
>>     >> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>>     >> the three parameters are the same for different accounts, and hence
>>     >> storing a second account information will override the first one.
>>
>>     Björn> Would it make sense to plug OAuth2.el into auth-source to store the
>>     Björn> authentication token safely inside an existing credential storage?
>>
>>     Björn> Various applications already do so when using the native credential
>>     Björn> storages such as Freedesktop.org or the macOS keyring.
>>
>> Yes. In fact thereʼs the auth-source-xoauth2 package that does
>> that. And oauth2 can already store stuff using plstore, so Iʼm sure it
>> can be extended to use auth-source.
>>
>
> auth-source-xoauth2 doesn't actually use auth-source
> (e.g. ~/.authinfo.gpg) to store the data it needs, but use a custom file
> storing an ELisp hash table to store the client-id, client-secret, etc.
> It does advice the authentication code to use the calculated token.

I have not seen it mentioned in this thread yet, so here goes: my
url-http-oauth package in GNU ELPA supports storing credentials in
~/.authinfo.gpg and refreshing them.  It would be nice if your OAuth2
work could get feature parity with it, then I could delete my package;
feel free to copy any code that makes sense.  (I do not use
url-http-oauth anymore, but I felt the need to write it when I was using
Excorporate and OAuth.)

Ideally you could get the result (and the xoauth2 support for IMAP and
SMTP) accepted in Emacs core.

(Then, extremely ideally, the FSF could work out legal agreements with
the various OAuth providers to get Emacs registered as an OAuth
application, like, e.g., Thunderbird.)

Thomas

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Hi Thomas,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>>>>>> On Tue, 30 Jul 2024 17:08:21 +0300, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> said:
>>>
>>>     Björn> Xiyue Deng <manphiz@gmail.com> writes:
>>>     >> The fourth patch may need a bit of background: oauth2.el (optionally)
>>>     >> uses plstore to save authentication data for future reuse, and the
>>>     >> plstore id for an account is computed using a combination of `auth-url',
>>>     >> `token-url', and `scope'.  However, this combination of data doesn't
>>>     >> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>>>     >> the three parameters are the same for different accounts, and hence
>>>     >> storing a second account information will override the first one.
>>>
>>>     Björn> Would it make sense to plug OAuth2.el into auth-source to store the
>>>     Björn> authentication token safely inside an existing credential storage?
>>>
>>>     Björn> Various applications already do so when using the native credential
>>>     Björn> storages such as Freedesktop.org or the macOS keyring.
>>>
>>> Yes. In fact thereʼs the auth-source-xoauth2 package that does
>>> that. And oauth2 can already store stuff using plstore, so Iʼm sure it
>>> can be extended to use auth-source.
>>>
>>
>> auth-source-xoauth2 doesn't actually use auth-source
>> (e.g. ~/.authinfo.gpg) to store the data it needs, but use a custom file
>> storing an ELisp hash table to store the client-id, client-secret, etc.
>> It does advice the authentication code to use the calculated token.
>
> I have not seen it mentioned in this thread yet, so here goes: my
> url-http-oauth package in GNU ELPA supports storing credentials in
> ~/.authinfo.gpg and refreshing them.  It would be nice if your OAuth2
> work could get feature parity with it, then I could delete my package;
> feel free to copy any code that makes sense.  (I do not use
> url-http-oauth anymore, but I felt the need to write it when I was using
> Excorporate and OAuth.)
>

Thanks for working on url-http-oauth!  I think it adds credential
management using auth-source, e.g. prompt for client-id and
client-secret and store them, which my other addon (that I'll post next
as it depends on the changes I made here) didn't do.  Ideally this
should be handled transparently by all auth-source backends and say Gnus
when you add a new account, but IIUC currently the JSON backend doesn't
support creation, which I'm using for ease to read and modify.

> Ideally you could get the result (and the xoauth2 support for IMAP and
> SMTP) accepted in Emacs core.
>

That would be great!  My other addon uses advice, but it would
definitely be better to be integrated in core (which already has partial
support)

> (Then, extremely ideally, the FSF could work out legal agreements with
> the various OAuth providers to get Emacs registered as an OAuth
> application, like, e.g., Thunderbird.)
>

That would be the best for the end user.  Imagine a Gnus user could just
add a new account and on launch Gnus the default browser will open the
login page (or be prompted an URL to visit), which then normally handles
all the login shenanigans (2FA, authenticator, etc.) and viola, you're
logged in.

> Thomas

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Thomas,
>
> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Robert Pluim <rpluim@gmail.com> writes:
>>>
>>>>>>>>> On Tue, 30 Jul 2024 17:08:21 +0300, Björn Bidar via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> said:
>>>>
>>>>     Björn> Xiyue Deng <manphiz@gmail.com> writes:
>>>>     >> The fourth patch may need a bit of background: oauth2.el (optionally)
>>>>     >> uses plstore to save authentication data for future reuse, and the
>>>>     >> plstore id for an account is computed using a combination of `auth-url',
>>>>     >> `token-url', and `scope'.  However, this combination of data doesn't
>>>>     >> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>>>>     >> the three parameters are the same for different accounts, and hence
>>>>     >> storing a second account information will override the first one.
>>>>
>>>>     Björn> Would it make sense to plug OAuth2.el into auth-source to store the
>>>>     Björn> authentication token safely inside an existing credential storage?
>>>>
>>>>     Björn> Various applications already do so when using the native credential
>>>>     Björn> storages such as Freedesktop.org or the macOS keyring.
>>>>
>>>> Yes. In fact thereʼs the auth-source-xoauth2 package that does
>>>> that. And oauth2 can already store stuff using plstore, so Iʼm sure it
>>>> can be extended to use auth-source.
>>>>
>>>
>>> auth-source-xoauth2 doesn't actually use auth-source
>>> (e.g. ~/.authinfo.gpg) to store the data it needs, but use a custom file
>>> storing an ELisp hash table to store the client-id, client-secret, etc.
>>> It does advice the authentication code to use the calculated token.
>>
>> I have not seen it mentioned in this thread yet, so here goes: my
>> url-http-oauth package in GNU ELPA supports storing credentials in
>> ~/.authinfo.gpg and refreshing them.  It would be nice if your OAuth2
>> work could get feature parity with it, then I could delete my package;
>> feel free to copy any code that makes sense.  (I do not use
>> url-http-oauth anymore, but I felt the need to write it when I was using
>> Excorporate and OAuth.)
>>
>
> Thanks for working on url-http-oauth!  I think it adds credential
> management using auth-source, e.g. prompt for client-id and
> client-secret and store them, which my other addon (that I'll post next
> as it depends on the changes I made here) didn't do.  Ideally this
> should be handled transparently by all auth-source backends and say Gnus
> when you add a new account, but IIUC currently the JSON backend doesn't
> support creation, which I'm using for ease to read and modify.


I think depending on the authentication storage it would make sense to
provide a custom setting with a sane default that allows the user to
configure a path inside the storage to store their credentials.
These could then have templates for the hostname, user, port etc just
like the regular search parameters that `auth-sources-search' supports.

>> Ideally you could get the result (and the xoauth2 support for IMAP and
>> SMTP) accepted in Emacs core.
>>
>
> That would be great!  My other addon uses advice, but it would
> definitely be better to be integrated in core (which already has partial
> support)
>

From Gnus point of view I think it would make it easier to support
auth-source as then Oauth support would come for free as then the
credentials can be retrieved just for basic authentication.

Some kind of hook or trigger to call renewal functions would be needed
though so that tokens can be refreshed transparently or manually if the
user needs to re-authenticate.

>> (Then, extremely ideally, the FSF could work out legal agreements with
>> the various OAuth providers to get Emacs registered as an OAuth
>> application, like, e.g., Thunderbird.)
>>
>
> That would be the best for the end user.  Imagine a Gnus user could just
> add a new account and on launch Gnus the default browser will open the
> login page (or be prompted an URL to visit), which then normally handles
> all the login shenanigans (2FA, authenticator, etc.) and viola, you're
> logged in.
>

To allow seamless authentication the FSF could make things much smoother
by registering Emacs with common providers, working with FSFE together
in this regard could help too (there are other countries than the U.S.).

The issue I see is that registering Emacs is not enough since Emacs is
in this context merely a runtime for these modes that want to use OAuth.
The scope of the permissions they need varies, so their names etc.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> The fourth patch may need a bit of background: oauth2.el (optionally)
>> uses plstore to save authentication data for future reuse, and the
>> plstore id for an account is computed using a combination of `auth-url',
>> `token-url', and `scope'.  However, this combination of data doesn't
>> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>> the three parameters are the same for different accounts, and hence
>> storing a second account information will override the first one.
>
> Would it make sense to plug OAuth2.el into auth-source to store the
> authentication token safely inside an existing credential storage?
>
> Various applications already do so when using the native credential
> storages such as Freedesktop.org or the macOS keyring.

As I mentioned to Robert, I do have another addon to do exactly this,
though through an awkward advice.  Would be great if auth-source can
make use of oauth2.el and handle that more gracefully.  I'll file
another bug to explore options once this one is done.

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Xiyue Deng <manphiz@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> The fourth patch may need a bit of background: oauth2.el (optionally)
>>> uses plstore to save authentication data for future reuse, and the
>>> plstore id for an account is computed using a combination of `auth-url',
>>> `token-url', and `scope'.  However, this combination of data doesn't
>>> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>>> the three parameters are the same for different accounts, and hence
>>> storing a second account information will override the first one.
>>
>> Would it make sense to plug OAuth2.el into auth-source to store the
>> authentication token safely inside an existing credential storage?
>>
>> Various applications already do so when using the native credential
>> storages such as Freedesktop.org or the macOS keyring.
>
> As I mentioned to Robert, I do have another addon to do exactly this,
> though through an awkward advice.  Would be great if auth-source can
> make use of oauth2.el and handle that more gracefully.  I'll file
> another bug to explore options once this one is done.

Care to post this advice? It's not an optimal solution but better than
nothing in the interim.

bug#72358: 29.4; oauth2.el improvements

[Xiyue Deng]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> The fourth patch may need a bit of background: oauth2.el (optionally)
>>>> uses plstore to save authentication data for future reuse, and the
>>>> plstore id for an account is computed using a combination of `auth-url',
>>>> `token-url', and `scope'.  However, this combination of data doesn't
>>>> guarantee uniqueness for accounts for a same provider, e.g. for Gmail,
>>>> the three parameters are the same for different accounts, and hence
>>>> storing a second account information will override the first one.
>>>
>>> Would it make sense to plug OAuth2.el into auth-source to store the
>>> authentication token safely inside an existing credential storage?
>>>
>>> Various applications already do so when using the native credential
>>> storages such as Freedesktop.org or the macOS keyring.
>>
>> As I mentioned to Robert, I do have another addon to do exactly this,
>> though through an awkward advice.  Would be great if auth-source can
>> make use of oauth2.el and handle that more gracefully.  I'll file
>> another bug to explore options once this one is done.
>
> Care to post this advice? It's not an optimal solution but better than
> nothing in the interim.

It currently depends on this modified oauth2.el.  Once the patches are
accepted I'll post it in a separate bug.

-- 
Xiyue Deng

bug#72358: 29.4; oauth2.el improvements

[Andrew Cohen]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

I have been using the existing oauth2.el and auth-source.el to use both
gmail and outlook (through my university) with oauth2 for several years
now (I posted a bit about it some time ago on the devel list). I
didn't need to change much to get it to work so I thought as long as the
changes by Xiyue are being considered (all of which look good to me) I
would chime in. I'm happy to provide more info about my setup and usage
if anyone is interested.

Firstly, I note that I have gmail working fine without the change in
patchset 2 (although I see nothing wrong with the change, I wonder why
it isn't necessary for me but is for Xiyue).

Secondly, there is one other important change that I have been using
which should probably be added to oauth2.el (I communicated the change
to Julien a long time ago, but he said he is no longer actively
maintaining oauth2.el): in refreshing the token the access-response is
ignored (as is the response-error). The access-response contains
information about the token expiration so its needed in order to control
when to fetch a new token. The simple patch below stores the
access-response in the appropriate slot in the token:


[-- Attachment #2: store access-response on refresh --]
[-- Type: application/octet-stream, Size: 3101 bytes --]

diff --git a/oauth2.el b/oauth2.el
index 7da9702..e460b01 100644
--- a/oauth2.el
+++ b/oauth2.el
@@ -119,28 +119,35 @@ Return an `oauth2-token' structure."
 (defun oauth2-refresh-access (token)
   "Refresh OAuth access TOKEN.
 TOKEN should be obtained with `oauth2-request-access'."
-  (setf (oauth2-token-access-token token)
-        (cdr (assoc 'access_token
-                    (oauth2-make-access-request
-                     (oauth2-token-token-url token)
-                     (concat "client_id=" (oauth2-token-client-id token)
-			     (when (oauth2-token-client-secret token)
-                               (concat "&client_secret=" (oauth2-token-client-secret token)))
-                             "&refresh_token=" (oauth2-token-refresh-token token)
-                             "&grant_type=refresh_token")))))
-  ;; If the token has a plstore, update it
-  (let ((plstore (oauth2-token-plstore token)))
-    (when plstore
-      (plstore-put plstore (oauth2-token-plstore-id token)
-                   nil `(:access-token
-                         ,(oauth2-token-access-token token)
-                         :refresh-token
-                         ,(oauth2-token-refresh-token token)
-                         :access-response
-                         ,(oauth2-token-access-response token)
-                         ))
-      (plstore-save plstore)))
-  token)
+  (let ((response (oauth2-make-access-request
+                    (oauth2-token-token-url token)
+                    (concat "client_id=" (oauth2-token-client-id token)
+			    (when (oauth2-token-client-secret token)
+                              (concat "&client_secret=" (oauth2-token-client-secret token)))
+                            "&refresh_token=" (oauth2-token-refresh-token token)
+                            "&grant_type=refresh_token"))))
+    (if-let ((response-error (cdr (assoc 'error response))))
+        (message "oauth2 token refresh error: %s" response-error)
+      (setf (oauth2-token-access-token token)
+            (cdr (assoc 'access_token response)))
+      (when-let ((refresh_token (cdr (assoc 'refresh_token response))))
+        (setf (oauth2-token-refresh-token token) refresh_token))
+      (setf (oauth2-token-access-response token)
+            (assoc-delete-all 'refresh_token
+                              (assoc-delete-all 'access_token response)))
+      ;; If the token has a plstore, update it
+      (let ((plstore (oauth2-token-plstore token)))
+        (when plstore
+          (plstore-put plstore (oauth2-token-plstore-id token)
+                       nil `(:access-token
+                             ,(oauth2-token-access-token token)
+                             :refresh-token
+                             ,(oauth2-token-refresh-token token)
+                             :access-response
+                             ,(oauth2-token-access-response token)
+                             ))
+          (plstore-save plstore))))
+    token))
 
 ;;;###autoload
 (defun oauth2-auth (auth-url token-url client-id client-secret &optional scope state redirect-uri)

[-- Attachment #3: Type: text/plain, Size: 2542 bytes --]


Lastly, a brief description of how to get things to work with
auth-source and existing code (subject to the change I mentioned above):

auth-source entries using the plstore backend allow the secret to be a
function (which is passed the whole entry plist as an argument). All
that is needed then is a function that returns the access token (which
is then used in gnus and smtpmail, both of which already work properly
with an oauth2 access-token). A simple function to check the expiration
time and fetch a new access-token if necessary (and update the new token
and expiration information) and then return the access-token is what I
use. 

So I used auth-source to create  plstore entries for  gmail and
outlook containing the oauth2 tokens, and set the secret to the
following function

(defun gnus-refresh-access (plist)
  "Return an oauth2 access-token for PLIST.
If the current token has expired, fetch, save, and return a new one."
  (cl-destructuring-bind
      (&key user host port token last-update
            (expires_in (alist-get 'expires_in
                                   (oauth2-token-access-response token)))
            (create-args
             (list :type 'plstore :create
                   '(:encrypted (token client-secret-sav)
                                :unencrypted (auth-url scope redirect-uri last-update smtp-auth))))
            &allow-other-keys) plist
    (unless (and (numberp expires_in) (numberp last-update)
                 (< (float-time) (+ last-update expires_in)))
      (message "Getting new token for %s at %s:%s" user host port)
      (setq plist (plist-put plist :secret 'gnus-refresh-access))
      (setq plist (plist-put plist :last-update (truncate (float-time))))
      ;; get a new token and update the plist
      (setq plist (plist-put plist :token (oauth2-refresh-access token)))
      ;; update auth-source---if something in the plist has changed
      ;; then no entry will be found during the search, and the
      ;; create flag will be honored.
      (apply #'auth-source-search (append plist create-args)))
    ;; return the access token
    (oauth2-token-access-token (plist-get plist :token))))

By the way, I let auth-source handle the plstore rather than
oauth2.el. It seemed simpler to have only one of them managing the store
rather than both. 

By the by the way, there are some important bugs in auth-source.el that
I have fixed in my personal tree (and a few that I haven't). I'll post
about them in a separate bug report at some point.

Best,
Andy
-- 
Andrew Cohen