questions about correct reveal-mode usage to hide passwords

questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

Hello,

Following up on some old threads, I am trying to convert `authinfo-mode'
(which Lars wrote) into a proper minor mode that can be used anywhere. I
named the new minor mode `auth-source-reveal-mode' because in the Emacs
scope, it will primarily live in auth-source.el and be used for visually
hiding passwords in netrc/authinfo/JSON files.

I am including the change below, and it's also in the branch
scratch/tzz/auth-source-reveal-mode

I ran into two problems with converting `authinfo-mode':

* it doesn't update when the buffer is changed, but rather does one
  initial scan to install the overlays and then turns on `reveal-mode'.
  So further editing doesn't update, e.g. "password xyz" edited to "pass
  xyz" doesn't remove the hiding on xyz. I tried to add after-change
  functions but they don't seem to work well. Managing the overlays is a
  bit of a chore. It feels like reveal-mode should have a more automatic
  way of tracking this but I couldn't find it. I would appreciate some
  help there.

* it doesn't have JSON support, so I'll need to add that. Re-parsing the
  entire buffer is too expensive. Is there a way to scan a smaller
  region using the built-in JSON parser? Or should I drop down to
  regular expressions to match "password": "xyz"? Currently that's a
  (debug) TODO and not as important as the above.

(While researching this, I compared notes with the external packages
hidepw-mode and password-mode. One[1] uses font-lock mode to adapt to
change dynamically, which works well for live editing, but you can't see
the password when you're editing it. The other[2] also doesn't seem to
handle editing changes.)

[1] https://github.com/jekor/hidepw/blob/master/hidepw.el
[2] https://github.com/juergenhoetzel/password-mode/blob/master/password-mode.el

Thanks!
Ted

--------------------------------------------------------------------

;;; Tiny minor mode for editing .netrc/.authinfo modes (that basically
;;; just hides passwords).

(defcustom auth-source-reveal-regex "password"
  "Regexp matching tokens or JSON keys in .authinfo/.netrc/JSON files.
The text following the tokens or under the JSON keys will be hidden."
  :type 'regexp
  :version "27.1")

(defcustom auth-source-reveal-json-modes '(json-mode js-mode js2-mode rjsx-mode)
  "List of symbols for modes that should use JSON parsing logic."
  :type 'list
  :version "27.1")

(defun auth-source-reveal--propertize (start end hide)
  (save-excursion
    (goto-char start)
    (if (member major-mode auth-source-reveal-json-modes)
        ;; JSON modes
        (debug)
      ;; non-JSON modes
      (save-restriction
        (narrow-to-region (min (point-at-bol) start)
                          (max (point-at-eol) end))
        (cl-dolist (o (overlays-in (point-min) (point-max)))
          (when (overlay-get o 'display)
            (delete-overlay o)))
        (while (re-search-forward (format "\\(\\s-\\|^\\)\\(%s\\)\\s-+"
                                          auth-source-reveal-regex)
                                  nil t)
          (when (auth-source-netrc-looking-at-token)
            (let ((overlay (make-overlay (match-beginning 0) (match-end 0))))
              (auth-source-reveal--display overlay hide)
              (overlay-put overlay 'reveal-toggle-invisible
                           #'auth-source-reveal--display))))))))

(defun auth-source-reveal--display (overlay hide)
  (if hide
      (overlay-put overlay 'display
                   ;; Make a string of * of the same size as the original
                   (propertize (make-string 6 ?*) 'face 'warning))
    (overlay-put overlay 'display nil)))

(defun auth-source-reveal-after-change-function (start stop n)
  (auth-source-reveal--propertize start stop auth-source-reveal-mode))

;; (progn
;;   (setq auth-source-reveal-json-modes '(emacs-lisp-mode lisp-interaction-mode))
;;   (auth-source-reveal-mode t))

;; (auth-source-reveal-mode -1)

;;;###autoload
(define-minor-mode auth-source-reveal-mode
  "Toggle password hiding for auth-source files using `reveal-mode'.

If called interactively, enable auth-source-reveal mode if ARG is
positive, and disable it if ARG is zero or negative.  If called
from Lisp, also enable the mode if ARG is omitted or nil, and
toggle it if ARG is toggle; disable the mode otherwise.

When auth-source-reveal mode is enabled, password will be hidden
using an overlay.  See `auth-source-password-hide-regex' for the
regex matching the tokens and keys associated with passwords."
  ;; The initial value.
  :init-value nil
  ;; The indicator for the mode line.
  :lighter " asr"
  :group 'auth-source

  (auth-source-do-trivia "Setting auth-source-reveal-mode to %S"
                         auth-source-reveal-mode)
  (if auth-source-reveal-mode
      (add-hook 'after-change-functions #'auth-source-reveal-after-change-function nil t)
    (remove-hook 'after-change-functions #'auth-source-reveal-after-change-function t))
  (auth-source-reveal--propertize (point-min) (point-max) auth-source-reveal-mode)
  (reveal-mode (if auth-source-reveal-mode 1 -1)))

Re: questions about correct reveal-mode usage to hide passwords

[Clément Pit-Claudel]

On 11/06/2020 10.09, Ted Zlatanov wrote:
> One[1] uses font-lock mode to adapt to
> change dynamically, which works well for live editing, but you can't see
> the password when you're editing it.

font-lock is the perfect way to implement this, and it should be easy to tweak it to reveal passwords at point.  Look at how prettify-symbols-mode works for inspiration (it uses fontlock and it removes the font-locking when prettify-symbols-unprettify-at-point is set)

Re: questions about correct reveal-mode usage to hide passwords

[Stefan Monnier]

>> One[1] uses font-lock mode to adapt to
>> change dynamically, which works well for live editing, but you can't see
>> the password when you're editing it.
> font-lock is the perfect way to implement this, and it should be easy to

I'd recommend using jit-lock rather than font-lock.
It might be a case of bikeshedding, but I think it will be both easier
to implement and more robust.


        Stefan


PS: I assume it installs *overlays* rather than text properties.

Re: questions about correct reveal-mode usage to hide passwords

[Clément Pit-Claudel]

On 11/06/2020 13.33, Stefan Monnier wrote:
>>> One[1] uses font-lock mode to adapt to
>>> change dynamically, which works well for live editing, but you can't see
>>> the password when you're editing it.
>> font-lock is the perfect way to implement this, and it should be easy to
> 
> I'd recommend using jit-lock rather than font-lock.
> It might be a case of bikeshedding, but I think it will be both easier
> to implement and more robust.

Hmm.  I meant font-lock-keywords and text properties (which will benefit from jit-lock, of course), rather than hooking up directly into jit-lock and using overlays.  Is there something wrong with the text-properties approach?

Re: questions about correct reveal-mode usage to hide passwords

[Stefan Monnier]

>>>> One[1] uses font-lock mode to adapt to
>>>> change dynamically, which works well for live editing, but you can't see
>>>> the password when you're editing it.
>>> font-lock is the perfect way to implement this, and it should be easy to
>> 
>> I'd recommend using jit-lock rather than font-lock.
>> It might be a case of bikeshedding, but I think it will be both easier
>> to implement and more robust.
>
> Hmm.  I meant font-lock-keywords and text properties (which will benefit
> from jit-lock, of course), rather than hooking up directly into jit-lock and
> using overlays.  Is there something wrong with the text-properties approach?

It's just that using `font-lock-keywords` is fiddly, and then you also
have to set `font-lock-extra-managed-props`, ...
And next thing you know a user wants to use your thing but doesn't want
font-lock coloring, ...


        Stefan

Re: questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

On Thu, 11 Jun 2020 13:49:00 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> I'd recommend using jit-lock rather than font-lock.
>>> It might be a case of bikeshedding, but I think it will be both easier
>>> to implement and more robust.
>> 
>> Hmm.  I meant font-lock-keywords and text properties (which will benefit
>> from jit-lock, of course), rather than hooking up directly into jit-lock and
>> using overlays.  Is there something wrong with the text-properties approach?

SM> It's just that using `font-lock-keywords` is fiddly, and then you also
SM> have to set `font-lock-extra-managed-props`, ...
SM> And next thing you know a user wants to use your thing but doesn't want
SM> font-lock coloring, ...

jit-lock seems to require lots of extra work compared to adapting
prettify-symbols-mode and asking the user to add extra settings.
jit-lock is not tied to regexp lookups, which is OK with me (a scanning
function is better for my specific need, since secrets are often
contextual).

The cleanest example I saw was glasses.el but that mode doesn't reveal
the transformed text when you're inside the text.

Is there an example using jit-lock that behaves like
prettify-symbols-mode in the following cases?

* transforms the text when outside it
* temporarily shows the original text when inside it or on its right edge
* updates when the user makes changes

Thank you again
Ted

Re: questions about correct reveal-mode usage to hide passwords

[Stefan Monnier]

> Is there an example using jit-lock that behaves like
> prettify-symbols-mode in the following cases?
>
> * transforms the text when outside it
> * temporarily shows the original text when inside it or on its right edge
> * updates when the user makes changes

Maybe if you look at the `csv-align-mode` in GNU ELPA's `csv-mode.el`
(tho it mixes the job of aligning columns and of truncating/hiding the
text of columns that are too wide).


        Stefan

Re: questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

On Thu, 11 Jun 2020 15:51:27 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Is there an example using jit-lock that behaves like
>> prettify-symbols-mode in the following cases?
>> 
>> * transforms the text when outside it
>> * temporarily shows the original text when inside it or on its right edge
>> * updates when the user makes changes

SM> Maybe if you look at the `csv-align-mode` in GNU ELPA's `csv-mode.el`
SM> (tho it mixes the job of aligning columns and of truncating/hiding the
SM> text of columns that are too wide).

Hi Stefan. Thank you for the suggestion.

I spent a few hours in that direction, but prettify-symbols-mode was
just too easy to extend... I've pushed my proposed patch to
scratch/tzz/auth-source-reveal-mode as follows:

* allow regexps in prettify-symbols-alist using a new extended format. I
  wasn't sure here if I should go with the current format, and the
  replacement lookup does extra string-match calls, so it's not ideal.
  This specific extension was requested fairly often, looking back at
  the last few years on various Emacs forums.

* define a new local variable prettify-symbols-compose-replacer for the
  function that looks up and replaces the symbol

* create the new auth-source-reveal-mode that behaves as proposed, both
  in JSON and in netrc/authinfo buffers. It warns if
  prettify-symbols-unprettify-at-point is nil, since I think the
  experience is much less pleasant without it. It doesn't check syntax
  as carefully as prettify-symbols-mode.

* document the composition layout format enough that users can customize
  auth-source-reveal-hider to show a fancy string instead of a single
  character. I think this will appeal to prettify-symbols-mode users in
  general, since I have seen quite a few hacks for it in various Emacs
  forums and even a helper library https://github.com/Ilazki/prettify-utils.el/blob/master/prettify-utils.el

* the new code in auth-source.el and prog-mode.el is much less than the
  other approaches I tried, and much much less confusing. I like that :)

Let me know if you have strong objections or any suggestions, otherwise
in a day or two I'll document and push this patch.

Thank you
Ted

Re: questions about correct reveal-mode usage to hide passwords

[Clément Pit-Claudel]

On 12/06/2020 12.17, Ted Zlatanov wrote:
> Let me know if you have strong objections or any suggestions, otherwise
> in a day or two I'll document and push this patch.

If you go this route, I would recommend extracting functionality shared between this and prettify-symbols-mode into a separate library; otherwise, won't you have conflicts between the password-hiding patterns and the prettification patterns? Additionally, I'd be surprised if turning of prettification also revealed passwords. 

Re: questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

On Fri, 12 Jun 2020 12:35:11 -0400 Clément Pit-Claudel <cpitclaudel@gmail.com> wrote: 

CP> On 12/06/2020 12.17, Ted Zlatanov wrote:
>> Let me know if you have strong objections or any suggestions, otherwise
>> in a day or two I'll document and push this patch.

CP> If you go this route, I would recommend extracting functionality shared between
CP> this and prettify-symbols-mode into a separate library; otherwise, won't you
CP> have conflicts between the password-hiding patterns and the prettification
CP> patterns? Additionally, I'd be surprised if turning of prettification also
CP> revealed passwords.

You're describing what I thought reveal-mode would do. It's a good
point, and factoring the functionality out to a library makes a lot of
sense.

Let's see if Stefan and others agree with the approach--remember, he
recommended using jit-lock. Once we have a solid foundation for this
functionality, I'm happy to start on the library.

Ted

Re: questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

On Fri, 12 Jun 2020 16:17:25 +0000 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I spent a few hours in that direction, but prettify-symbols-mode was
TZ> just too easy to extend... I've pushed my proposed patch to
TZ> scratch/tzz/auth-source-reveal-mode as follows:

TZ> * allow regexps in prettify-symbols-alist using a new extended format. I
TZ>   wasn't sure here if I should go with the current format, and the
TZ>   replacement lookup does extra string-match calls, so it's not ideal.
TZ>   This specific extension was requested fairly often, looking back at
TZ>   the last few years on various Emacs forums.

I extended the new format to allow any symbol as an identifier, and
speed up lookups based on the identifier. Each fixed entry is still
collected in one big optimized regexp by
prettify-symbols--make-fixed-matcher, and there is no identifier (we
just do a fast assoc as before). But in addition, each regexp entry in
prettify-symbols-alist now gets processed by
prettify-symbols--make-regexp-keywords, with the identifier propagated
down the call chain. I think this is fairly clean.

TZ> * define a new local variable prettify-symbols-compose-replacer for the
TZ>   function that looks up and replaces the symbol

TZ> * create the new auth-source-reveal-mode that behaves as proposed, both
TZ>   in JSON and in netrc/authinfo buffers. It warns if
TZ>   prettify-symbols-unprettify-at-point is nil, since I think the
TZ>   experience is much less pleasant without it. It doesn't check syntax
TZ>   as carefully as prettify-symbols-mode.

TZ> * document the composition layout format enough that users can customize
TZ>   auth-source-reveal-hider to show a fancy string instead of a single
TZ>   character. I think this will appeal to prettify-symbols-mode users in
TZ>   general, since I have seen quite a few hacks for it in various Emacs
TZ>   forums and even a helper library https://github.com/Ilazki/prettify-utils.el/blob/master/prettify-utils.el

TZ> * the new code in auth-source.el and prog-mode.el is much less than the
TZ>   other approaches I tried, and much much less confusing. I like that :)

Following Clément's suggestion, I started on the library path by
creating some new functions to install and remove prettifications.
Coupled with the new identifier symbols and regexps, I think this covers
all the use cases for touching or bypassing prettify-symbols-alist I've
seen.

prettify-symbols-add-prettification-entry
  "Add ENTRY to `prettify-symbols-alist' for the current buffer.
prettify-symbols-add-prettification-rx
  "Convenience wrapper of `prettify-symbols-add-prettification-entry' to prettify REGEXP with REPLACEMENT."
prettify-symbols-add-prettification-string
  "Convenience wrapper of `prettify-symbols-add-prettification-entry' to prettify FIXED-STRING with REPLACEMENT."
prettify-symbols-remove-prettification
  "Remove ENTRY to `prettify-symbols-alist' for the current buffer.
prettify-symbols-remove-prettifications
  "Remove all IDENTIFIER entries from `prettify-symbols-alist' for the current buffer.

TZ> Let me know if you have strong objections or any suggestions, otherwise
TZ> in a day or two I'll document and push this patch.

I haven't heard from Stefan, so I'm still holding off merging and would
appreciate feedback.

Ted

Re: questions about correct reveal-mode usage to hide passwords

[Ted Zlatanov]

On Tue, 16 Jun 2020 18:09:48 +0000 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Let me know if you have strong objections or any suggestions, otherwise
TZ> in a day or two I'll document and push this patch.

I've split up my commit into 1) a minor bugfix (preexisting), 2) the
prog-mode prettify-symbols-mode changes, and 3) the
auth-source-reveal-mode implementation and documentation. All in branch
scratch/tzz/auth-source-reveal-mode

The docs in doc/emacs/programs.texi should be updated with the
prettify-symbols-mode changes, but I'll hold off because it may be
better to move them out into a library. Right now it's an API but still
tightly bound to prettify-symbols-mode in naming and behavior.

Clément or anyone else interested, could you please review?

Thank you
Ted