[ELPA] New package: xr

[ELPA] New package: xr

[Mattias Engdegård]

This is a proposal to add xr to ELPA.

The xr package is the inverse of rx: it transforms an Emacs regexp string into rx form. Its purpose is to assist in migration to rx-style regexps, something I think a lot more people should do, and help understanding difficult regexps.

It is written to take all features and oddities in the Emacs regexp notation into account. This includes undocumented parts, since such regexps are found in actual code.

The package consists of a single file, right now at https://github.com/mattiase/xr . It is something I have had lying around for some time and found constantly useful, and have now had the opportunity to give some polish.

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

On 01/02/2019 09.09, Mattias Engdegård wrote:
> This is a proposal to add xr to ELPA.
> 
> The xr package is the inverse of rx: it transforms an Emacs regexp string into rx form. Its purpose is to assist in migration to rx-style regexps, something I think a lot more people should do, and help understanding difficult regexps.
> 
> It is written to take all features and oddities in the Emacs regexp notation into account. This includes undocumented parts, since such regexps are found in actual code.
> 
> The package consists of a single file, right now at https://github.com/mattiase/xr . It is something I have had lying around for some time and found constantly useful, and have now had the opportunity to give some polish.

Looks great. Here's a feature request: can you combine this with a font-lock rule to display existing regexps in rx syntax?  Something like prettify-symbols-mode, but for regular expressions.

Clément.

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 1 Feb 2019 10:25:12 -0500
> 
> Looks great. Here's a feature request: can you combine this with a font-lock rule to display existing regexps in rx syntax?  Something like prettify-symbols-mode, but for regular expressions.

Please, NOT like prettify-symbols-mode!  An overlay string or a
display property that shows the rx syntax should be much better.

TIA

Re: [ELPA] New package: xr

[Mattias Engdegård]

1 feb. 2019 kl. 16.25 skrev Clément Pit-Claudel <cpitclaudel@gmail.com>:
> 
> Looks great. Here's a feature request: can you combine this with a font-lock rule to display existing regexps in rx syntax?  Something like prettify-symbols-mode, but for regular expressions.

You mean a mode so that when the user is looking at the elisp source

  (if (looking-at "regexp-string") ...

he would actually see

  (if (looking-at (rx translated-expression)) ...

? Could be done, I suppose, but since there is no special syntax for regexp strings, it would probably either need to rely on a list of well-known functions with regexp arguments, or second-guess it from their argument names (like REGEXP or PATTERN). It sounds a bit fragile.

I suggest that someone who would like to explore this in detail build a prototype using xr as the translation engine; its public interface should be quite sufficient for the task.

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

On 01/02/2019 10.42, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 1 Feb 2019 10:25:12 -0500
>>
>> Looks great. Here's a feature request: can you combine this with a font-lock rule to display existing regexps in rx syntax?  Something like prettify-symbols-mode, but for regular expressions.
> 
> Please, NOT like prettify-symbols-mode!  An overlay string or a
> display property that shows the rx syntax should be much better.
> 
> TIA

:) I meant looking like prettify-symbols-mode; I wasn't suggesting to use composition for the implementation.

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

On 01/02/2019 10.51, Mattias Engdegård wrote:
> 1 feb. 2019 kl. 16.25 skrev Clément Pit-Claudel <cpitclaudel@gmail.com>:
>>
>> Looks great. Here's a feature request: can you combine this with a font-lock rule to display existing regexps in rx syntax?  Something like prettify-symbols-mode, but for regular expressions.
> 
> You mean a mode so that when the user is looking at the elisp source
> 
>   (if (looking-at "regexp-string") ...
> 
> he would actually see
> 
>   (if (looking-at (rx translated-expression)) ...
> 
> ? Could be done, I suppose, but since there is no special syntax for regexp strings, it would probably either need to rely on a list of well-known functions with regexp arguments, or second-guess it from their argument names (like REGEXP or PATTERN). It sounds a bit fragile.

I would suggest neither; instead, it would convert only strings that contain regexp constructs (these are the ones that are hard to read in the first place).
That;'s more or less what I do in https://github.com/cpitclaudel/easy-escape, and it works OK.

Re: [ELPA] New package: xr

[Mattias Engdegård]

1 feb. 2019 kl. 19.41 skrev Clément Pit-Claudel <cpitclaudel@gmail.com>:
> 
> I would suggest neither; instead, it would convert only strings that contain regexp constructs (these are the ones that are hard to read in the first place).
> That;'s more or less what I do in https://github.com/cpitclaudel/easy-escape, and it works OK.

I see. In that case, it seems that you are ideally positioned to make such an attempt, given your experience with easy-escape. Please forgive me for not pursuing your idea myself; although it has merit, I remain skeptical for two reasons:
First, in contrast to easy-escape, xr is an expanding substitution, sometimes strongly so, and would mess up the code layout, unless care is taken to reformat.
Second, I want to encourage people to actually use rx instead of string regexps, not just paint it over. Your proposal reminds me too much of the current dystopian trajectory of humanity, where we will be living in a polluted wasteland, all wearing AR goggles providing the illusion of an arcadian paradise.

Re: [ELPA] New package: xr

[Mattias Engdegård]

2 feb. 2019 kl. 10.43 skrev Mattias Engdegård <mattiase@acm.org>:
> 
> 1 feb. 2019 kl. 19.41 skrev Clément Pit-Claudel <cpitclaudel@gmail.com>:
>> 
>> I would suggest neither; instead, it would convert only strings that contain regexp constructs (these are the ones that are hard to read in the first place).
>> That;'s more or less what I do in https://github.com/cpitclaudel/easy-escape, and it works OK.

I realise that my reply looked like condescending nonsense - sorry. Please let me try again:

Thank you, that's an interesting idea! I don't think I'll work on it right now, but why don't you give it a go, using xr as the backend?

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On 02/02/2019 10.15, Mattias Engdegård wrote:
> Thank you, that's an interesting idea! I don't think I'll work on it right now, but why don't you give it a go, using xr as the backend?

I've attached an implementation and screenshots.  rx seems to work nicely!

The expansion issue is indeed a bit of a problem.  It's also why I don't use rx often, in fact.
Another issue is regular expressions built with concat, since the parts they are made of are typically not 
The last issue is that the code I posted doesn't reveal the underlying expression (i.e. doesn't remove the display property) when the point is on it, which would be needed for a robust implementation.

By the way, could forms like "[A-Za-z]" be converted to a more compact representation? At the moment xr produces (any "A-Z" "a-z"), but it seems that (any "A-Za-z") would work as well and be shorter.

Again, nice work!
Clément.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: xr-prettify.el --]
[-- Type: text/x-emacs-lisp; name="xr-prettify.el", Size: 3523 bytes --]

;;; xr-prettify.el --- Display regular expressions as xr forms

;; Copyright (C) 2018 Clément Pit--Claudel
;; Author: Clément Pit--Claudel <clement.pitclaudel@live.com>
;; Version: 0.1
;; Keywords: convenience, lisp, tools

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; `xr-prettify-minor-mode' uses syntax highlighting and display properties to make ELisp
;; regular expressions more readable.  More precisely, it replaces "complex
;; enough" regular expressions with an equivalent "rx" form.  The underlying
;; buffer text is not modified.
;;
;; For example, `xr-prettify` prettifies this:
;;   "\\(?:\\_<\\\\newc\\_>\\s-*\\)?"
;; into this (`^' indicates a different color):
;;   (opt symbol-start "\\newc" symbol-end (zero-or-more (syntax whitespace)))
;;
;; The appearance of the replacement can be customized using `xr-prettify-face'
;; (which see).
;;
;; Suggested setup:
;;   (add-hook 'lisp-mode-hook 'xr-prettify-minor-mode)

;;; Code:

(require 'font-lock)
(require 'xr)

(defgroup xr-prettify nil
  "Display regular expressions in `rx' form."
  :group 'programming)

;; FIXME reveal while typing

(defun xr-prettify--maybe-replace-string ()
  "Possibly display the string starting before (point) as an `rx' form."
  (save-excursion
    (let ((ppss-state (syntax-ppss (point))))
      (when (nth 3 ppss-state)
        (let* ((start (1- (point)))
               (eol (point-at-eol))
               (end (save-excursion
                      (parse-partial-sexp (point) (1+ eol) nil
                                          nil ppss-state 'syntax-table)
                      (point))))
          (when (and (<= end eol)
                     (search-forward "\\\\" end t))
            (let* ((txt (buffer-substring-no-properties start end))
                   (str (ignore-errors (read txt)))
                   (rx (and str (ignore-errors (xr str))))
                   (pretty (and rx (prin1-to-string `(rx ,rx)))))
              (when (and pretty (not (string= pretty txt)))
                (put-text-property start end 'display pretty)))))))
    nil))

(defconst xr-prettify--keywords
  '(("\"" (0 (progn (xr-prettify--maybe-replace-string) nil))))
  "Font-lock keyword list used internally.")

;;;###autoload
(define-minor-mode xr-prettify-minor-mode
  "Display regular expressions in `rx' form to improve readability.
When this mode is active, strings are displayed in `rx' syntax,
and highlighted with `xr-prettify-face'."
  :lighter " xr"
  :group 'xr-prettify
  (cond
   (xr-prettify-minor-mode
    (font-lock-add-keywords nil xr-prettify--keywords)
    (make-local-variable 'font-lock-extra-managed-props)
    (add-to-list 'font-lock-extra-managed-props 'display))
   (t (font-lock-remove-keywords nil xr-prettify--keywords)))
  (if (>= emacs-major-version 25) (font-lock-flush)
    (with-no-warnings (font-lock-fontify-buffer))))

(provide 'xr-prettify)
;;; xr-prettify.el ends here

[-- Attachment #3: rx.png --]
[-- Type: image/png, Size: 74332 bytes --]

[-- Attachment #4: re.png --]
[-- Type: image/png, Size: 66925 bytes --]

Re: [ELPA] New package: xr

[Mattias Engdegård]

2 feb. 2019 kl. 19.41 skrev Clément Pit-Claudel <cpitclaudel@gmail.com>:
> 
> The expansion issue is indeed a bit of a problem.  It's also why I don't use rx often, in fact.

The verbosity of rx is very much a question of habituation. It is to me a very small price to pay for its numerous advantages.
I recommend reading Olin Shivers's original SRE text [https://scsh.net/docu/post/sre.html] for a well-reasoned rationale.

> By the way, could forms like "[A-Za-z]" be converted to a more compact representation? At the moment xr produces (any "A-Z" "a-z"), but it seems that (any "A-Za-z") would work as well and be shorter.

That's a close call, but I tried it and it's probably no less readable so I made the change. Thank you.

Re: [ELPA] New package: xr

[Juri Linkov]

> I've attached an implementation and screenshots.  rx seems to work nicely!

It also could help to find bugs, e.g. in your example the symbol ‘nonl’
inside “loaddefs.el” indicates it should be a literal ‘.’.

Re: [ELPA] New package: xr

[Mattias Engdegård]

3 feb. 2019 kl. 21.11 skrev Juri Linkov <juri@linkov.net>:
> 
> It also could help to find bugs, e.g. in your example the symbol ‘nonl’
> inside “loaddefs.el” indicates it should be a literal ‘.’.

Very eagled-eyed! Perhaps someone would fix it right away?

For that matter, I have had the same experience: bugs that hide well in a jungle of leaning toothpicks are deprived of their natural camouflage when converted to rx. Some inefficiencies also tend to become visible. The first things to go are usually the useless capture groups.

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

On 03/02/2019 15.11, Juri Linkov wrote:
>> I've attached an implementation and screenshots.  rx seems to work nicely!
> 
> It also could help to find bugs, e.g. in your example the symbol ‘nonl’
> inside “loaddefs.el” indicates it should be a literal ‘.’.

Indeed! Good catch :)

Re: [ELPA] New package: xr

[Stefan Monnier]

> This is a proposal to add xr to ELPA.

Done, thank you.
It's now in the `externals/xr` branch in elpa.git and the corresponding
1.0 release package should appear in GNU ELPA shortly.

As always with elpa.git, this does not automatically track remote
branches, so you'll need to `git push` to that branch when necessary.


        Stefan

Re: [ELPA] New package: xr

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> 1.0 release package should appear in GNU ELPA shortly.

It has appeared.

Mattias, a question about `xr-pp':

(1) The docstring says "This basically does `(pp (rx RE-STRING))'
[...]", but there seems to be a typo, it should say "xr" instead of
"rx", right?

(2) Could `xr--pp-rx-to-str' be called `xr-pp-rx-to-str'?  I think it's
a too important and useful function to name it with a double hyphen.


Thanks,

Michael.

Re: [ELPA] New package: xr

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > This is a proposal to add xr to ELPA.
>
> Done, thank you.

BTW, did someone notice that the functionality of "pcre2el" in GNU Elpa
overlaps with the newly added package?

Michael.

Re: [ELPA] New package: xr

[Mattias Engdegård]

5 feb. 2019 kl. 17.15 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> Done, thank you.
> It's now in the `externals/xr` branch in elpa.git and the corresponding
> 1.0 release package should appear in GNU ELPA shortly.

Thanks. You were unhappy with the tests, and rightly so; I've moved them to a separate file and now use ert.

> As always with elpa.git, this does not automatically track remote
> branches, so you'll need to `git push` to that branch when necessary.

I have no push access to elpa.git. Should I request it?
The source could live directly in elpa.git instead of being an external package as far as I am concerned. Do you have any preference?

Regarding the possible mail loop: I haven't observed it before. If it keeps causing trouble, I'll investigate.

Re: [ELPA] New package: xr

[Mattias Engdegård]

5 feb. 2019 kl. 23.37 skrev Michael Heerdegen <michael_heerdegen@web.de>:
> 
> (1) The docstring says "This basically does `(pp (rx RE-STRING))'
> [...]", but there seems to be a typo, it should say "xr" instead of
> "rx", right?

Thanks, now fixed.

> (2) Could `xr--pp-rx-to-str' be called `xr-pp-rx-to-str'?  I think it's
> a too important and useful function to name it with a double hyphen.

Certainly; done.

Re: [ELPA] New package: xr

[Stefan Monnier]

> BTW, did someone notice that the functionality of "pcre2el" in GNU Elpa
> overlaps with the newly added package?

I missed that.  Could you mention it in the Commentary (I already added
a mention of lex-parse-re which also overlaps)?


        Stefan

Re: [ELPA] New package: xr

[Stefan Monnier]

> I have no push access to elpa.git. Should I request it?

If you intend to keep actively maintaining the package, then it would
make sense, yes.

> The source could live directly in elpa.git instead of being an external
> package as far as I am concerned.  Do you have any preference?

If the elpa.git repository can be the official "upstream" then it avoids
the need to keep it in sync, so that's what I recommend, but it's up to you.

> Regarding the possible mail loop: I haven't observed it before.
> If it keeps causing trouble, I'll investigate.

I'll let you know if it re-occurs.


        Stefan

Re: [ELPA] New package: xr

[Noam Postavsky]

On Tue, 5 Feb 2019 at 19:11, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > BTW, did someone notice that the functionality of "pcre2el" in GNU Elpa
> > overlaps with the newly added package?
>
> I missed that.  Could you mention it in the Commentary (I already added
> a mention of lex-parse-re which also overlaps)?

pcre2el is not in GNU ELPA, as far as I know.

Re: [ELPA] New package: xr

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> pcre2el is not in GNU ELPA, as far as I know.

Oh, yes, indeed.  I have it in ".emacs.d/elpa/pcre2el-20161120.2103" and
only read "elpa" but that's just where all the stuff gets installed.

Sorry,

Michael.

Re: [ELPA] New package: xr

[Stefan Monnier]

>> > BTW, did someone notice that the functionality of "pcre2el" in GNU Elpa
>> > overlaps with the newly added package?
>> I missed that.  Could you mention it in the Commentary (I already added
>> a mention of lex-parse-re which also overlaps)?
> pcre2el is not in GNU ELPA, as far as I know.

It's still Free Software, so it's not a reason not to mention it,


        Stefan

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Did we get a copyright assignment for xr?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Mattias Engdegård]

6 feb. 2019 kl. 01.11 skrev Stefan Monnier <monnier@IRO.UMontreal.CA>:
> 
>> BTW, did someone notice that the functionality of "pcre2el" in GNU Elpa
>> overlaps with the newly added package?
> 
> I missed that.  Could you mention it in the Commentary (I already added
> a mention of lex-parse-re which also overlaps)?

Done.

It is a much more ambitious package but seems to suffer from some bugs. I'm not sure if it is maintained.
I'll contact the author.

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Wed, 06 Feb 2019 02:02:01 -0500
> Cc: emacs-devel@gnu.org
> 
> Did we get a copyright assignment for xr?

Mattias has a copyright assignment on file, yes.

pcre2el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is a much more ambitious package but seems to suffer from some
  > bugs. I'm not sure if it is maintained.

As a general question, is it a good idea to keep packages in ELPA
which are not maintained?  Perhaps in some cases that is ok and in
others that is not.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Did we get a copyright assignment for xr?

  > Mattias has a copyright assignment on file, yes.

Is the assignment for "changes to Emacs"?  If so, he should send a
message saying that this file constitutes a change to Emacs, and then
please send that message to assign@gnu.org.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: pcre2el

[John Wiegley]

>>>>> "RS" == Richard Stallman <rms@gnu.org> writes:

RS> As a general question, is it a good idea to keep packages in ELPA which
RS> are not maintained? Perhaps in some cases that is ok and in others that is
RS> not.

If they continue to work without much intervention, I don't see much harm in
it. We don't advertise ELPA as being "curated, maintained packages" do we?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Wed, 06 Feb 2019 23:36:27 -0500
> 
>   > Mattias has a copyright assignment on file, yes.
> 
> Is the assignment for "changes to Emacs"?  If so, he should send a
> message saying that this file constitutes a change to Emacs, and then
> please send that message to assign@gnu.org.

Sorry, I don't understand.  We don't have separate assignments for
ELPA, we have always treated packages on ELPA as being part of Emacs,
for the copyright assignment purposes.  Is that incorrect for some
reason?

Re: pcre2el

[Stefan Monnier]

> As a general question, is it a good idea to keep packages in ELPA
> which are not maintained?

I don't see "maintained" as a significant part of the decision.  To me
the question should rather be whether it's still working, and whether
the code is still useful (either as a basis for other code, or for
end-users).

I've added to GNU ELPA some packages specifically because they were
unmaintained but still in use (psgml and filladapt come to mind).


        Stefan

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is the assignment for "changes to Emacs"?  If so, he should send a
  > > message saying that this file constitutes a change to Emacs, and then
  > > please send that message to assign@gnu.org.

  > Sorry, I don't understand.  We don't have separate assignments for
  > ELPA, we have always treated packages on ELPA as being part of Emacs,
  > for the copyright assignment purposes.  Is that incorrect for some
  > reason?

That is correct, but it's a different issue.

The point is when someone contributes changes to Emacs which consist
of a separate module -- so that it isn't clearly "a change to Emacs"
-- we should ask per to affirm explicitly that it _is_ a change to
Emacs, and therefore his assignment covers it.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Stefan Monnier]

> The point is when someone contributes changes to Emacs which consist
> of a separate module -- so that it isn't clearly "a change to Emacs"
> -- we should ask per to affirm explicitly that it _is_ a change to
> Emacs, and therefore his assignment covers it.

In the past we've considered explicit requests to contribute the package
to Emacs or GNU ELPA as being such a statement.


        Stefan

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 07 Feb 2019 22:50:59 -0500
> 
> > The point is when someone contributes changes to Emacs which consist
> > of a separate module -- so that it isn't clearly "a change to Emacs"
> > -- we should ask per to affirm explicitly that it _is_ a change to
> > Emacs, and therefore his assignment covers it.
> 
> In the past we've considered explicit requests to contribute the package
> to Emacs or GNU ELPA as being such a statement.

And if that is not enough for some reason, then what exactly should
the contributor of a package say, and where, to express such a
statement?

Also, there are packages distributed with Emacs which can be
considered not "clearly" a change to Emacs -- should the same policy
be used for them?  E.g., Gnus or Org or ERC -- they are add-ons that
have no direct effect on the Emacs core.

Re: [ELPA] New package: xr

[Stephen Berman]

On Fri, 08 Feb 2019 09:18:33 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Thu, 07 Feb 2019 22:50:59 -0500
>> 
>> > The point is when someone contributes changes to Emacs which consist
>> > of a separate module -- so that it isn't clearly "a change to Emacs"
>> > -- we should ask per to affirm explicitly that it _is_ a change to
>> > Emacs, and therefore his assignment covers it.
>> 
>> In the past we've considered explicit requests to contribute the package
>> to Emacs or GNU ELPA as being such a statement.
>
> And if that is not enough for some reason, then what exactly should
> the contributor of a package say, and where, to express such a
> statement?

Would it suffice (in addition to a copyright assignment, of course) if
the files in GNU ELPA contain this sentence?

  This file is part of GNU Emacs.

Currently, some do but many don't.

> Also, there are packages distributed with Emacs which can be
> considered not "clearly" a change to Emacs -- should the same policy
> be used for them?  E.g., Gnus or Org or ERC -- they are add-ons that
> have no direct effect on the Emacs core.

I think all (or at least all Lisp) files in those packages do contain
the above sentence, though not all files in the emacs source tree do
(e.g. README, INSTALL, CONTRIBUTE).

Steve Berman

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Would it suffice (in addition to a copyright assignment, of course) if
  > the files in GNU ELPA contain this sentence?

  >   This file is part of GNU Emacs.

  > Currently, some do but many don't.

IANAL, but I think that all of the source files in ELPA should have
the standard license notice, including saying "part of GNU Emacs" --
if for legal purposes we consider it part of GNU Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In the past we've considered explicit requests to contribute the package
  > to Emacs or GNU ELPA as being such a statement.

IANAL but I worry that that may not be explicit enough to it clear to
outsiders that the file will become part of Emacs.  When you are
talking to a person who works with you in a group, you can easily
convey a point clearly _to per_ with words that won't convey the point
clearly to outsiders.

So if we want to rely on this, we should make sure to make that invitation
explicitly clear about this very point.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And if that is not enough for some reason, then what exactly should
  > the contributor of a package say, and where, to express such a
  > statement?

I expect that saying "my file foo is a change to Emacs" is enough.

  > Also, there are packages distributed with Emacs which can be
  > considered not "clearly" a change to Emacs -- should the same policy
  > be used for them?  E.g., Gnus or Org or ERC -- they are add-ons that
  > have no direct effect on the Emacs core.

We treat ERC, Gnus and Org Mode as parts of Emacs for copyright purposes.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Fri, 08 Feb 2019 22:37:15 -0500
> 
>   > Would it suffice (in addition to a copyright assignment, of course) if
>   > the files in GNU ELPA contain this sentence?
> 
>   >   This file is part of GNU Emacs.
> 
>   > Currently, some do but many don't.
> 
> IANAL, but I think that all of the source files in ELPA should have
> the standard license notice, including saying "part of GNU Emacs" --
> if for legal purposes we consider it part of GNU Emacs.

Would doing that in the source file be enough to serve as a legal
statement that the Emacs copyright assignment covers the package?  Or
will something else be required?

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Fri, 08 Feb 2019 22:38:00 -0500
> 
>   > And if that is not enough for some reason, then what exactly should
>   > the contributor of a package say, and where, to express such a
>   > statement?
> 
> I expect that saying "my file foo is a change to Emacs" is enough.

Saying where?

>   > Also, there are packages distributed with Emacs which can be
>   > considered not "clearly" a change to Emacs -- should the same policy
>   > be used for them?  E.g., Gnus or Org or ERC -- they are add-ons that
>   > have no direct effect on the Emacs core.
> 
> We treat ERC, Gnus and Org Mode as parts of Emacs for copyright purposes.

I know, but what I don't understand is how ERC etc. are different from
a package stored on ELPA.  Could you please clarify the difference, if
there is one?

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I know, but what I don't understand is how ERC etc. are different from
  > a package stored on ELPA.

I did not say they are different.  My understanding is that we treat ALL
the packages in ELPA as part of Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I expect that saying "my file foo is a change to Emacs" is enough.

  > Saying where?

In email to you; you forward that to assign@gnu.org.

Putting it in a version of the file, that you get fron the contributor
and install, would also be ok.

Maybe some other ways would also be good enough, but they don'tcome to
my mind just now.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Sun, 10 Feb 2019 00:51:58 -0500
> 
>   > > I expect that saying "my file foo is a change to Emacs" is enough.
> 
>   > Saying where?
> 
> In email to you; you forward that to assign@gnu.org.

In addition to saying in the file that "this file is part of GNU
Emacs"?  Or is the latter enough?

Re: [ELPA] New package: xr

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Sun, 10 Feb 2019 00:51:58 -0500
> 
>   > I know, but what I don't understand is how ERC etc. are different from
>   > a package stored on ELPA.
> 
> I did not say they are different.  My understanding is that we treat ALL
> the packages in ELPA as part of Emacs.

But for packages we add to Emacs, and place into the Emacs Git
repository, we didn't until now request any statement from the author,
we assumed that being in the repository is a clear sign that the
package is part of Emacs.  We also assumed that being in the ELPA
repository is a clear sign of the same.  Now it sounds like we were
mistaken?

Re: [ELPA] New package: xr

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In addition to saying in the file that "this file is part of GNU
  > Emacs"?  Or is the latter enough?

If that statement is present in the file when the contributor emails
it to you, I think that does the job.  To make sure the FSF knows this,
please forward that email to the FSF clerk.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [ELPA] New package: xr

[Michael Heerdegen]

Hi Mattias,

> The xr package is the inverse of rx: it transforms an Emacs regexp
> string into rx form. Its purpose is to assist in migration to rx-style
> regexps, something I think a lot more people should do, and help
> understanding difficult regexps.

Would it be possible to define a variable to control what kind of
operators `xr--postfix' chooses?  Currently the verbose names are
hardcoded, and even if there is no perfectly nice solution, it would be
good if there was some way to control this.  I'm not sure if there are
other places besides `xr--postfix' where such a variable would be to
consulted.


Thanks,

Michael.

Re: [ELPA] New package: xr

[Mattias Engdegård]

27 feb. 2019 kl. 16.06 skrev Michael Heerdegen <michael_heerdegen@web.de>:
> 
> Would it be possible to define a variable to control what kind of
> operators `xr--postfix' chooses?  Currently the verbose names are
> hardcoded, and even if there is no perfectly nice solution, it would be
> good if there was some way to control this.  I'm not sure if there are
> other places besides `xr--postfix' where such a variable would be to
> consulted.

The identifiers chosen by xr will never suit the taste of everyone, but I couldn't make up my mind about what to do about it. I'd be interested in hearing what other people think.
One way would be to add an optional translation alist to xr:

(xr "a*.?"
    '((zero-or-more . 0+) (nonl . not-newline)))
=> (seq (0+ "a") (opt not-newline))

but this could be done almost as easily by post-processing the resulting list tree, maybe like this:

(defun rename-rx (substitution-alist rx)
  "Rename all identifiers in RX according to SUBSTITUTION-ALIST."
  (cond
   ((symbolp rx) (alist-get rx substitution-alist rx))
   ((consp rx)
    ;; Translate the first element of each list even if not a symbol,
    ;; to allow translation of ? and ?? (space and ?).
    (cons (alist-get (car rx) substitution-alist (car rx))
	  (mapcar (lambda (elem) (rename-rx substitution-alist elem))
		  (cdr rx))))
   (t rx)))

Post-processing is more generally powerful since it can do structural replacement and not mere symbol replacement.

A customisable setting would also work, but I prefer keeping xr purely functional.

Re: [ELPA] New package: xr

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> A customisable setting would also work, but I prefer keeping xr purely
> functional.

Then maybe at least define several settings somewhere and make them
available to be used as argument or replacement list or whatever.  It
would be good if it was possible to get a different behavior without too
much brain usage: Make it so that the calls needed can be kept short.


Michael.

Re: [ELPA] New package: xr

[Mattias Engdegård]

27 feb. 2019 kl. 18.09 skrev Michael Heerdegen <michael_heerdegen@web.de>:
> 
> Then maybe at least define several settings somewhere and make them
> available to be used as argument or replacement list or whatever.  It
> would be good if it was possible to get a different behavior without too
> much brain usage: Make it so that the calls needed can be kept short.

That's probably a good idea; the optional argument could be either an alist or a symbol like 'short or 'verbose.

Good defaults are just as important. I thought that one-or-more would be more descriptive than + or 1+ for those not familiar with the rx notation, but perhaps this was a mistake.

Which would you prefer as default symbol from each of these sets?

 one-or-more 1+ +
 zero-or-more 0+ *
 zero-or-one optional opt ?
 repeat **  (for lower-upper-bounded repetition)
 repeat =   (for exact-count repetition)
 any char in
 not-newline nonl
 group submatch
 line-start bol
 line-end eol
 string-start buffer-start bos bot
 string-end buffer-end eos eot
 word-start bow
 word-end eow
 sequence seq and :
 or |
 regexp regex

Re: [ELPA] New package: xr

[Clément Pit-Claudel]

Here are my picks :)

On 28/02/2019 09.10, Mattias Engdegård wrote:
>  1+
>  0+
>  ?
>  repeat ** (not sure)
>  repeat = (not sure)>  any
>  nonl
>  group
>  bol
>  eol
>  string-start buffer-start
>  string-end buffer-end
>  word-start
>  word-end
>  seq
>  or
>  regex

I'm not sure whether it's be better to be consistent about the 'bo…' and 'eo…' family.

Re: [ELPA] New package: xr

[Stefan Monnier]

>  one-or-more 1+ +
>  zero-or-more 0+ *
>  zero-or-one optional opt ?

I'm definitely in favor of using the standard * + and ?

>  any char in

I think I prefer `char` or `in`.

>  line-start bol
>  line-end eol
>  string-start buffer-start bos bot
>  string-end buffer-end eos eot
>  word-start bow
>  word-end eow

I like the boX/eoX nomenclature.

The main benefit of RX is to make the structure more visible, and the
main downside is to make regexp more verbose, so I think short
identifiers are preferable.

>  sequence seq and :

I'm strongly opposed to `and` because that should mean the
conjunction/intersection of two regexps (i.e. a string matches it only
if it matches both sub-regexps) rather than the sequential concatenation.
[ lex.el supports such intersections.  ]


        Stefan

Re: [ELPA] New package: xr

[Van L]

> Which would you prefer as default symbol from each of these sets?

  one-or-more
  zero-or-more
  zero-or-one
  repeat
  repeat
  any
  not-newline
  group
  line-start
  line-end
  string-start
  string-end
  word-start
  word-end
  sequence
  or
  regexp

for readability in the worst case use case where an end-user/person
operates the patterns fewer than five times a year, year after year,
in an ETL scenario.

Re: [ELPA] New package: xr

[Mattias Engdegård]

1 mars 2019 kl. 00.06 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> I'm definitely in favor of using the standard * + and ?

They are suggestive but I find `?' a bit on the hacky side. For instance, you can't break the line after the operator, since it requires a space following.

>> any char in
> 
> I think I prefer `char` or `in`.

Interestingly, (not (any ...)) and (not (in ...)) are valid, but (not (char ...)) isn't (although there is `not-char').

Another implementation leak: since `any' is also an alias for `not-newline', `char' and `in' are as well:
(rx any char in) => "..."

>> line-start bol
>> line-end eol
>> string-start buffer-start bos bot
>> string-end buffer-end eos eot
>> word-start bow
>> word-end eow
> 
> I like the boX/eoX nomenclature.
> 
> The main benefit of RX is to make the structure more visible, and the
> main downside is to make regexp more verbose, so I think short
> identifiers are preferable.

There is definitely merit to that, although I also have sympathy for the verbosity preferred by others.
Lisp seems to cope with long identifiers better than most other languages (dashes permitted, easy line breaking).

I'm leaning towards giving xr a brief and verbose options, and let the default be in the middle somewhere.

>> sequence seq and :
> 
> I'm strongly opposed to `and` because that should mean the
> conjunction/intersection of two regexps (i.e. a string matches it only
> if it matches both sub-regexps) rather than the sequential concatenation.
> [ lex.el supports such intersections.  ]

Strongly agreed!

Re: [ELPA] New package: xr

[Stefan Monnier]

>> I'm definitely in favor of using the standard * + and ?
> They are suggestive but I find `?' a bit on the hacky side. For instance,
> you can't break the line after the operator, since it requires
> a space following.

Oh, indeed, that's a problem.

> Interestingly, (not (any ...)) and (not (in ...)) are valid, but (not (char
> ...)) isn't (although there is `not-char').

I guess this means we should go with `in`.
Tho it should be easy to add (not (char ...)) if needed.

BTE, I don't like (not (any ...)) and (not (in ...)) in any case:
the negation should be within the `char`, `in`, or `any` not around it.
This is because the negation of a regexp RE could be defined as a regexp
which matches all strings not matched by RE (and hence when RE is (any
...), it will also match all strings of length != 1).

This form of negation is supported in lex.el, for example.

> Another implementation leak: since `any' is also an alias for `not-newline',
> `char' and `in' are as well:
> (rx any char in) => "..."

But XR doesn't have to abuse this, luckily.


        Stefan

Re: [ELPA] New package: xr

[Stefan Monnier]

> <longer form for all options>
> For readability in the worst case use case where an end-user/person
> operates the patterns fewer than five times a year, year after year,
> in an ETL scenario.

While I personally prefer short forms, there is definitely something to
be said for the longer forms.  And you could argue that people who
like it terse know where to find it.


        Stefan

Re: [ELPA] New package: xr

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> Good defaults are just as important. I thought that one-or-more would
> be more descriptive than + or 1+ for those not familiar with the rx
> notation, but perhaps this was a mistake.

Not a mistake, not just what everybody wants.  Dunno what the majority
wants.  But I think as a default it could be right, because it's
something at least everybody understands.

> Which would you prefer as default symbol from each of these sets?

I didn't bother with regexps very often, so count my opinion as the
opinion of some random user.

>  one-or-more 1+ +
>  zero-or-more 0+ *
>  zero-or-one optional opt ?

+, *, ?, because they are short and what I'm used to from string regexps.

>  repeat **  (for lower-upper-bounded repetition)
>  repeat =   (for exact-count repetition)
>  any char in
>  not-newline nonl
>  group submatch

Don't have preferences yet here.

>  line-start bol
>  line-end eol
>  string-start buffer-start bos bot
>  string-end buffer-end eos eot
>  word-start bow
>  word-end eow

bol, eol, bos, eos, bow, eow: short and easy to remember.

>  sequence seq and :

I guess seq.

>  or |

|.

>  regexp regex

I always look up which is the right one.  Oh - they work both?  Cool.


Michael.