Lack of tooling slowing down contributions

Lack of tooling slowing down contributions

[Damien Cassou]

Hi,

I've submitted several patches to Emacs.  Because I don't do it very
often, I tend to forget the details in CONTRIBUTE.  Because I'm lazy,
I'm not reading CONTRIBUTE before each new contribution.  As a result,
my patches are not good enough and reviewers have to spend time telling
me how to do things.  I don't think reviewers should spend their
valuable time checking for details that can be checked by a machine.

These checks could be done either (by order of preference):

1. on the contributor's Emacs instance while s·he writes (e.g., flymake
   on .texi files and commit messages);

2. on the contributor's system before s·he sends the patches (e.g.,
   ./check_patches.sh *.patch);

3. on a continuous integration server that would automatically answer by
   email on thread with patches, listing the problems.

Here is a list of checks that could probably be automatized, saving time
for reviewers and contributors:

- the commit message format (e.g., dots after every sentence,
  double-space after dots, no indentation, line length, the
  Copyright-paperwork-exempt token, every single change is documented,
  no colon if another function of the same file has the same comment…)

- NEWS format (e.g., there is always either a '+++' or a '---', the
  first sentence stops at the first line, double space after dots, …)

- the .texi format (e.g., double space after dots…)

- the elisp format (e.g., a :version property on new defcustoms, all
  arguments are listed on the first docstring line…)

It would also be nice that tooling within Emacs helps contributors get
their contributions right. For example, I would expect M-q to add the
double-spacing at the end of sentences.

BTW, can you please C.C. me if you answer this message because I'm not
subscribed to the mailing list?

Thanks for reading and sorry for the length of this email.

-- 
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Lack of tooling slowing down contributions

[Eli Zaretskii]

> From: Damien Cassou <damien@cassou.me>
> Date: Fri, 14 Jun 2019 09:38:00 +0200
> 
> I've submitted several patches to Emacs.  Because I don't do it very
> often, I tend to forget the details in CONTRIBUTE.  Because I'm lazy,
> I'm not reading CONTRIBUTE before each new contribution.  As a result,
> my patches are not good enough and reviewers have to spend time telling
> me how to do things.  I don't think reviewers should spend their
> valuable time checking for details that can be checked by a machine.
> 
> These checks could be done either (by order of preference):

I suggest to file a feature-request bug report with these proposals.

Thanks.

Re: Lack of tooling slowing down contributions

[Noam Postavsky]

On Fri, 14 Jun 2019 at 03:38, Damien Cassou <damien@cassou.me> wrote:

> It would also be nice that tooling within Emacs helps contributors get
> their contributions right. For example, I would expect M-q to add the
> double-spacing at the end of sentences.

If you have set sentence-end-double-space to t, then text like

foo bar.
etc etc.

Will be refilled by M-q with double spaces. But if you write

foo bar. etc etc.

I.e, using single spacing, then M-q won't change that, because Emacs
doesn't recognize that as being two separate sentences.

Re: Lack of tooling slowing down contributions

[Juri Linkov]

>> It would also be nice that tooling within Emacs helps contributors get
>> their contributions right. For example, I would expect M-q to add the
>> double-spacing at the end of sentences.
>
> If you have set sentence-end-double-space to t, then text like
>
> foo bar.
> etc etc.
>
> Will be refilled by M-q with double spaces. But if you write
>
> foo bar. etc etc.
>
> I.e, using single spacing, then M-q won't change that, because Emacs
> doesn't recognize that as being two separate sentences.

This feature is sorely missed: every time I copy text from web browsers
to Emacs I have to manually insert additional space at the end of every
copied sentence before typing M-q.

Re: Lack of tooling slowing down contributions

[Stefan Monnier]

>> I.e, using single spacing, then M-q won't change that, because Emacs
>> doesn't recognize that as being two separate sentences.
>
> This feature is sorely missed: every time I copy text from web browsers
> to Emacs I have to manually insert additional space at the end of every
> copied sentence before typing M-q.

What would you want it to do? 
Does setting sentence-end-double-space to nil solve it?
If not, what's the difference?  Would you want to recognize single-space
end of sentence but change them to a double space?


        Stefan

Re: Lack of tooling slowing down contributions

[Phil Sainty]

On 2019-06-17 07:20, Juri Linkov wrote:
>>> It would also be nice that tooling within Emacs helps contributors 
>>> get
>>> their contributions right. For example, I would expect M-q to add the
>>> double-spacing at the end of sentences.
>> 
>> If you have set sentence-end-double-space to t, then text like
>> 
>> foo bar.
>> etc etc.
>> 
>> Will be refilled by M-q with double spaces. But if you write
>> 
>> foo bar. etc etc.
>> 
>> I.e, using single spacing, then M-q won't change that, because Emacs
>> doesn't recognize that as being two separate sentences.
> 
> This feature is sorely missed: every time I copy text from web browsers
> to Emacs I have to manually insert additional space at the end of every
> copied sentence before typing M-q.


You can't do this automatically, can you?  One of the reasons that the
double-spaced sentence convention is useful in the first place is that
it unambiguously differentiates sentence endings from other uses of that
punctuation character which do *not* indicate the end of a sentence.

E.g. for example, should not become "E.g.  for example".

P. G. Wodehouse should not end up as "P.  G.  Wodehouse".

Prof. Moriarty should not end up as "Prof.  Moriarty"

When the same punctuation character is used for multiple things, I don't
imagine there's any way for Emacs to do this accurately, short of 
natural-
language parsing (which sounds like a giant can of worms) ?


-Phil

Re: Lack of tooling slowing down contributions

[Van L]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 2019-06-17 07:20, Juri Linkov wrote:
... [snip]
>>
>> This feature is sorely missed: every time I copy text from web browsers
>> to Emacs I have to manually insert additional space at the end of every
>> copied sentence before typing M-q.
>
... [snip]
> E.g. for example, should not become "E.g.  for example".
>
> P. G. Wodehouse should not end up as "P.  G.  Wodehouse".
>
> Prof. Moriarty should not end up as "Prof.  Moriarty"
>
> When the same punctuation character is used for multiple things, I don't
> imagine there's any way for Emacs to do this accurately, short of
> natural-
> language parsing (which sounds like a giant can of worms) ?

You can keep score in an
'admin/ambitious-challenges-TODO' file for people
to throw in the can an amount of $1 or €1 or ¥100 or 10¢ or 2¢ or
as much as they like for when the challenge is
solved by the GNU hero in the future.

For me, the double-space sentence ending is easier to read.

-- 
© 2019 VanL
gpg using EEF2 37E9 3840 0D5D 9183  251E 9830 384E 9683 B835
          'The mouth is the dirtiest part on the human body.' - Bill Wallace

Re: Lack of tooling slowing down contributions

[Damien Cassou]

Juri Linkov <juri@linkov.net> writes:
> This feature is sorely missed: every time I copy text from web browsers
> to Emacs I have to manually insert additional space at the end of every
> copied sentence before typing M-q.

Please discuss on bug 36208.

-- 
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Lack of tooling slowing down contributions

[Marcin Borkowski]

On 2019-06-17, at 01:06, Phil Sainty <psainty@orcon.net.nz> wrote:

> On 2019-06-17 07:20, Juri Linkov wrote:
>>>> It would also be nice that tooling within Emacs helps contributors
>>>> get
>>>> their contributions right. For example, I would expect M-q to add the
>>>> double-spacing at the end of sentences.
>>>
>>> If you have set sentence-end-double-space to t, then text like
>>>
>>> foo bar.
>>> etc etc.
>>>
>>> Will be refilled by M-q with double spaces. But if you write
>>>
>>> foo bar. etc etc.
>>>
>>> I.e, using single spacing, then M-q won't change that, because Emacs
>>> doesn't recognize that as being two separate sentences.
>>
>> This feature is sorely missed: every time I copy text from web browsers
>> to Emacs I have to manually insert additional space at the end of every
>> copied sentence before typing M-q.
>
>
> You can't do this automatically, can you?  One of the reasons that the
> double-spaced sentence convention is useful in the first place is that
> it unambiguously differentiates sentence endings from other uses of that
> punctuation character which do *not* indicate the end of a sentence.
>
> E.g. for example, should not become "E.g.  for example".
>
> P. G. Wodehouse should not end up as "P.  G.  Wodehouse".
>
> Prof. Moriarty should not end up as "Prof.  Moriarty"
>
> When the same punctuation character is used for multiple things, I don't
> imagine there's any way for Emacs to do this accurately, short of
> natural-
> language parsing (which sounds like a giant can of worms) ?
>
>
> -Phil

TeX uses a nice approximate solution where a period after a capital
letter does not mean a sentence ending.  This does not solve everything
(of course), but helps in cases like "P. G. Wodehouse" (I love his
Bertram Wooster stories, btw).  On the other hand, it can be tripped by
things like "He used to work in NASA.  Now he's a full-time comic strip
author".

Best,

--
Marcin Borkowski
http://mbork.pl

Re: Lack of tooling slowing down contributions

[Yuri Khan]

On Mon, Jun 17, 2019 at 4:56 PM Marcin Borkowski <mbork@mbork.pl> wrote:

> TeX uses a nice approximate solution where a period after a capital
> letter does not mean a sentence ending.  This does not solve everything
> (of course), but helps in cases like "P. G. Wodehouse" (I love his
> Bertram Wooster stories, btw).  On the other hand, it can be tripped by
> things like "He used to work in NASA.  Now he's a full-time comic strip
> author".

I have a hypothesis that in all cases where U+002E FULL STOP is used
in a role other than sentence ending punctuation, it should either not
be followed by any spaces (as in decimals such as 3.14 or object
member access like rect.width), or be followed by a U+00A0 NO-BREAK
SPACE rather than any number of ordinary U+0020 SPACEs (as in
contractions and initials such as e.~g., Dr.~Pepper, or
P.~G.~Wodehouse). So any false positive of the simple rule “Every
U+002E U+0020 is a sentence end” is a typographical mistake to be
fixed, rather than a legitimate exception to be worked around. Any
counterexamples?

On the other hand, I have another hypothesis that this convention
would be much harder to enforce than the double-spacing one.

Re: Lack of tooling slowing down contributions

[Juri Linkov]

>>> I.e, using single spacing, then M-q won't change that, because Emacs
>>> doesn't recognize that as being two separate sentences.
>>
>> This feature is sorely missed: every time I copy text from web browsers
>> to Emacs I have to manually insert additional space at the end of every
>> copied sentence before typing M-q.
>
> What would you want it to do?
> Does setting sentence-end-double-space to nil solve it?
> If not, what's the difference?

I tried to set sentence-end-double-space to nil, but reverted back to t
because paragraphs with sentences separated by double space are easier to read
when editing text using a monospaced font in Emacs (there is no such problem
with proportional fonts in web browsers).

> Would you want to recognize single-space end of sentence but change
> them to a double space?

Yes, maybe a separate command is needed, e.g. M-x sentence-use-double-space
that would recognize single-space that doesn't separate an abbreviation
and insert additional space.

Re: Lack of tooling slowing down contributions

[Juri Linkov]

>> This feature is sorely missed: every time I copy text from web browsers
>> to Emacs I have to manually insert additional space at the end of every
>> copied sentence before typing M-q.
>
> You can't do this automatically, can you?  One of the reasons that the
> double-spaced sentence convention is useful in the first place is that
> it unambiguously differentiates sentence endings from other uses of that
> punctuation character which do *not* indicate the end of a sentence.
>
> E.g. for example, should not become "E.g.  for example".
>
> P. G. Wodehouse should not end up as "P.  G.  Wodehouse".
>
> Prof. Moriarty should not end up as "Prof.  Moriarty"
>
> When the same punctuation character is used for multiple things, I don't
> imagine there's any way for Emacs to do this accurately, short of natural-
> language parsing (which sounds like a giant can of worms) ?

I believe it's possible to find some simple heuristics (maybe even
just regexp-based) that would cover more than 90% of cases.

And like when a spell-checker finds an unknown word, the user can add it
to the personal dictionary, a new double-spacer command should allow
the user to add exceptions to the personal customization.

Re: Lack of tooling slowing down contributions

[Juri Linkov]

>> E.g. for example, should not become "E.g.  for example".
>>
>> P. G. Wodehouse should not end up as "P.  G.  Wodehouse".
>>
>> Prof. Moriarty should not end up as "Prof.  Moriarty"
>
> I believe it's possible to find some simple heuristics (maybe even
> just regexp-based) that would cover more than 90% of cases.

A simple heuristics relying on the sentence length would cover most cases.
Here is experimental code that works surprisingly well for adding double space
to sentences longer than 5 letters:

  (defun canonically-double-space-region (beg end)
    (interactive "*r")
    (canonically-space-region beg end)
    (unless (markerp end) (setq end (copy-marker end t)))
    (let* ((sentence-end-double-space nil) ; to get right regexp below
           (end-spc-re (concat "\\([^.,]\\{5,\\}\\)\\(?:" (sentence-end) "\\)")))
      (save-excursion
        (goto-char beg)
        (while (and (< (point) end)
                    (re-search-forward end-spc-re end t))
          (unless (or (>= (point) end)
                      (looking-back "[[:space:]]\\{2\\}\\|\n"))
            (insert " "))))))

  (advice-add 'fill-paragraph :before
              (lambda (&rest _args)
                (when (use-region-p)
                  (canonically-double-space-region
                   (region-beginning)
                   (region-end))))
              '((name . fill-paragraph-double-space)))

It's one drawback is that it doesn't handle such false negatives as short
sentences.  Alas. But such cases are extremely rare.  False positives
such as long abbreviations are rare too.

BTW, I tried to replace the regexp above with ‘rx’ that is easier to read:

  (rx (group (>= 5 (not (any " ."))))
      (group (regexp (sentence-end))))

but it fails.  Not sure if this will be fixed in bug#36237.

Re: Lack of tooling slowing down contributions

[Andy Moreton]

On Wed 19 Jun 2019, Juri Linkov wrote:
> BTW, I tried to replace the regexp above with ‘rx’ that is easier to read:
>
>   (rx (group (>= 5 (not (any " ."))))
>       (group (regexp (sentence-end))))
>
> but it fails.  Not sure if this will be fixed in bug#36237.

The regexp subform takes a string argument, so you need:

   `(rx (group (>= 5 (not (any " ."))))
        (group (regexp ,(sentence-end))))

HTH,

    AndyM

Re: Lack of tooling slowing down contributions

[Juri Linkov]

>> BTW, I tried to replace the regexp above with ‘rx’ that is easier to read:
>>
>>   (rx (group (>= 5 (not (any " ."))))
>>       (group (regexp (sentence-end))))
>>
>> but it fails.  Not sure if this will be fixed in bug#36237.
>
> The regexp subform takes a string argument, so you need:
>
>    `(rx (group (>= 5 (not (any " ."))))
>         (group (regexp ,(sentence-end))))

But 'eval' doesn't require backquoting:

  (rx (group (>= 5 (not (any " ."))))
      (group (eval (sentence-end))))

The only problem with 'eval' is that it applies 'regexp-quote'
on the result of evaluation that is counterproductive here.

Re: Lack of tooling slowing down contributions

[Noam Postavsky]

On Wed, 19 Jun 2019 at 17:18, Juri Linkov <juri@linkov.net> wrote:

> >>   (rx (group (>= 5 (not (any " ."))))
> >>       (group (regexp (sentence-end))))
> >>
> >> but it fails.  Not sure if this will be fixed in bug#36237.

Yes, it will.

>   (rx (group (>= 5 (not (any " ."))))
>       (group (eval (sentence-end))))
>
> The only problem with 'eval' is that it applies 'regexp-quote'
> on the result of evaluation that is counterproductive here.

The other problem with 'eval' is that it calls 'sentence-end' at
macroexpansion time.