replace-regexp

replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

I was using `replace-regexp' and it worked fine but then
I byte-compiled and it told me something that was in the help
as well, namely

  This function is for interactive use only; in Lisp code use
  `re-search-forward' and `replace-match' instead.

so I wrote this and it does the job but reports the error
"while: Invalid search bound (wrong side of point)"

(defun md-latex (beg end)
  (interactive "r")
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "_\\(.*\\)_" end t)
      (replace-match  "\\\\textit{\\1}") )))

Also this "interactive use only", in general, what's the deal
with that?

If it is just a matter of re-writing it the way I did (only
without the error) I don't see why that cannot be dealt with
so the user can just call the/a function with a simple
interface, be it interactively or in Lisp...

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Tassilo Horn]

Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> writes:

> so I wrote this and it does the job but reports the error
> "while: Invalid search bound (wrong side of point)"
>
> (defun md-latex (beg end)
>   (interactive "r")
>   (save-excursion
>     (goto-char beg)
>     (while (re-search-forward "_\\(.*\\)_" end t)
>       (replace-match  "\\\\textit{\\1}") )))

I think you didn't call your command interactively but through code like
(md-latex 170 82) where the BEG was larger than END.  When being called
interactively the "r" spec makes sure that BEG is always the smaller
one, no matter if point is before mark or the other way round.

> Also this "interactive use only", in general, what's the deal with
> that?
>
> If it is just a matter of re-writing it the way I did (only without
> the error) I don't see why that cannot be dealt with so the user can
> just call the/a function with a simple interface, be it interactively
> or in Lisp...

I guess it's because `replace-regexp' does much more than just
`re-search-forward' and `replace-match' in order to provide a convenient
interface for the user.  Also it supports stuff like \? as replacement
which will query the user.  You'd normally wouldn't want/need that.  So
if all you want is replace some regexp, `re-search-forward' with
`replace-match' is the non-overhead way to go.

Bye,
Tassilo

Re: replace-regexp

[Jean Louis]

* Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> [2021-05-07 02:07]:
> I was using `replace-regexp' and it worked fine but then
> I byte-compiled and it told me something that was in the help
> as well, namely

Why not use 

replace-regexp-in-string is a compiled Lisp function in ‘subr.el’.

(replace-regexp-in-string REGEXP REP STRING &optional FIXEDCASE
LITERAL SUBEXP START)


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Tassilo Horn wrote:

>> so I wrote this and it does the job but reports the error
>> "while: Invalid search bound (wrong side of point)"
>>
>> (defun md-latex (beg end)
>>   (interactive "r")
>>   (save-excursion
>>     (goto-char beg)
>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>       (replace-match  "\\\\textit{\\1}") )))
>
> I think you didn't call your command interactively but
> through code like (md-latex 170 82) where the BEG was larger
> than END. When being called interactively the "r" spec makes
> sure that BEG is always the smaller one, no matter if point
> is before mark or the other way round.

Nope, I called it with M-x. Tried again now, still the
same error. It works but I get the error.

Maybe it has something to do with the replacement being longer
than the original string?

>> If it is just a matter of re-writing it the way I did (only
>> without the error) I don't see why that cannot be dealt
>> with so the user can just call the/a function with a simple
>> interface, be it interactively or in Lisp...
>
> I guess it's because `replace-regexp' does much more than
> just `re-search-forward' and `replace-match' in order to
> provide a convenient interface for the user. Also it
> supports stuff like \? as replacement which will query the
> user. You'd normally wouldn't want/need that. So if all you
> want is replace some regexp, `re-search-forward' with
> `replace-match' is the non-overhead way to go.

But why can't there be a function that does this so one
doesn't have to write all that Lisp for every use case?

Also, how about a general function that branches on
"interactiveness"?

It gets even more strange since the non-recommended function
works just fine...

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Jean Louis wrote:

>> I was using `replace-regexp' and it worked fine but then
>> I byte-compiled and it told me something that was in the
>> help as well, namely
>
> Why not use 
>
> replace-regexp-in-string is a compiled Lisp function in ‘subr.el’.
>
> (replace-regexp-in-string REGEXP REP STRING &optional
> FIXEDCASE LITERAL SUBEXP START)

Because I want to use it directly to the buffer region...

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

>>> so I wrote this and it does the job but reports the error
>>> "while: Invalid search bound (wrong side of point)"
>>>
>>> (defun md-latex (beg end)
>>>   (interactive "r")
>>>   (save-excursion
>>>     (goto-char beg)
>>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>>       (replace-match  "\\\\textit{\\1}") )))
>>
>> I think you didn't call your command interactively but
>> through code like (md-latex 170 82) where the BEG was
>> larger than END. When being called interactively the "r"
>> spec makes sure that BEG is always the smaller one, no
>> matter if point is before mark or the other way round.
>
> Nope, I called it with M-x. Tried again now, still the same
> error. It works but I get the error.
>
> Maybe it has something to do with the replacement being
> longer than the original string?

That's it, I think, as this doesn't give me the error:

(defun md-latex-reduce (beg end)
  (interactive "r")
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "_\\(.*\\)_" end t)
      (replace-match "\\1") )))

(Now the function doesn't make any sense, so this is just to
try my theory...)

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

>>>> so I wrote this and it does the job but reports the error
>>>> "while: Invalid search bound (wrong side of point)"
>>>>
>>>> (defun md-latex (beg end)
>>>>   (interactive "r")
>>>>   (save-excursion
>>>>     (goto-char beg)
>>>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>>>       (replace-match  "\\\\textit{\\1}") )))
>>>
>>> I think you didn't call your command interactively but
>>> through code like (md-latex 170 82) where the BEG was
>>> larger than END. When being called interactively the "r"
>>> spec makes sure that BEG is always the smaller one, no
>>> matter if point is before mark or the other way round.
>>
>> Nope, I called it with M-x. Tried again now, still the same
>> error. It works but I get the error.
>>
>> Maybe it has something to do with the replacement being
>> longer than the original string?
>
> That's it, I think, as this doesn't give me the error:
>
> (defun md-latex-reduce (beg end)
>   (interactive "r")
>   (save-excursion
>     (goto-char beg)
>     (while (re-search-forward "_\\(.*\\)_" end t)
>       (replace-match "\\1") )))
>
> (Now the function doesn't make any sense, so this is just to
> try my theory...)

I thought it might be a configuration issue but I've now tried
with 'emacs -Q' and I get the samme error. So either that's
not the correct way to put it or it is a bug.

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.5, cairo version 1.16.0) of 2020-10-23

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Yuri Khan]

On Sat, 8 May 2021 at 07:02, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> > Maybe it has something to do with the replacement being
> > longer than the original string?
>
> That's it, I think, as this doesn't give me the error:
>
> (defun md-latex-reduce (beg end)
>   (interactive "r")
>   (save-excursion
>     (goto-char beg)
>     (while (re-search-forward "_\\(.*\\)_" end t)
>       (replace-match "\\1") )))

And this is probably one of the reasons why most[citation needed]
tutorials, when implementing a function operating on a region, do a
‘narrow-to-region’ first thing.

    (defun md-latex-reduce (beg end)
      (interactive "r")
      (save-excursion
        (save-restriction
          (narrow-to-region beg end)

          (goto-char (point-min))
          (while (re-search-forward "_\\(.*\\)_" nil t)
            (replace-match "\\\\textit{\\1}"))))

You might probably solve the same issue by carefully tracking the
lengthening of text — e.g. by incrementing ‘end’ by (- (length
"\\\\textit{}") (length "__")) on each iteration — or by putting a
marker at end and then passing that marker in re-search-forward. But
narrowing is easier.

Re: replace-regexp

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> > Maybe it has something to do with the replacement being
>> > longer than the original string?
>>
>> That's it, I think, as this doesn't give me the error:
>>
>> (defun md-latex-reduce (beg end)
>>   (interactive "r")
>>   (save-excursion
>>     (goto-char beg)
>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>       (replace-match "\\1") )))
>
> And this is probably one of the reasons why most[citation needed]
> tutorials, when implementing a function operating on a region, do a
> ‘narrow-to-region’ first thing.
>
>     (defun md-latex-reduce (beg end)
>       (interactive "r")
>       (save-excursion
>         (save-restriction
>           (narrow-to-region beg end)
>
>           (goto-char (point-min))
>           (while (re-search-forward "_\\(.*\\)_" nil t)
>             (replace-match "\\\\textit{\\1}"))))
>
> You might probably solve the same issue by carefully tracking the
> lengthening of text — e.g. by incrementing ‘end’ by (- (length
> "\\\\textit{}") (length "__")) on each iteration — or by putting a
> marker at end and then passing that marker in re-search-forward. But
> narrowing is easier.

I recommend avoiding the use of narrowing as much as possible since it
can have other undesirable effects (e.g. it changes the result of
things like `syntax-ppss`).

There are indeed 2 "standard" solutions:
1- start your loop at the end, moving towards the beginning.
2- use a marker to keep track of the end.

You say "narrowing is easier", but using a marker is as simple as:

    (setq end (copy-marker end t))

so it's arguably simpler.


        Stefan

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> There are indeed 2 "standard" solutions:
> 1- start your loop at the end, moving towards the beginning.
> 2- use a marker to keep track of the end.

That so? Here is what it says in the help:

  This function is usually the wrong thing to use in a Lisp program.
  What you probably want is a loop like this:
    (while (re-search-forward REGEXP nil t)
      (replace-match TO-STRING nil nil))
  which will run faster and will not set the mark or print anything.

Not so good, ey? Standard solution 2 not mentioned and, yes,
isn't it the opposite of standard solution 1?

Again, I don't think the user, not even the Lisp user, should
do this thing. I'll see if I can show you what I mean, later.
As a side note, (replace-match TO-STRING nil nil) - but aren't
nil nil the default anyway?

> You say "narrowing is easier", but using a marker is as
> simple as:
>
>     (setq end (copy-marker end t))
>
> so it's arguably simpler.

You mean like this? Yes, that works with no error.

(defun md-latex (beg end)
  (interactive "r")
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "_\\(.*\\)_" end t)
      (setq end (copy-marker end t))
      (replace-match "\\\\textit{\\1}") )))

I have used replace-match 6 times before, I wonder why I never
had a problem until now (and now the problem appeared
instantly). Examining...

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> There are indeed 2 "standard" solutions:
> 1- start your loop at the end, moving towards the beginning.

You mean like this? Also works. Maybe better than the
`copy-marker' stuff?

(defun md-latex-back (beg end)
  (interactive "r")
  (save-excursion
    (goto-char end)
    (while (re-search-backward "_\\(.*\\)_" beg t)
      (replace-match "\\\\textit{\\1}") )))

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Stefan Monnier via Users list for the GNU Emacs text editor]

> (defun md-latex (beg end)
>   (interactive "r")
>   (save-excursion
>     (goto-char beg)
>     (while (re-search-forward "_\\(.*\\)_" end t)
>       (setq end (copy-marker end t))
>       (replace-match "\\\\textit{\\1}") )))

Almost: move the (setq end (copy-marker end t)) outside the loop.


        Stefan

Re: replace-regexp

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> There are indeed 2 "standard" solutions:
>> 1- start your loop at the end, moving towards the beginning.
>
> You mean like this?
>
> (defun md-latex-back (beg end)
>   (interactive "r")
>   (save-excursion
>     (goto-char end)
>     (while (re-search-backward "_\\(.*\\)_" beg t)
>       (replace-match "\\\\textit{\\1}") )))

Yup.

> Also works.  Maybe better than the `copy-marker' stuff?

My comment wasn't just for regexp-replacement but more generally for
operating on a region.  So better or not will depend on the
specific case.  In many cases either way is about as good.


        Stefan

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> My comment wasn't just for regexp-replacement but more
> generally for operating on a region. So better or not will
> depend on the specific case. In many cases either way is
> about as good.

In general I don't like to use `setq' and in this case it also
makes the code more cryptic.

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

>> (defun md-latex (beg end)
>>   (interactive "r")
>>   (save-excursion
>>     (goto-char beg)
>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>       (setq end (copy-marker end t))
>>       (replace-match "\\\\textit{\\1}") )))
>
> Almost: move the (setq end (copy-marker end t)) outside
> the loop.

What does it do to end that makes it work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Tassilo Horn]

Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> writes:

> Stefan Monnier via Users list for the GNU Emacs text editor wrote:
>
>>> (defun md-latex (beg end)
>>>   (interactive "r")
>>>   (save-excursion
>>>     (goto-char beg)
>>>     (while (re-search-forward "_\\(.*\\)_" end t)
>>>       (setq end (copy-marker end t))
>>>       (replace-match "\\\\textit{\\1}") )))
>>
>> Almost: move the (setq end (copy-marker end t)) outside
>> the loop.
>
> What does it do to end that makes it work?

The t in (copy-marker end t) says that the marked should stay after
inserted text, i.e., it automatically moves backwards if text is
inserted before it.  With your original example, end was just a position
(integer) which did not change and the last replacement made point move
beyond it.

Bye,
Tassilo

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Tassilo Horn wrote:

>> What does it do to end that makes it work?
>
> The t in (copy-marker end t) says that the marked should
> stay after inserted text, i.e., it automatically moves
> backwards if text is inserted before it. With your original
> example, end was just a position (integer) which did not
> change and the last replacement made point move beyond it.

Yes, I suspected something to that "end", but how can it work
if just done once? It enables some function that will
update end every time mark is, perhaps...

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: replace-regexp

[Drew Adams]

> Yes, I suspected something to that "end", but how can it work
> if just done once? It enables some function that will
> update end every time mark is, perhaps...
> 
> --
> underground experts united
> https://urldefense.com/v3/__https://dataswamp.org/*incal__;fg!!GqivPVa7Brio!P
> DWWg-SvTuQaqe46FNHFNJyefPVOI9rEF77mcFxa8apnRdrIEs_C8lfNIYFDCQkj$

https://www.gnu.org/software/emacs/manual/html_node/elisp/Markers.html

https://www.gnu.org/software/emacs/manual/html_node/elisp/Marker-Insertion-Types.html

avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> I recommend avoiding the use of narrowing as much as
> possible since it can have other undesirable effects (e.g.
> it changes the result of things like `syntax-ppss`).

I checked my Elisp for `narrow-to-region' and found just one
use, namely

(defun sort-lines-random (beg end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil
                 #'forward-line
                 #'end-of-line
                 nil nil
                 (lambda (_ __) (zerop (random 2)) )))))
(defalias 'r #'sort-lines-random)

;; test here:
;; aaa
;; ccc
;; ddd
;; bbb

If one were to remove it, how would one specify the end of
the operation?

https://dataswamp.org/~incal/emacs-init/sort-incal.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Stefan Monnier via Users list for the GNU Emacs text editor]

Emanuel Berg via Users list for the GNU Emacs text editor [2021-05-08 23:54:03] wrote:
> Stefan Monnier via Users list for the GNU Emacs text editor wrote:
>> My comment wasn't just for regexp-replacement but more
>> generally for operating on a region. So better or not will
>> depend on the specific case. In many cases either way is
>> about as good.
> In general I don't like to use `setq' and in this case it also

You can use (let ((end (copy-marker end t))) ...), of course.
I don't find it cryptic at all, but I'm probably not well placed to judge.
In any case, my main point was that using narrowing can have undesirable
side effects because narrowing has a far-reaching impact on operations
in the buffer, so depending on what you need to do within your loop you
might prefer to stay away from it (unless this far-reaching impact is
exactly what you're looking for, obviously).


        Stefan

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

>>> My comment wasn't just for regexp-replacement but more
>>> generally for operating on a region. So better or not will
>>> depend on the specific case. In many cases either way is
>>> about as good.
>>
>> In general I don't like to use `setq' and in this case it
>> also
>
> You can use (let ((end (copy-marker end t))) ...), of
> course. I don't find it cryptic at all, but I'm probably not
> well placed to judge.

Let's say to a person who knows programming and Lisp to some
extent but isn't an expert on Emacs internals.

> In any case, my main point was that using narrowing can have
> undesirable side effects because narrowing has
> a far-reaching impact on operations in the buffer, so
> depending on what you need to do within your loop you might
> prefer to stay away from it (unless this far-reaching impact
> is exactly what you're looking for, obviously).

Yeah, that I'll not use.

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> depending on what you need to do within your loop you might
>> prefer to stay away from it (unless this far-reaching impact
>> is exactly what you're looking for, obviously).
> Yeah, that I'll not use.

Note that occasionally you don't really get to choose (e.g. with
`sort-subr`, IIRC).


        Stefan

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

>>> depending on what you need to do within your loop you
>>> might prefer to stay away from it (unless this
>>> far-reaching impact is exactly what you're looking for,
>>> obviously).
>>
>> Yeah, that I'll not use.
>
> Note that occasionally you don't really get to choose (e.g.
> with `sort-subr`, IIRC).

Ah, you read my other post, got me "fooled" there for a
second :)

OK, so that's a good use case then...

-- 
underground experts united
https://dataswamp.org/~incal

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

> Again, I don't think the user, not even the Lisp user,
> should do this thing. I'll see if I can show you what
> I mean, later.

Here, what is wrong with this?

(defun replace-regexp-1 (regexp to-string &optional beg end)
  (interactive
   `(,(read-from-minibuffer "regexp: ")
     ,(read-from-minibuffer "to string: ")
     ,@(if (use-region-p)
           (list (region-beginning) (region-end))
         (list (point-min) (point-max))) ))
  (let ((beg-set (or beg (point-min)))
        (end-set (or end (point-max))) )
    (save-excursion
      (goto-char end-set)
      (while (re-search-backward regexp beg-set t)
        (replace-match to-string) ))))

Works interactively and non-interactively using the same good
function... or?

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Yuri Khan]

On Sun, 9 May 2021 at 06:00, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

>       (sort-subr nil
>                  #'forward-line
>                  #'end-of-line
>                  nil nil
>                  (lambda (_ __) (zerop (random 2)) )))))

Note that this is not a suitable sorting predicate. It violates all
axioms of a strict weak ordering:

* Consistency: If (f a b) returns t once, it must return t when called
again with the same arguments.
* Irreflexivity: (f a a) must return nil. Your predicate returns nil
or t randomly.
* Antisymmetry: of (f a b) and (f b a), no more than one may return t.
In your case, both can return t.
* Transitivity: If (f a b) and (f b c) both return t, (f a c) must
also return t.
* Transitivity of equivalence: if (f a b), (f b a), (f b c), (f c b)
all return nil, then (f a c) and (f c a) must also return nil.

Generic sorting algorithms typically require the predicate to conform
to all of the above; otherwise, they may signal an error or enter an
endless loop.

I have not analyzed whether ‘sort-subr’ has any issues with
inconsistent orderings, and its docstring does not mention these
requirements, but it would be a good idea to avoid that anyway.

Also, using a sorting algorithm to randomize an ordered sequence is a
bit of an overkill. Sorting has an asymptotic complexity of O(n log
n), but the Fisher–Yates–Durstenfeld shuffle algorithm is O(n).

https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

>>       (sort-subr nil
>>                  #'forward-line
>>                  #'end-of-line
>>                  nil nil
>>                  (lambda (_ __) (zerop (random 2)) )))))
>
> Note that this is not a suitable sorting predicate.
> It violates all axioms of a strict weak ordering

That's right, I am. But we well know that the reason most of
us are here is because of our affinity for disobedience.

> * Consistency: If (f a b) returns t once, it must return
> t when called again with the same arguments. [...]

But here, we want to randomize _every time_.

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Yuri Khan]

On Sun, 9 May 2021 at 13:10, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> > * Consistency: If (f a b) returns t once, it must return
> > t when called again with the same arguments. [...]
>
> But here, we want to randomize _every time_.

Every time you do a random shuffle, but not within a single shuffle.

A theoretically sound (but still unnecessarily slow) sorting-based
shuffle would first assign a random weight to each line, and then sort
by that weight.

But contrast with this:

* Count the lines to be shuffled. Call it N.
* Generate a random number K_1 from 1 to N. Swap line 1 with line K_1.
* Generate a random number K_2 from 2 to N. Swap line 2 with line K_2.
…
* Generate a random number K_{N-1} from N-1 to N. Swap line N-1 with
line K_{N-1}.

Almost easy enough to put in Elisp, and guarantees uniform
distribution (as long as individual random indexes are uniformly
distributed).

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

>>> Consistency: If (f a b) returns t once, it must return
>>> t when called again with the same arguments. [...]
>>
>> But here, we want to randomize _every time_.
>
> Every time you do a random shuffle, but not within
> a single shuffle.

Why not?

> A theoretically sound (but still unnecessarily slow)
> sorting-based shuffle would first assign a random weight to
> each line, and then sort by that weight.
>
> But contrast with this:
>
> * Count the lines to be shuffled. Call it N.
> * Generate a random number K_1 from 1 to N. Swap line 1 with line K_1.
> * Generate a random number K_2 from 2 to N. Swap line 2 with line K_2.
> ...
> * Generate a random number K_{N-1} from N-1 to N. Swap line N-1 with
> line K_{N-1}.

DIY Yuri.

> Almost easy enough to put in Elisp

...

> and guarantees uniform distribution (as long as individual
> random indexes are uniformly distributed).

BTW, here is my other stuff, I'm pretty sure it all conforms
to modern-day computer science theory of sorting, but I'll let
you verify it just to be sure.

  https://dataswamp.org/~incal/emacs-init/sort-incal.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Jean Louis]

* Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> [2021-05-09 10:01]:
> BWT, hree is my otehr sftuf, I'm ptrtey srue it all cnfomros 
> to modern-day comupter secince tehory of sotring, but Il'l let 
> you vrfeiy it jsut to be srue .
> 
>   https://dataswamp.org/~incal/emacs-init/sort-incal.el

I did not konw taht scrmabling txet is siecnce .

;;;; ━━━━━━━━━━━━━━━━━━━━━━━━━━━━
;;;;   SCRAMBLED TEXT FUNCTIONS
;;;; ━━━━━━━━━━━━━━━━━━━━━━━━━━━━

;; Reference:

;; Why your brain can read jumbled letters:
;; https://www.mnn.com/lifestyle/arts-culture/stories/why-your-brain-can-read-jumbled-letters

(defun scramble-word (word)
  "Randoimze the cahcraters of a sritng but not fisrt and lsat "
  (let* ((first (substring word 0 1))
	 (length (length word))
	 (last (substring word (1- length) length)))
    (if (<= length 3)
	word
      (if (= length 4)
	  (concat first (substring word 2 3) (substring word 1 2) last)
	(let* ((middle (substring word 1 (1- length)))
	       (rnd (length middle))
	       (empty-str "")
	       (chars (delete empty-str (split-string middle empty-str)))
	       (rand-chars (sort chars (lambda (_ __) (zerop (random rnd)))))
	       (rand-str (mapconcat 'identity rand-chars ""))
	       (new-word (concat first rand-str last)))
	  new-word)))))

(defun scramble-string (string)
  "Returns string of scrambled words"
  (let* ((split (split-string string)))
    (with-output-to-string
      (while split
	(princ (scramble-word (pop split)))
	(princ " ")))))

(defun scramble-region (start end)
  "Scramble mkared rgeoin of txet "
  (interactive "r")
  (if (region-active-p)
    (let* ((s (buffer-substring-no-properties start end))
           (replacement (scramble-string s)))
      (delete-region start end)
      (insert replacement))
    (message "Tehre was no aivcte reigon to scmrable ")))

(defun randomize-string (string)
  "Randomize the characters of a string"
  (let* ((rnd (length string))
	 (empty-str  "")
         (chars (delete empty-str (split-string string empty-str)))
         (rand-chars (sort chars (lambda (_ __) (zerop (random rnd)))))
         (rand-str (mapconcat 'identity rand-chars "")))
    rand-str))


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: replace-regexp

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

>>> depending on what you need to do within your loop you
>>> might prefer to stay away from it (unless this
>>> far-reaching impact is exactly what you're looking for,
>>> obviously).
>>
>> Yeah, that I'll not use.
>
> Note that occasionally you don't really get to choose (e.g.
> with `sort-subr`, IIRC).

What about this?

(defun sort-lines-random-2 (beg end)
  (interactive "r")
  (let ((ents  (buffer-substring-no-properties beg end))
        (start (point)) )
    (delete-region beg end)
    (insert (with-temp-buffer
              (insert ents)
              (goto-char (point-min))
              (sort-subr nil
                         #'forward-line
                         #'end-of-line
                         nil nil
                         (lambda (_ __) (zerop (random 2)) ))
              (buffer-substring-no-properties (point-min) (point-max))) )
    (goto-char start) ))
(defalias 'r #'sort-lines-random-2)

;; test here:
;; aaa
;; ccc
;; bbb
;; ddd

(message "%c%c%c%c%c%c%c" 115 107 105 108 108 115 33)

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Jean Louis wrote:

>> BWT, hree is my otehr sftuf, I'm ptrtey srue it all
>> cnfomros to modern-day comupter secince tehory of sotring,
>> but Il'l let you vrfeiy it jsut to be srue .
>> 
>>   https://dataswamp.org/~incal/emacs-init/sort-incal.el
>
> I did not konw taht scrmabling txet is siecnce .

Me neithre :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Yuri Khan]

On Sun, 9 May 2021 at 14:00, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> >>> Consistency: If (f a b) returns t once, it must return
> >>> t when called again with the same arguments. [...]
> >>
> >> But here, we want to randomize _every time_.
> >
> > Every time you do a random shuffle, but not within
> > a single shuffle.
>
> Why not?

From the Wikipedia article I linked to previously, which you
apparently chose to read later:

===
In principle this shuffling method can even result in program failures
like endless loops or access violations, because the correctness of a
sorting algorithm may depend on properties of the order relation (like
transitivity) that a comparison producing random values will certainly
not have.[12] While this kind of behaviour should not occur with
sorting routines that never perform a comparison whose outcome can be
predicted with certainty (based on previous comparisons), there can be
valid reasons for deliberately making such comparisons. For instance
the fact that any element should compare equal to itself allows using
them as sentinel value for efficiency reasons, and if this is the
case, a random comparison function would break the sorting algorithm.
===

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

> From the Wikipedia article I linked to previously, which you
> apparently chose to read later:
>
> In principle this shuffling method can even result in
> program failures like endless loops or access violations,
> because the correctness of a sorting algorithm may depend on
> properties of the order relation (like transitivity) that
> a comparison producing random values will certainly not
> have.

In principle maybe - but not in practice, it seems.

Proposal: Instead of quoting computer science theory, why
don't _you_ write some Elisp and show us how you think it
should be done?

It isn't because it is easier and more comfortable to broadly
criticize someone or something else, than to be active and
creative oneself - is it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Yuri Khan]

On Sun, 9 May 2021 at 17:54, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> Proposal: Instead of quoting computer science theory, why
> don't _you_ write some Elisp and show us how you think it
> should be done?

Mostly because I don’t usually write code to solve tasks I never
encounter in my own workflow. (Unless specifically employed to write
such code, or doing an interview with a prospective employer.)

> It isn't because it is easier and more comfortable to broadly
> criticize someone or something else, than to be active and
> creative oneself - is it?

I’m not criticizing you, I’m advising you about a possible issue with
your code. At the same time, I’m taking an opportunity to raise
general awareness of sorting predicate requirements.

Now, because you asked nicely, here’s my take on implementing the
random shuffle algorithm.

(defun yk-shuffle-lines (begin end)
  "Reorder lines between BEGIN and END randomly.
If BEGIN or END is in the middle of a line, that line is included.
Any markers inside the region may move
to the beginning or end of their respective lines,
and won’t follow the text they were associated with."
  (interactive "*r")
  (save-excursion
    (let ((nlines (count-lines begin end)))
      (goto-char begin)
      (while (> nlines 1)
        (let* ((begin1 (line-beginning-position))
               (line1 (let ((end1 (line-end-position)))
                        (prog1 (buffer-substring begin1 end1)
                          (delete-region begin1 end1))))
               (_ (forward-line (random nlines)))
               (line2 (let* ((begin2 (line-beginning-position))
                             (end2 (line-end-position)))
                        (prog1 (buffer-substring begin2 end2)
                          (delete-region begin2 end2)))))
          (insert line1)
          (goto-char begin1)
          (insert line2))
        (forward-line)
        (setq nlines (1- nlines))))))

(The structure of let*/let/let* bindings may seem excessively complex,
but I’m taking care here to not expose bindings beyond their validity.
E.g. as soon as (delete-region begin1 end1) is executed, end1 no
longer refers to anything meaningful, so I drop that binding asap.)

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

> (defun yk-shuffle-lines (begin end)
>   "Reorder lines between BEGIN and END randomly.
> If BEGIN or END is in the middle of a line, that line is included.
> Any markers inside the region may move
> to the beginning or end of their respective lines,
> and won’t follow the text they were associated with."
>   (interactive "*r")
>   (save-excursion
>     (let ((nlines (count-lines begin end)))
>       (goto-char begin)
>       (while (> nlines 1)
>         (let* ((begin1 (line-beginning-position))
>                (line1 (let ((end1 (line-end-position)))
>                         (prog1 (buffer-substring begin1 end1)
>                           (delete-region begin1 end1))))
>                (_ (forward-line (random nlines)))
>                (line2 (let* ((begin2 (line-beginning-position))
>                              (end2 (line-end-position)))
>                         (prog1 (buffer-substring begin2 end2)
>                           (delete-region begin2 end2)))))
>           (insert line1)
>           (goto-char begin1)
>           (insert line2))
>         (forward-line)
>         (setq nlines (1- nlines))))))
>
> (The structure of let*/let/let* bindings may seem
> excessively complex, but I’m taking care here to not expose
> bindings beyond their validity. E.g. as soon as
> (delete-region begin1 end1) is executed, end1 no longer
> refers to anything meaningful, so I drop that binding asap.)

Alright, that more like it. Scientific randomness at
you service!

Well, yes, `let' within a let body I think I've seen and even
done a couple of times, let inside a let varlist I believe is
a new experience but why not :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

> (defun yk-shuffle-lines (begin end)
>   "Reorder lines between BEGIN and END randomly.
> If BEGIN or END is in the middle of a line, that line is included.
> Any markers inside the region may move
> to the beginning or end of their respective lines,
> and won’t follow the text they were associated with."
>   (interactive "*r")
>   (save-excursion
>     (let ((nlines (count-lines begin end)))
>       (goto-char begin)
>       (while (> nlines 1)
>         (let* ((begin1 (line-beginning-position))
>                (line1 (let ((end1 (line-end-position)))
>                         (prog1 (buffer-substring begin1 end1)
>                           (delete-region begin1 end1))))
>                (_ (forward-line (random nlines)))
>                (line2 (let* ((begin2 (line-beginning-position))
>                              (end2 (line-end-position)))
>                         (prog1 (buffer-substring begin2 end2)
>                           (delete-region begin2 end2)))))
>           (insert line1)
>           (goto-char begin1)
>           (insert line2))
>         (forward-line)
>         (setq nlines (1- nlines))))))

Only the `save-excursion' seems to be insufficient? I don't
know when that works (most of the times) and when it doesn't
(sometimes) so this is as good a time as any to ask that...

OK, so now we have one hacker/hacky solution and one
scientist/scientific, so what is the next step? It is ...

the engineer solution!

Task: separate the material (i.e., the data type) from the
algorithm, so that the algorithm can be applied to any form of
data, be it a string, words on a line, lines as we have seen,
a list of arbitrary elements...

That way, we can do Jean's whole file scientific!

Only one problem ... who is the engineer around here?
Not Jean himself, is it??!!

Hahaha :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

> OK, so now we have one hacker/hacky solution and one
> scientist/scientific, so what is the next step? It is ...
>
> the engineer solution!
>
> Task: separate the material (i.e., the data type) from the
> algorithm, so that the algorithm can be applied to any form of
> data, be it a string, words on a line, lines as we have seen,
> a list of arbitrary elements...

Let's do it like this, if we can get the algorithm to work on
an arbitrary list, then all data-type issues can be solved
easily, by adding two functions, one disassembly function that
splits the data into a list, and one assembly function that
puts it back together, after the algorithm has randomized
it!

So for example if the data is the string "data":

1. "data" -> '("d" "a" "t" "a")
2. run the algorithm, get e.g '("a" "d" "t" "a")
3. '("a" "d" "t" "a") -> "adta"

Because Lisp is already based on the universal data structure
which can express all human knowledge, isn't it possible that
there is already a Lisp list randomizer that fulfills even
Yuri's most uncompromising demands?

-- 
underground experts united
https://dataswamp.org/~incal

Re: avoid narrow-to-region (was: Re: replace-regexp)

[Emanuel Berg via Users list for the GNU Emacs text editor]

> Let's do it like this, if we can get the algorithm to work
> on an arbitrary list, then all data-type issues can be
> solved easily, by adding two functions, one disassembly
> function that splits the data into a list, and one assembly
> function that puts it back together, after the algorithm has
> randomized it!
>
> So for example if the data is the string "data":
>
> 1. "data" -> '("d" "a" "t" "a")
> 2. run the algorithm, get e.g '("a" "d" "t" "a")
> 3. '("a" "d" "t" "a") -> "adta"
>
> Because Lisp is already based on the universal data
> structure which can express all human knowledge, isn't it
> possible that there is already a Lisp list randomizer that
> fulfills even Yuri's most uncompromising demands?

What about this?

;; https://stackoverflow.com/a/49505968

(require 'cl-lib)

(defun nshuff (sq)
  (cl-loop
     for i from (length sq) downto 2
     do (cl-rotatef (elt sq (cl-random i))
                    (elt sq (1- i)) ))
  sq)
  
;; (nshuff '(a b c)) ; (b c a) (a b c) (b c a) ...

Shufflepuck Café!

-- 
underground experts united
https://dataswamp.org/~incal

same sound random sort everywhere (was: Re: avoid narrow-to-region (was: Re: replace-regexp))

[Emanuel Berg via Users list for the GNU Emacs text editor]

Here we see it implemented for strings. If the nshuff
algorithm is sound data type should - you guessed it - not
influence anything in either way to make it more or less
sound. See comments.

;;; -*- lexical-binding: t -*-
;;;
;;; this file:
;;;   http://user.it.uu.se/~embe8573/emacs-init/random-generic.el
;;;   https://dataswamp.org/~incal/emacs-init/random-generic.el

(require 'cl-lib)

;; add more datatypes here... (for each type, one disassembler,
;; one call to nshuff, and one assembler is needed)

(defun string-random (s)
  (let*((str-lst (string-to-list s))
        (lst     (nshuff str-lst))
        (str-shuff (mapconcat (lambda (c) (char-to-string c)) lst "")) )
    str-shuff) )
;; (string-random "Elisp is my favorite E")

;; ... all use the same random-sort algorithm

(defun nshuff (sq)
  (cl-loop for i from (length sq) downto 2
           do (cl-rotatef
               (elt sq (cl-random i))
               (elt sq (1- i)) ))
  sq)
;; (nshuff '(a b c)) ; (b c a) (a b c) (b c a) ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: same sound random sort everywhere (was: Re: avoid narrow-to-region (was: Re: replace-regexp))

[Jean Louis]

* Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> [2021-05-21 20:37]:
> Here we see it implemented for strings. If the nshuff
> algorithm is sound data type should - you guessed it - not
> influence anything in either way to make it more or less
> sound. See comments.
> 
> ;;; -*- lexical-binding: t -*-
> ;;;
> ;;; this file:
> ;;;   http://user.it.uu.se/~embe8573/emacs-init/random-generic.el
Not Found

The requested URL /~embe8573/emacs-init/random-generic.el was not found on this server.

> ;;;   https://dataswamp.org/~incal/emacs-init/random-generic.el
Found

> (require 'cl-lib)

For 1 single loop you download 26 kilobytes... ☻

> ;; add more datatypes here... (for each type, one disassembler,
> ;; one call to nshuff, and one assembler is needed)

That above I can understand somehow, but not this below, do you wish
to expand it?

(defun string-random (s)
  (let*((str-lst (string-to-list s))
        (lst     (nshuff str-lst))
        (str-shuff (mapconcat (lambda (c) (char-to-string c)) lst "")) )
    str-shuff) )
;; (string-random "Elisp is my favorite E") ⇒ " ismitrEyaE lf oivpe s"

(defun nshuff (sq)
  (cl-loop for i from (length sq) downto 2
           do (cl-rotatef
               (elt sq (cl-random i))
               (elt sq (1- i)) ))
  sq)
;; (nshuff '(a b c)) ; (b c a) (a b c) (b c a) ...

Great, now you can make Black Jack win/loose simulator.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/

Re: same sound random sort everywhere (was: Re: avoid narrow-to-region (was: Re: replace-regexp))

[Emanuel Berg via Users list for the GNU Emacs text editor]

Jean Louis wrote:

>> Here we see it implemented for strings. If the nshuff
>> algorithm is sound data type should - you guessed it - not
>> influence anything in either way to make it more or less
>> sound. See comments.
>> 
>> ;;; -*- lexical-binding: t -*-
>> ;;;
>> ;;; this file:
>> ;;;   http://user.it.uu.se/~embe8573/emacs-init/random-generic.el
> Not Found

?

> For 1 single loop you download 26 kilobytes... ☻

???

I use cl-loop 26 times and FYI not just loops are in it...

>> ;; add more datatypes here... (for each type, one disassembler,
>> ;; one call to nshuff, and one assembler is needed)
>
> That above I can understand somehow, but not this below, do
> you wish to expand it?

Use the source James :)

> (defun string-random (s)
>   (let*((str-lst (string-to-list s))
>         (lst     (nshuff str-lst))
>         (str-shuff (mapconcat (lambda (c) (char-to-string c)) lst "")) )
>     str-shuff) )
> ;; (string-random "Elisp is my favorite E") ⇒ " ismitrEyaE lf oivpe s"
>
> (defun nshuff (sq)
>   (cl-loop for i from (length sq) downto 2
>            do (cl-rotatef
>                (elt sq (cl-random i))
>                (elt sq (1- i)) ))
>   sq)
> ;; (nshuff '(a b c)) ; (b c a) (a b c) (b c a) ...
>
> Great, now you can make Black Jack win/loose simulator.

check out upcoming virtual-vegas.el release

-- 
underground experts united
https://dataswamp.org/~incal