TIL about string-rectangle

TIL about string-rectangle

[Skip Montanaro]

I needed to replace the first comma on each line of a CSV file so I
could more conveniently sort by the first field (date, arranged as
MM/DD/YYYY). After a bit of fussing around, I got it squared away with
query-replace-regexp, did my sort, then needed to revert the
replacement text back to comma. Unfortunately, there were a few other
instances of the replacement text elsewhere in the file which caused
problems. (If I'd had my thinking cap on I would have checked before
doing the original replace, but it seems I was wearing my dunce
cap...) Poking around for help from Emacs against a future need, I
discovered replace-rectangle, which turns out to be an alias for
string-rectangle (bound to C-x r t). It looks to be quite handy when
you need to replace text within a rectangle. It's sort of (but not
quite) like narrow-to-region + query-replace(-regexp)?. It doesn't
query for a string to replace. It simply replaces the rectangle's
contents with the given string on each line.

Kinda cool,

Skip

Re: TIL about string-rectangle

[Tim Landscheidt]

Skip Montanaro <skip.montanaro@gmail.com> wrote:

> I needed to replace the first comma on each line of a CSV file so I
> could more conveniently sort by the first field (date, arranged as
> MM/DD/YYYY). After a bit of fussing around, I got it squared away with
> query-replace-regexp, did my sort, then needed to revert the
> replacement text back to comma. Unfortunately, there were a few other
> instances of the replacement text elsewhere in the file which caused
> problems. (If I'd had my thinking cap on I would have checked before
> doing the original replace, but it seems I was wearing my dunce
> cap...) Poking around for help from Emacs against a future need, I
> discovered replace-rectangle, which turns out to be an alias for
> string-rectangle (bound to C-x r t). It looks to be quite handy when
> you need to replace text within a rectangle. It's sort of (but not
> quite) like narrow-to-region + query-replace(-regexp)?. It doesn't
> query for a string to replace. It simply replaces the rectangle's
> contents with the given string on each line.

For complex sorts, I once wrote a function that enhances
sort-regexp-fields (and needs a docstring and comments that
I could understand after eight years):

| (defun tl-sort-regexp-fields (reverse record-regexp key-regexp beg end)
|   "Extended sort function."
|   (interactive "P\nsRegexp specifying records to sort: \nsRegexp specifying key within record: \nr")
|   (if (string-match "\\`\\(?:-\\\\[1-9]\\|\\(?:-?\\\\[1-9]\\)\\{2,\\}\\)\\'" key-regexp)
|       (let
|           ((i (length key-regexp)))
|         (while (> i 0)
|           (let ((key-reverse (and (> i 2) (= (aref key-regexp (- i 3)) ?-)))
|                 (key (substring key-regexp (- i 2) i)))
|             (sort-regexp-fields (if reverse (not key-reverse) key-reverse) record-regexp key beg end)
|             (if key-reverse
|                 (setq i (- i 1)))
|             (setq i (- i 2)))))
|     (sort-regexp-fields reverse record-regexp key-regexp beg end)))

Basically, you can select a region of a CSV file, then enter
(untested)
"^,\([0-9][0-9]\)/\([0-9][0-9]\)/\([0-9][0-9][0-9][0-9]\),.*$"
at the first prompt and "\3\1\2" at the (incorrectly named)
second, and then the lines would be sorted by year, month,
day.  You can reverse the order for individual keys by pre-
pending a "-".  It takes advantage of the (not guaranteed?)
fact that Emacs's sort functions are stable, i. e. the order
of lines with the same key is not changed.

If someone wants to clean that up and include it either as a
separate function in Emacs or semi-backwards-compatibly in
sort-regexp-fields, much appreciated.

Tim

Re: TIL about string-rectangle

[Skip Montanaro]

> For complex sorts, I once wrote a function that enhances
> sort-regexp-fields (and needs a docstring and comments that
> I could understand after eight years): ...

Thanks. I've used the Unix sort(1) command for a few decades now. It's
kind of burned into my brain. I know Emacs Lisp is exceedingly
powerful (I used it as my Lisp interpreter as the one and only Lisp
class I took), but sort(1) does just what I wanted here other than not
terminating the third (year) field at the comma. It's kind of hard to
argue with its economy. Regular expressions are quite handy, but also
pretty verbose for this task.

Skip

Re: TIL about string-rectangle

[Yuri Khan]

On Thu, 31 Dec 2020 at 05:13, Skip Montanaro <skip.montanaro@gmail.com> wrote:

> > For complex sorts, I once wrote a function that enhances
> > sort-regexp-fields (and needs a docstring and comments that
> > I could understand after eight years): ...
>
> Thanks. I've used the Unix sort(1) command for a few decades now. It's
> kind of burned into my brain. I know Emacs Lisp is exceedingly
> powerful (I used it as my Lisp interpreter as the one and only Lisp
> class I took), but sort(1) does just what I wanted here other than not
> terminating the third (year) field at the comma. It's kind of hard to
> argue with its economy. Regular expressions are quite handy, but also
> pretty verbose for this task.

You will have an easier time sorting by date if you keep dates in the
ISO 8601 format: YYYY-MM-DD.

Re: TIL about string-rectangle

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

> You will have an easier time sorting by date if you keep
> dates in the ISO 8601 format: YYYY-MM-DD.

It is also an intuitive format, the year is complete so no Y2K
bug or need to build an underground shelter to survive it
(bummer), and it goes from big to small which is the natural
order of things...

"God intended dates to look like..." ISO 8601

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: TIL about string-rectangle

[Skip Montanaro]

>
> You will have an easier time sorting by date if you keep dates in the
> ISO 8601 format: YYYY-MM-DD.
>

Oh, no question. * My source does it the wrong way (a big financial
institution who should know better), and the source updates almost daily,
so I felt it easier to just leave the dates as-is. Perhaps I should add an
extra transformation to the shell pipeline which preprocesses the new
inputs.

Skip

* I use my wife's birthday as a numeric PIN in one situation where we both
need to know the number. I tried to explain to her that it is her birthday
in sorting order. (She wouldn't know "ISO-8601" from a hole in the ground.)
She didn't get it. :-)

>

Re: TIL about string-rectangle

[Tim Landscheidt]

Skip Montanaro <skip.montanaro@gmail.com> wrote:

>> For complex sorts, I once wrote a function that enhances
>> sort-regexp-fields (and needs a docstring and comments that
>> I could understand after eight years): ...

> Thanks. I've used the Unix sort(1) command for a few decades now. It's
> kind of burned into my brain. I know Emacs Lisp is exceedingly
> powerful (I used it as my Lisp interpreter as the one and only Lisp
> class I took), but sort(1) does just what I wanted here other than not
> terminating the third (year) field at the comma. It's kind of hard to
> argue with its economy. Regular expressions are quite handy, but also
> pretty verbose for this task.

No doubt.  For me, it's the other way round: Whenever
"sort", "sort -n" or "sort -h" is not sufficient, I (usual-
ly) resort to Emacs or Perl.  I have looked at sort's -k
option a thousand times and never used it successfully.

Tim

RE: TIL about string-rectangle

[Drew Adams]

> Poking around for help from Emacs against a future need, I
> discovered replace-rectangle, which turns out to be an alias for
> string-rectangle (bound to C-x r t). It looks to be quite handy when
> you need to replace text within a rectangle. It's sort of (but not
> quite) like narrow-to-region + query-replace(-regexp)?. It doesn't
> query for a string to replace. It simply replaces the rectangle's
> contents with the given string on each line.

You can also use the usual replacement commands
in a rectangular region.  `C-x SPC' to establish
the region, then `C-M-%' or whatever, to search
and replace only with that rectangular region.
___

And if you use library `isearch+.el' then you can
constrain Isearch to within just the region,
including a rectangular region.

https://www.emacswiki.org/emacs/IsearchPlus#isearchp-restrict-to-region-flag


https://www.emacswiki.org/emacs/download/isearch%2b.el

Re: TIL about string-rectangle

[Skip Montanaro]

> You can also use the usual replacement commands
> in a rectangular region.  `C-x SPC' to establish
> the region, then `C-M-%' or whatever, to search
> and replace only with that rectangular region.

Hmmm... That doesn't work for me. I've used rectangles a lot over the
years and have never seen query-replace and friends be constrained by
the currently defined rectangle. I must be missing something.

Skip

RE: TIL about string-rectangle

[Drew Adams]

> > You can also use the usual replacement commands
> > in a rectangular region.  `C-x SPC' to establish
> > the region, then `C-M-%' or whatever, to search
> > and replace only with that rectangular region.
> 
> Hmmm... That doesn't work for me. I've used rectangles a lot over the
> years and have never seen query-replace and friends be constrained by
> the currently defined rectangle.

Are you sure?  After you activate the (rectangular)
region, you may need to use `C-x C-x', so point is
before mark, because query-replacing moves forward
from point.

Works for me, with `emacs -Q' (no init file).
___

(But as I said, to limit Isearch (not query-replace)
to the region you'll need library isearch+.el.)

RE: TIL about string-rectangle

[Drew Adams]

> Are you sure?  After you activate the (rectangular)
> region, you may need to use `C-x C-x', so point is
> before mark, because query-replacing moves forward
> from point.
> 
> Works for me, with `emacs -Q' (no init file).

Actually, it works without `C-x C-x', even if point
is after mark.

You need Emacs 25 or later for it to work.

Re: TIL about string-rectangle

[Skip Montanaro]

> > Hmmm... That doesn't work for me. I've used rectangles a lot over the
> > years and have never seen query-replace and friends be constrained by
> > the currently defined rectangle.
>
> Are you sure?  After you activate the (rectangular)
> region, you may need to use `C-x C-x', so point is
> before mark, because query-replacing moves forward
> from point.

I used query-replace which highlights what it will replace. When I saw
it highlight strings outside the region, I thought, "Hmmm... That
doesn't look right." I accepted one substitution, and it moved onto
the next. It moved onto the next instance, which was outside the
rectangle.

> Works for me, with `emacs -Q' (no init file).

Just tried with -Q. Same result. It highlighted and wanted to replace
strings outside the defined rectangle. Not sure what's going on.

Skip

Re: TIL about string-rectangle

[Sivaram Neelakantan]

On Wed, Dec 30 2020,Skip Montanaro wrote:


[snipped 17 lines]

> Just tried with -Q. Same result. It highlighted and wanted to replace
> strings outside the defined rectangle. Not sure what's going on.
>

worked for me on Windows Emacs 27.1.  And it seems to be the default
as I didn't try it with Emacs -Q.

sivaram
--