How to do a massive unfill paragraph operation over several hundred files?

How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Hi

emacs (latest git checkout, on linux) and org-mode are my main environment for doing (natural language) text related work.

Recently I changed the way text is processed and stored:

I no longer use hard coded line breaks introduced automatically at fill-column with auto-fill-mode enabled;

instead I now use visual-line-mode together with visual-fill-column mode to break lines only VISUALLY at fill-column: there are no real hard-coded line breaks in the file, each paragraph is just one long line.

I would like to harmonize all my existing Org mode and LaTeX text files in this way, i.e.: "unfill" every paragraph within them, -- while ideally preserving Org mode constructs like lists, BEGIN/END blocks etc.

MY QUESTION: What would be a good way to AUTOMATE such an unfill operation, removing all line breaks from all paragraphs over a large number of files?


Thanks

Gerald.

-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[ken]

On 09/28/2018 04:16 AM, Gerald Wildgruber wrote:
> Hi
>
> emacs (latest git checkout, on linux) and org-mode are my main environment for doing (natural language) text related work.
>
> Recently I changed the way text is processed and stored:
>
> I no longer use hard coded line breaks introduced automatically at fill-column with auto-fill-mode enabled;
>
> instead I now use visual-line-mode together with visual-fill-column mode to break lines only VISUALLY at fill-column: there are no real hard-coded line breaks in the file, each paragraph is just one long line.
>
> I would like to harmonize all my existing Org mode and LaTeX text files in this way, i.e.: "unfill" every paragraph within them, -- while ideally preserving Org mode constructs like lists, BEGIN/END blocks etc.
>
> MY QUESTION: What would be a good way to AUTOMATE such an unfill operation, removing all line breaks from all paragraphs over a large number of files?
>
>
> Thanks
>
> Gerald.

Gerald,

It's been decades since I messed with it, but as I vaguely recall
there's a batch mode for emacs.  Basically, you call emacs from the
command line with the "--batch" and other options ("--script
[file_containing_elisp]").

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> MY QUESTION: What would be a good way to
> AUTOMATE such an unfill operation, removing all
> line breaks from all paragraphs over a large
> number of files?

Script to insert a string into files
1, 2, and 3:

    #! /bin/zsh

    local script=do-this.el

    emacs -Q --batch --script $script 1 2 3

Elisp to do it:

    ;; do-this.el

    (dolist (f argv)
      (find-file f)
      (insert "Sail Ho!") ; do your thing here
      (save-buffer) )

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

ken wrote:

> It's been decades since I messed with it, but
> as I vaguely recall there's a batch mode for
> emacs.  Basically, you call emacs from the
> command line with the "--batch" and other
> options ("--script [file_containing_elisp]").

See? I was right even decades ago!

:)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Van L]

>> It's been decades since I messed with it, but
>> as I vaguely recall there's a batch mode for
>> emacs.  Basically, you call emacs from the
>> command line with the "--batch" and other
>> options ("--script [file_containing_elisp]").
> 
> See? I was right even decades ago!
> 
> :)

You had this formulation:

  ┌────
  │ 1  #! /bin/zsh
  │ 2  
  │ 3  # Script to insert a string into files
  │ 4  # 1, 2, and 3:
  │ 5  
  │ 6  local script=do-this.el
  │ 7  
  │ 8  emacs -Q --batch --script $script 1 2 3
  └────

  ┌────
  │ 1  ;; do-this.el
  │ 2  
  │ 3  ;; Elisp to do it:
  │ 4  
  │ 5  (dolist (f argv)
  │ 6    (find-file f)
  │ 7    (insert "Sail away, sail away, sail away…") ; do your thing here
  │ 8    (save-buffer) )
  └────

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Hi,

thanks for the hint!

What I do now is the following: in every file I want to normalize (= unwrap) I use a kbd macro consisting of two consecutive functions:

1. an unfill function (originally posted by Stefan Monnier if I remember correctly):

(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

2. org-forward-paragraph

I then repeat the macro with an infinite argument.

This works quite well and the result requires very little manual correction; all Org mode constructs like lists, blocks etc are preserved.

But these are of course interactive functions; how would I script them to non-interactive code in the way you proposed for emacs batch processing, i.e. not expecting any input except a number files to operate on? Can I load org-mode on the fly in order for emacs to understand the second function (org-forward-paragraph)?

Thanks

Gerald.



On Sa, Sep 29 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> MY QUESTION: What would be a good way to
>> AUTOMATE such an unfill operation, removing all
>> line breaks from all paragraphs over a large
>> number of files?
>
> Script to insert a string into files
> 1, 2, and 3:
>
>     #! /bin/zsh
>
>     local script=do-this.el
>
>     emacs -Q --batch --script $script 1 2 3
>
> Elisp to do it:
>
>     ;; do-this.el
>
>     (dolist (f argv)
>       (find-file f)
>       (insert "Sail Ho!") ; do your thing here
>       (save-buffer) )

---------------------
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> But these are of course interactive
> functions; how would I script them to
> non-interactive code in the way you proposed
> for emacs batch processing, i.e.
> not expecting any input except a number files
> to operate on? Can I load org-mode on the fly
> in order for emacs to understand the second
> function (org-forward-paragraph)?

I don't know if org-mode loads automatically
with Emacs, if it does, it should be enough to
remove the -Q option from the Emacs invocation
command, to make the org-mode functions
available. If it doesn't load automatically,
removing the -Q option should still make it
work since this will include your regular init.
If it still doesn't work, load it manually in
the Emacs script.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

I tried that: loading org works without problems.

But how can I use the unfill-paragraph function programmatically (non-interactively) in a script, so that every paragraph of the file is processed by it (as is done through repeated execution of a kbd macro)?

On So, Sep 30 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> But these are of course interactive
>> functions; how would I script them to
>> non-interactive code in the way you proposed
>> for emacs batch processing, i.e.
>> not expecting any input except a number files
>> to operate on? Can I load org-mode on the fly
>> in order for emacs to understand the second
>> function (org-forward-paragraph)?
>
> I don't know if org-mode loads automatically
> with Emacs, if it does, it should be enough to
> remove the -Q option from the Emacs invocation
> command, to make the org-mode functions
> available. If it doesn't load automatically,
> removing the -Q option should still make it
> work since this will include your regular init.
> If it still doesn't work, load it manually in
> the Emacs script.


-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> But how can I use the unfill-paragraph
> function programmatically (non-interactively)
> in a script

(unfill-paragraph ARGS)

C-h f unfill-paragraph RET

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> But how can I use the unfill-paragraph
>> function programmatically (non-interactively)
>> in a script
>
> (unfill-paragraph ARGS)

Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:

(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

I cannot see what to put here.

-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> Yes, that is the whole point I didn't
> understand! What does "ARGS" have to be when
> called non-interactively in a script.
> From the doc string of the function:
>
> (defun unfill-paragraph (&optional region)
>   "Takes a multi-line paragraph and makes it into a single line of text."
>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>   (let ((fill-column (point-max)))
>     (fill-paragraph nil region)))
>
> I cannot see what to put here.

When you call that interactively what happens
is `barf-if-buffer-read-only', which

    [s]ignal[s] a `buffer-read-only' error if
    the current buffer is read-only

but if that doesn't hold (i.e. the buffer isn't
read-only), then "region", the optional
argument, is assigned `t'.

However when you call it from Lisp, NONE of
this happens so "region" remains nil!

Observe:

(defun test-interactive-optional-args (&optional arg)
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (message "arg: %s" arg) )
  
(call-interactively
#'test-interactive-optional-args)     ; arg: t

 (test-interactive-optional-args)     ; arg: nil
 
 (test-interactive-optional-args t)   ; arg: t

and of course:

 (test-interactive-optional-args nil) ; arg: nil

So in so many words, why don't you pass `t' as
the argument? :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Robert Pluim]

Gerald Wildgruber <wildgruber@tu-berlin.de> writes:

> On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:
>
>> Gerald Wildgruber wrote:
>>
>>> But how can I use the unfill-paragraph
>>> function programmatically (non-interactively)
>>> in a script
>>
>> (unfill-paragraph ARGS)
>
> Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:
>
> (defun unfill-paragraph (&optional region)
>   "Takes a multi-line paragraph and makes it into a single line of text."
>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>   (let ((fill-column (point-max)))
>     (fill-paragraph nil region)))

Itʼs optional, which means it will be 'nil' if you donʼt specify it,
which means (from the docsting of 'fill-paragraph')

    The REGION argument is non-nil if called interactively; in that
    case, if Transient Mark mode is enabled and the mark is active,
    call `fill-region' to fill each of the paragraphs in the active
    region, instead of just filling the current paragraph.

So 'nil' just means 'fill the current paragraph'. It might work for
you if you mark the whole buffer, and call

(unfill-paragraph t)

instead, since that should take care of iterating over all the
paragraphs (untested).

Robert

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Thanks Emanuel and Robert for explanations:

I now have a script unfill.el containing these lines:

------------------------------------------------------------

(require 'org)

;; https://www.emacswiki.org/emacs/UnfillParagraph
(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

(dolist (f argv)
  (find-file f)
  (mark-whole-buffer)
  (unfill-paragraph t)
  (org-forward-paragraph)
  (save-buffer))

------------------------------------------------------------

I call this from a terminal like so:

emacs -Q --batch --script unfill.el FILE.org

Unfortunately, nothing happens; I get two "Mark set" messages in the terminal, but the file itself remains untouched. It isn't edited at all, same timestamp etc.

Sorry for the inconvenience, but I have the impression I'm getting something completely wrong here, but can't see what.

Do you?


On Mo, Okt 01 2018, Robert Pluim <rpluim@gmail.com> wrote:

> Gerald Wildgruber <wildgruber@tu-berlin.de> writes:
>
>> On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:
>>
>>> Gerald Wildgruber wrote:
>>>
>>>> But how can I use the unfill-paragraph
>>>> function programmatically (non-interactively)
>>>> in a script
>>>
>>> (unfill-paragraph ARGS)
>>
>> Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:
>>
>> (defun unfill-paragraph (&optional region)
>>   "Takes a multi-line paragraph and makes it into a single line of text."
>>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>>   (let ((fill-column (point-max)))
>>     (fill-paragraph nil region)))
>
> Itʼs optional, which means it will be 'nil' if you donʼt specify it,
> which means (from the docsting of 'fill-paragraph')
>
>     The REGION argument is non-nil if called interactively; in that
>     case, if Transient Mark mode is enabled and the mark is active,
>     call `fill-region' to fill each of the paragraphs in the active
>     region, instead of just filling the current paragraph.
>
> So 'nil' just means 'fill the current paragraph'. It might work for
> you if you mark the whole buffer, and call
>
> (unfill-paragraph t)
>
> instead, since that should take care of iterating over all the
> paragraphs (untested).
>
> Robert


-- 
Dr. Gerald Wildgruber
Institut für Philosophie, Literatur-, Wissenschafts- und Technikgeschichte
Literaturwissenschaft mit Schwerpunkt Literatur und Wissenschaft
Technische Universität Berlin
Straße des 17. Juni 135
D-10623 Berlin
http://www.philosophie.tu-berlin.de/menue/home/
T. +49 (0)30 314 25924
F. +49 (0)30 314 23107
wildgruber@tu-berlin.de
---------------------
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> (dolist (f argv) (find-file f)
> (mark-whole-buffer) (unfill-paragraph t)
> (org-forward-paragraph) (save-buffer))

This looks a bit wierd but I'm unsure because
I'm not as close to whatever it is you're
trying to do. What does `org-forward-paragraph'
do? Shouldn't you have a loop for that as well
if you you want to do this to several
paragraphs? But then, isn't `mark-whole-buffer'
something that runs contradictory to this?
Even so, something should happen what I can
see. Like the FIRST paragraph getting unfilled
at the very least.

Try to replace it with something simpler.
Like adding a single char to each file:

    (insert ?a)

If this works, your problem is with the
`dolist' stuff and you just need the details
figured out.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Noam Postavsky]

On Tue, 2 Oct 2018 at 08:11, Gerald Wildgruber <wildgruber@tu-berlin.de> wrote:

> (dolist (f argv)
>   (find-file f)
>   (mark-whole-buffer)
>   (unfill-paragraph t)
>   (org-forward-paragraph)
>   (save-buffer))
>
> ------------------------------------------------------------
>
> I call this from a terminal like so:
>
> emacs -Q --batch --script unfill.el FILE.org
>
> Unfortunately, nothing happens; I get two "Mark set" messages in the terminal, but the file itself remains untouched. It isn't edited at all, same timestamp etc.

I think the problem is that find-file only makes sense for interactive
functions, because it switches the displayed buffer (which takes
effect after the current command finishes), but doesn't actually set
the current buffer (i.e., no immediate change).

So you should have

    (with-current-buffer (find-file-noselect f)
      (mark-whole-buffer)
      ...)

By the way, I don't see the benefit of invoking Emacs in batch mode
here, especially since this sounds like a one-off transformation. Just
put the expression which does the work into *scratch* and evaluate.
Something like this:

    (let ((fill-column most-positive-fixnum))
      (dolist (f (directory-files-recursively
                  "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
        (with-current-buffer (find-file-noselect f)
          (fill-region (point-min) (point-max)) ; Unfill, per `fill-column'
          (save-buffer))))

(of course replace the (directory-files-recursively...) thing with
(list "test-file.org") first to make sure it works right.)

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Thanks.

On Di, Okt 02 2018, Noam Postavsky <npostavs@gmail.com> wrote:

> I think the problem is that find-file only makes sense for interactive
> functions, because it switches the displayed buffer (which takes
> effect after the current command finishes), but doesn't actually set
> the current buffer (i.e., no immediate change).
>
> So you should have
>
>     (with-current-buffer (find-file-noselect f)
>       (mark-whole-buffer)
>       ...)

The script then looks like this:

(dolist (f argv)
  (with-current-buffer (find-file-noselect f)
  (mark-whole-buffer)
  (unfill-paragraph t)
  (org-forward-paragraph)
  (save-buffer)))

It turns out that this works just on the FIRST paragraph of the file! Only this one is edited. The script does not move to the next paragraph.

So, what works well in a kbd macro (moving point forward to next paragraph via org-forward-paragraph) does not in this script.

Cf. docstring of this function: "Move forward to beginning of next paragraph or equivalent. The function moves point to the beginning of the next visible structural element, which can be a paragraph, a table, a list item, moving point to beginning".

> put the expression which does the work into *scratch* and evaluate.
> Something like this:
>
>     (let ((fill-column most-positive-fixnum))
>       (dolist (f (directory-files-recursively
>                   "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
>         (with-current-buffer (find-file-noselect f)
>           (fill-region (point-min) (point-max)) ; Unfill, per `fill-column'
>           (save-buffer))))

Yes, that almost did it!

But: for some reason the "fill-region" function you use does not work on list constructs like:

1. text text text text text text text text text
2. text text text text text text text text text text text
3. text text text text text text text text text text

The text in these constructs doesn't get unfilled, -- whereas it is correctly unfilled by using the "unfill-paragraph" function (which ultimately calls "fill-paragraph").

I tried to rewrite your code to use "fill-paragraph" instead of "fill-region" but didn't succeed.

Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.

--
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> It turns out that this works just on the
> FIRST paragraph of the file! Only this one is
> edited. The script does not move to the
> next paragraph.

Yeah, told you so :)

For this to work, one way would be to put
a loop inside the loop, where the outer loop
will iterate the files and the inner loop will
iterate the paragraphs.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Noam Postavsky]

On Wed, 3 Oct 2018 at 06:11, Gerald Wildgruber <wildgruber@tu-berlin.de> wrote:

> (dolist (f argv)
>   (with-current-buffer (find-file-noselect f)
>   (mark-whole-buffer)
>   (unfill-paragraph t)
>   (org-forward-paragraph)
>   (save-buffer)))
>
> It turns out that this works just on the FIRST paragraph of the file! Only this one is edited. The script does not move to the next paragraph.

Right, I guess the mark-whole-buffer thing doesn't have the intended
effect because it relies on transient-mark-mode, which is probably
tied up with the command loop.

> So, what works well in a kbd macro (moving point forward to next paragraph via org-forward-paragraph) does not in this script.

It makes sense that the paragraph movement doesn't work, because you
only call it once per file, so you unfill the 1st paragraph, move to
the 2nd, and then leave.

> Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.

Perhaps this?

(let ((fill-column most-positive-fixnum))
  (dolist (f (directory-files-recursively
              "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
    (with-current-buffer (find-file-noselect f)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

On Do, Okt 04 2018, Noam Postavsky <npostavs@gmail.com> wrote:

>> Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.
>
> Perhaps this?
>
> (let ((fill-column most-positive-fixnum))
>   (dolist (f (directory-files-recursively
>               "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
>     (with-current-buffer (find-file-noselect f)
>       (while (not (eobp))
>         (fill-paragraph)
>         (org-forward-paragraph))
>       (save-buffer))))

Thanks Noam, this almost did it! but I couldn't see through a last problem, it sometimes worked, sometimes didn't. Here's the solution someone from the emacs Org mode mailing list pointed me to:

http://lists.gnu.org/archive/html/emacs-orgmode/2018-10/msg00070.html

I had to UNFOLD the entries before working on them, even programmatically, via "(org-show-all)".

I do this now from a lisp buffer and both functions work exactly as expected:

single file:
============

(let ((fill-column most-positive-fixnum))
  (dolist (f (list "~/lorem.org"))
    (with-current-buffer (find-file-noselect f)
      (org-show-all)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))


recursively through dir tree:
=============================

(let ((fill-column most-positive-fixnum))
  (dolist (f (directory-files-recursively
              "/dirs/with/org/files/" (rx (or ".org" ".outl") eos)))
    (with-current-buffer (find-file-noselect f)
      (org-show-all)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))


That's great; I'm glad it works now from within emacs itself (and not via sed, awk or tr as I tried before), harnessing all the knowledge the editor has of its own constructs, especially with some of the more complicated list and enumeration structures, -- all of them are correctly unfilled. This is solved now.

Thanks again to everyone who contributed!

Gerald.


-- 
Sent with mu4e