Re: How to do a massive unfill paragraph operation over several hundred files?

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> MY QUESTION: What would be a good way to
> AUTOMATE such an unfill operation, removing all
> line breaks from all paragraphs over a large
> number of files?

Script to insert a string into files
1, 2, and 3:

    #! /bin/zsh

    local script=do-this.el

    emacs -Q --batch --script $script 1 2 3

Elisp to do it:

    ;; do-this.el

    (dolist (f argv)
      (find-file f)
      (insert "Sail Ho!") ; do your thing here
      (save-buffer) )

-- 
underground experts united
http://user.it.uu.se/~embe8573

history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[Emanuel Berg]

> (dolist (f argv)
>   (find-file f)
>   (insert "Sail Ho!") ; do your thing here
>   (save-buffer) )

BTW I always thought, without thinking about
it, that "argv" is a convention in and from the
C programming language!

But Lisp is older than C (1958 to 1972; Elisp
in particular tho a 1985 youngster) so perhaps
it is the other way around?

Or argv may trace its origin to yet some other
language or piece of technology!

What do you say?

PS. Here, technically it is an alias for
    `command-line-args-left' which is
    a variable in startup.el. Ain't it
    cool stuff?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[Skip Montanaro]

>
> BTW I always thought, without thinking about
> it, that "argv" is a convention in and from the
> C programming language!
>
> But Lisp is older than C (1958 to 1972; Elisp
> in particular tho a 1985 youngster) so perhaps it is the other way around?
>

Though Lisp is older than C, it hasn't been static that entire time. It's
not surprising to me that particular implementations would adapt to the
ecosystems in which they are dropped (especially, as with Unix, that
becomes the largest installed base for a couple decades), then pick those
adaptations up and take them to other environments. I suspect that if
Multics was still the main Emacs ecosystem, we'd see Multics-like names at
the boundaries. In fact, they might be there and I just don't recognize
them. <wink>

Skip

>

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[John Yates]

On Sat, Sep 29, 2018 at 8:45 AM Skip Montanaro <skip.montanaro@gmail.com> wrote:
>
> I suspect that if
> Multics was still the main Emacs ecosystem, we'd see Multics-like names at
> the boundaries. In fact, they might be there and I just don't recognize
> them. <wink>

I never used Multics but I was part of Apollo Computer from its earliest days.

Apollo was born in Prime Computer's R&D group before the Unix tsunami hit the
computer industry.  Prime included a large contingent of Multics alumni. The
main emotional impetus of that R&D group was to preserve as much of the Multics
computing architecture as possible on more limited hardware. By contrast,
Thompson and Ritchie, Multics' best known alumni, were more intent on recreating
the experience of community that a shared computing environment fostered.  The
hardware each effort targeted greatly influence its designs.

Multics' original hardware - the GE-645, a mainframe with pricing to match -
provided a 36-bit virtual address space composed of 2^18 segments, each
containing up to 2^18 36-bit words. These segments formed a "Single Level
Store": all segments were addressable though not necessarily accessible.
ACL-based protection was segment-granular. Segments were the user visible
elements of persistent storage. Hierarchical directories mapped names to segment
numbers (think DNS to IP address or Unix path to inode number).  The output of
the compilation tool chain was what we would consider a shared library: an image
exporting various symbols and requiring additional symbols to be provided by the
environment into which it got loaded.  Upon connection a user was provide a
process running the command shell.  To invoke a command the shell translated the
name to a segment ID and asked to have it "made known" (i.e. import its exported
symbols into the CLS - Combined Linkage Segment).  Linking was maximally
dynamic.  References to heretofore unbound symbols (both instruction and data)
were resolved "on the fly" using the CLS. So for instance, for the shell to list
the contents of the current directory the shell would locate the list segment,
ask to make it known and then call its entrypoint.  The clear implication here
was that a single process' stack contained both stack frames for the invoking
shell and stack frames for the list_directory image.  The corollary is that
there was no forking, no cloning of address spaces.

Thompson and Ritchie, started earlier and targeted the obsolescing PDP-7.  The
PDP-7 had an 18-bit architecture nearly unchanged from DEC's original PDP-1.  It
had an ability to address at most 16 banks of 4K 18-bit words.  Each bank was
effectively a separate segment.  The most convenient program image occupied no
more than a single bank (i.e. 4K words).  Having two or more images present at
the same time in a process' 4K address space was nearly unthinkable.  This, at
least in part, explains why, unlike Multics, Unix emphasizes creating light
weight processes.

Nearly a decade later the Prime folk targeted Motorola's forthcoming 68000.
This microprocessor had a 24-bit byte-granular address space.  (The original
68000 did not support virtual memory.  Still it was cheap enough that Prime was
able to cobble together a virtual memory scheme via an off chip MMU and a second
68000 to handle page faults.  When the MMU detected a page miss it simply halted
the clock to the main processor.  The second chip serviced the page fault and
then re-enabled the main processors clock.)  This 16MB virtual address space was
large enough to implement Multics' single level store architecture.  Because the
68000 supported neither segment addressing nor segment faults those had to be
finessed by explicitly mapping files into the address space.  There were not
stream I/O operations.  That file mapping was the only means of performing I/O
testifies to the system being a true SLS.  The large address space also allowed
in-process image activation.  Full on-the-fly dynamic linking was not possible.
Trampolines might have supported calls to unbound external entrypoints but there
was no obvious way to handle unbound data reference.  Therefore adding an image
into a process resolved all possible references in either direction.

As Skip intuited, in spite of the radically different systems that emerged the
common Multics heritage is visible in the culture of abbreviated command names
and in some of the names themselves (e.g. ls, pwd, etc):

https://multicians.org/multics-commands.html

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[Emanuel Berg]

John Yates wrote:

> I never used Multics but I was part of Apollo
> Computer from its earliest days.
>
> Apollo was born in Prime Computer's R&D group
> before the Unix tsunami hit the computer
> industry. Prime included a large contingent
> of Multics alumni. The main emotional impetus
> of that R&D group was to preserve as much of
> the Multics computing architecture as
> possible on more limited hardware.
> By contrast, Thompson and Ritchie, Multics'
> best known alumni, were more intent on
> recreating the experience of community that
> a shared computing environment fostered.
> The hardware each effort targeted greatly
> influence its designs.
>
> Multics' original hardware - the GE-645,
> a mainframe with pricing to match - provided
> a 36-bit virtual address space composed of
> 2^18 segments, each containing up to 2^18
> 36-bit words. These segments formed a "Single
> Level Store": all segments were addressable
> though not necessarily accessible. ACL-based
> protection was segment-granular.
> Segments were the user visible elements of
> persistent storage. Hierarchical directories
> mapped names to segment numbers (think DNS to
> IP address or Unix path to inode number).
> The output of the compilation tool chain was
> what we would consider a shared library: an
> image exporting various symbols and requiring
> additional symbols to be provided by the
> environment into which it got loaded.
> Upon connection a user was provide a process
> running the command shell. To invoke
> a command the shell translated the name to
> a segment ID and asked to have it "made
> known" (i.e. import its exported symbols into
> the CLS - Combined Linkage Segment).
> Linking was maximally dynamic. References to
> heretofore unbound symbols (both instruction
> and data) were resolved "on the fly" using
> the CLS. So for instance, for the shell to
> list the contents of the current directory
> the shell would locate the list segment, ask
> to make it known and then call its
> entrypoint. The clear implication here was
> that a single process' stack contained both
> stack frames for the invoking shell and stack
> frames for the list_directory image.
> The corollary is that there was no forking,
> no cloning of address spaces.
>
> Thompson and Ritchie, started earlier and
> targeted the obsolescing PDP-7. The PDP-7 had
> an 18-bit architecture nearly unchanged from
> DEC's original PDP-1. It had an ability to
> address at most 16 banks of 4K 18-bit words.
> Each bank was effectively a separate segment.
> The most convenient program image occupied no
> more than a single bank (i.e. 4K words).
> Having two or more images present at the same
> time in a process' 4K address space was
> nearly unthinkable. This, at least in part,
> explains why, unlike Multics, Unix emphasizes
> creating light weight processes.
>
> Nearly a decade later the Prime folk targeted
> Motorola's forthcoming 68000.
> This microprocessor had a 24-bit
> byte-granular address space. (The original
> 68000 did not support virtual memory.
> Still it was cheap enough that Prime was able
> to cobble together a virtual memory scheme
> via an off chip MMU and a second 68000 to
> handle page faults. When the MMU detected
> a page miss it simply halted the clock to the
> main processor. The second chip serviced the
> page fault and then re-enabled the main
> processors clock.) This 16MB virtual address
> space was large enough to implement Multics'
> single level store architecture. Because the
> 68000 supported neither segment addressing
> nor segment faults those had to be finessed
> by explicitly mapping files into the address
> space. There were not stream I/O operations.
> That file mapping was the only means of
> performing I/O testifies to the system being
> a true SLS. The large address space also
> allowed in-process image activation.
> Full on-the-fly dynamic linking was not
> possible. Trampolines might have supported
> calls to unbound external entrypoints but
> there was no obvious way to handle unbound
> data reference. Therefore adding an image
> into a process resolved all possible
> references in either direction.
>
> As Skip intuited, in spite of the radically
> different systems that emerged the common
> Multics heritage is visible in the culture of
> abbreviated command names and in some of the
> names themselves (e.g. ls, pwd, etc):
>
> https://multicians.org/multics-commands.html

I'm certainly glad I brought this up for you
all to share these stories!

    "Government forces claim they have found
    evidence of interesting pieces of
    computer history in J. Yates' person."

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[Emanuel Berg]

Skip Montanaro wrote:

>> BTW I always thought, without thinking about
>> it, that "argv" is a convention in and from
>> the C programming language! But Lisp is
>> older than C (1958 to 1972; Elisp in
>> particular tho a 1985 youngster) so perhaps
>> it is the other way around?
>
> Though Lisp is older than C, it hasn't been
> static that entire time. It's not surprising
> to me that particular implementations would
> adapt to the ecosystems in which they are
> dropped (especially, as with Unix, that
> becomes the largest installed base for
> a couple decades), then pick those
> adaptations up and take them to other
> environments. I suspect that if Multics was
> still the main Emacs ecosystem, we'd see
> Multics-like names at the boundaries.
> In fact, they might be there and I just don't
> recognize them. <wink>

Riiight... so it *is* from C?

Multics was first released in 1969 while work
started at MIT as early as 1965 [1].

The first Unix manual was published in 1971 and
it appeared outside of Bell in 1973.
Work on Unix started in 1969.

The word "Unix" should be a pun/joke on
"Multics", right? Because Multics was supposed
to be multi-user, only wasn't, while Unix,
despite its name, is...

[1] https://en.wikipedia.org/wiki/Multics

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[Barry Margolin]

In article <mailman.1481.1538225108.1284.help-gnu-emacs@gnu.org>,
 Skip Montanaro <skip.montanaro@gmail.com> wrote:

> >
> > BTW I always thought, without thinking about
> > it, that "argv" is a convention in and from the
> > C programming language!
> >
> > But Lisp is older than C (1958 to 1972; Elisp
> > in particular tho a 1985 youngster) so perhaps it is the other way around?
> >
> 
> Though Lisp is older than C, it hasn't been static that entire time. It's
> not surprising to me that particular implementations would adapt to the
> ecosystems in which they are dropped (especially, as with Unix, that
> becomes the largest installed base for a couple decades), then pick those
> adaptations up and take them to other environments. I suspect that if
> Multics was still the main Emacs ecosystem, we'd see Multics-like names at
> the boundaries. In fact, they might be there and I just don't recognize
> them. <wink>

GNU Emacs also wasn't the first Emacs written in Lisp. Multics Emacs 
might have been the first, but there were other Emacs-like editors 
implemented in Lisp at MIT, Stanford, and CMU around the same time, I'm 
not sure which was actually first. GNU Emacs came much later than all of 
these, and it wasn't even the first one for Unix systems -- that would 
have been either Gosling Emacs or Z-Emacs.

Multics Emacs had the save-excursion macro, I think the others picked it 
up from there.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: history of argv (was: Re: How to do a massive unfill paragraph operation over several hundred files?)

[James K. Lowden]

On Sat, 29 Sep 2018 14:04:36 +0200
Emanuel Berg <moasen@zoho.com> wrote:

> BTW I always thought, without thinking about
> it, that "argv" is a convention in and from the
> C programming language!
> ...
> Or argv may trace its origin to yet some other
> language or piece of technology!

It's a little difficult to disentangle C from Unix, but the argv
convention owes more to Unix than to C.  

ISO C distinguishes between freestanding and hosted environments.  
main()  is defined only in a hosted environment, and the names of the
arguments are not defined.  So, fair to say, argv is a C convention but
not defined by C.  

Unix on the other hand defines its exec(3) family of functions.  Any
program thus invoked is provided an array of pointer-to-char and the
length of that array.  The documentation for exec(3) typically refers
to the array as argv.  

It's hard to overstate the effect Unix had on other operating systems
and programmer expectations.  Would MS-DOS or even VMS support argv, but
for the influence of Unix?  Not C, certainly; C didn't dominate the
landscape in 1980 as it does now.  That was the era of ADA,
Modula-2, Turbo-Pascal, and "4th generation" languages.  And Cobol, of
course.  

C is the native API for Windows, but the Windows entry point
(conventionally, "WinMain") isn't provided with argv.  It's given the
command line verbatim; parse it as you will.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms633559
(v=vs.85).aspx

No landscape is more alien to C than IBM, and argv on IBM systems is
round peg in a square hole.  Cf. IMS and CICS at:

https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbclx01/comlin.htm.

Coming back on topic, there's no small intersection between the Unix and
elisp communities.  When the need arose to name the array of strings
passed to the exec'd program, it's little wonder it got named "argv".  

--jkl

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Hi,

thanks for the hint!

What I do now is the following: in every file I want to normalize (= unwrap) I use a kbd macro consisting of two consecutive functions:

1. an unfill function (originally posted by Stefan Monnier if I remember correctly):

(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

2. org-forward-paragraph

I then repeat the macro with an infinite argument.

This works quite well and the result requires very little manual correction; all Org mode constructs like lists, blocks etc are preserved.

But these are of course interactive functions; how would I script them to non-interactive code in the way you proposed for emacs batch processing, i.e. not expecting any input except a number files to operate on? Can I load org-mode on the fly in order for emacs to understand the second function (org-forward-paragraph)?

Thanks

Gerald.



On Sa, Sep 29 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> MY QUESTION: What would be a good way to
>> AUTOMATE such an unfill operation, removing all
>> line breaks from all paragraphs over a large
>> number of files?
>
> Script to insert a string into files
> 1, 2, and 3:
>
>     #! /bin/zsh
>
>     local script=do-this.el
>
>     emacs -Q --batch --script $script 1 2 3
>
> Elisp to do it:
>
>     ;; do-this.el
>
>     (dolist (f argv)
>       (find-file f)
>       (insert "Sail Ho!") ; do your thing here
>       (save-buffer) )

---------------------
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> But these are of course interactive
> functions; how would I script them to
> non-interactive code in the way you proposed
> for emacs batch processing, i.e.
> not expecting any input except a number files
> to operate on? Can I load org-mode on the fly
> in order for emacs to understand the second
> function (org-forward-paragraph)?

I don't know if org-mode loads automatically
with Emacs, if it does, it should be enough to
remove the -Q option from the Emacs invocation
command, to make the org-mode functions
available. If it doesn't load automatically,
removing the -Q option should still make it
work since this will include your regular init.
If it still doesn't work, load it manually in
the Emacs script.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

I tried that: loading org works without problems.

But how can I use the unfill-paragraph function programmatically (non-interactively) in a script, so that every paragraph of the file is processed by it (as is done through repeated execution of a kbd macro)?

On So, Sep 30 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> But these are of course interactive
>> functions; how would I script them to
>> non-interactive code in the way you proposed
>> for emacs batch processing, i.e.
>> not expecting any input except a number files
>> to operate on? Can I load org-mode on the fly
>> in order for emacs to understand the second
>> function (org-forward-paragraph)?
>
> I don't know if org-mode loads automatically
> with Emacs, if it does, it should be enough to
> remove the -Q option from the Emacs invocation
> command, to make the org-mode functions
> available. If it doesn't load automatically,
> removing the -Q option should still make it
> work since this will include your regular init.
> If it still doesn't work, load it manually in
> the Emacs script.


-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> But how can I use the unfill-paragraph
> function programmatically (non-interactively)
> in a script

(unfill-paragraph ARGS)

C-h f unfill-paragraph RET

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:

> Gerald Wildgruber wrote:
>
>> But how can I use the unfill-paragraph
>> function programmatically (non-interactively)
>> in a script
>
> (unfill-paragraph ARGS)

Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:

(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

I cannot see what to put here.

-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Robert Pluim]

Gerald Wildgruber <wildgruber@tu-berlin.de> writes:

> On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:
>
>> Gerald Wildgruber wrote:
>>
>>> But how can I use the unfill-paragraph
>>> function programmatically (non-interactively)
>>> in a script
>>
>> (unfill-paragraph ARGS)
>
> Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:
>
> (defun unfill-paragraph (&optional region)
>   "Takes a multi-line paragraph and makes it into a single line of text."
>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>   (let ((fill-column (point-max)))
>     (fill-paragraph nil region)))

Itʼs optional, which means it will be 'nil' if you donʼt specify it,
which means (from the docsting of 'fill-paragraph')

    The REGION argument is non-nil if called interactively; in that
    case, if Transient Mark mode is enabled and the mark is active,
    call `fill-region' to fill each of the paragraphs in the active
    region, instead of just filling the current paragraph.

So 'nil' just means 'fill the current paragraph'. It might work for
you if you mark the whole buffer, and call

(unfill-paragraph t)

instead, since that should take care of iterating over all the
paragraphs (untested).

Robert

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Thanks Emanuel and Robert for explanations:

I now have a script unfill.el containing these lines:

------------------------------------------------------------

(require 'org)

;; https://www.emacswiki.org/emacs/UnfillParagraph
(defun unfill-paragraph (&optional region)
  "Takes a multi-line paragraph and makes it into a single line of text."
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (let ((fill-column (point-max)))
    (fill-paragraph nil region)))

(dolist (f argv)
  (find-file f)
  (mark-whole-buffer)
  (unfill-paragraph t)
  (org-forward-paragraph)
  (save-buffer))

------------------------------------------------------------

I call this from a terminal like so:

emacs -Q --batch --script unfill.el FILE.org

Unfortunately, nothing happens; I get two "Mark set" messages in the terminal, but the file itself remains untouched. It isn't edited at all, same timestamp etc.

Sorry for the inconvenience, but I have the impression I'm getting something completely wrong here, but can't see what.

Do you?


On Mo, Okt 01 2018, Robert Pluim <rpluim@gmail.com> wrote:

> Gerald Wildgruber <wildgruber@tu-berlin.de> writes:
>
>> On Mo, Okt 01 2018, Emanuel Berg <moasen@zoho.com> wrote:
>>
>>> Gerald Wildgruber wrote:
>>>
>>>> But how can I use the unfill-paragraph
>>>> function programmatically (non-interactively)
>>>> in a script
>>>
>>> (unfill-paragraph ARGS)
>>
>> Yes, that is the whole point I didn't understand! What does "ARGS" have to be when called non-interactively in a script. From the doc string of the function:
>>
>> (defun unfill-paragraph (&optional region)
>>   "Takes a multi-line paragraph and makes it into a single line of text."
>>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>>   (let ((fill-column (point-max)))
>>     (fill-paragraph nil region)))
>
> Itʼs optional, which means it will be 'nil' if you donʼt specify it,
> which means (from the docsting of 'fill-paragraph')
>
>     The REGION argument is non-nil if called interactively; in that
>     case, if Transient Mark mode is enabled and the mark is active,
>     call `fill-region' to fill each of the paragraphs in the active
>     region, instead of just filling the current paragraph.
>
> So 'nil' just means 'fill the current paragraph'. It might work for
> you if you mark the whole buffer, and call
>
> (unfill-paragraph t)
>
> instead, since that should take care of iterating over all the
> paragraphs (untested).
>
> Robert


-- 
Dr. Gerald Wildgruber
Institut für Philosophie, Literatur-, Wissenschafts- und Technikgeschichte
Literaturwissenschaft mit Schwerpunkt Literatur und Wissenschaft
Technische Universität Berlin
Straße des 17. Juni 135
D-10623 Berlin
http://www.philosophie.tu-berlin.de/menue/home/
T. +49 (0)30 314 25924
F. +49 (0)30 314 23107
wildgruber@tu-berlin.de
---------------------
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Noam Postavsky]

On Tue, 2 Oct 2018 at 08:11, Gerald Wildgruber <wildgruber@tu-berlin.de> wrote:

> (dolist (f argv)
>   (find-file f)
>   (mark-whole-buffer)
>   (unfill-paragraph t)
>   (org-forward-paragraph)
>   (save-buffer))
>
> ------------------------------------------------------------
>
> I call this from a terminal like so:
>
> emacs -Q --batch --script unfill.el FILE.org
>
> Unfortunately, nothing happens; I get two "Mark set" messages in the terminal, but the file itself remains untouched. It isn't edited at all, same timestamp etc.

I think the problem is that find-file only makes sense for interactive
functions, because it switches the displayed buffer (which takes
effect after the current command finishes), but doesn't actually set
the current buffer (i.e., no immediate change).

So you should have

    (with-current-buffer (find-file-noselect f)
      (mark-whole-buffer)
      ...)

By the way, I don't see the benefit of invoking Emacs in batch mode
here, especially since this sounds like a one-off transformation. Just
put the expression which does the work into *scratch* and evaluate.
Something like this:

    (let ((fill-column most-positive-fixnum))
      (dolist (f (directory-files-recursively
                  "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
        (with-current-buffer (find-file-noselect f)
          (fill-region (point-min) (point-max)) ; Unfill, per `fill-column'
          (save-buffer))))

(of course replace the (directory-files-recursively...) thing with
(list "test-file.org") first to make sure it works right.)

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

Thanks.

On Di, Okt 02 2018, Noam Postavsky <npostavs@gmail.com> wrote:

> I think the problem is that find-file only makes sense for interactive
> functions, because it switches the displayed buffer (which takes
> effect after the current command finishes), but doesn't actually set
> the current buffer (i.e., no immediate change).
>
> So you should have
>
>     (with-current-buffer (find-file-noselect f)
>       (mark-whole-buffer)
>       ...)

The script then looks like this:

(dolist (f argv)
  (with-current-buffer (find-file-noselect f)
  (mark-whole-buffer)
  (unfill-paragraph t)
  (org-forward-paragraph)
  (save-buffer)))

It turns out that this works just on the FIRST paragraph of the file! Only this one is edited. The script does not move to the next paragraph.

So, what works well in a kbd macro (moving point forward to next paragraph via org-forward-paragraph) does not in this script.

Cf. docstring of this function: "Move forward to beginning of next paragraph or equivalent. The function moves point to the beginning of the next visible structural element, which can be a paragraph, a table, a list item, moving point to beginning".

> put the expression which does the work into *scratch* and evaluate.
> Something like this:
>
>     (let ((fill-column most-positive-fixnum))
>       (dolist (f (directory-files-recursively
>                   "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
>         (with-current-buffer (find-file-noselect f)
>           (fill-region (point-min) (point-max)) ; Unfill, per `fill-column'
>           (save-buffer))))

Yes, that almost did it!

But: for some reason the "fill-region" function you use does not work on list constructs like:

1. text text text text text text text text text
2. text text text text text text text text text text text
3. text text text text text text text text text text

The text in these constructs doesn't get unfilled, -- whereas it is correctly unfilled by using the "unfill-paragraph" function (which ultimately calls "fill-paragraph").

I tried to rewrite your code to use "fill-paragraph" instead of "fill-region" but didn't succeed.

Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.

--
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Noam Postavsky]

On Wed, 3 Oct 2018 at 06:11, Gerald Wildgruber <wildgruber@tu-berlin.de> wrote:

> (dolist (f argv)
>   (with-current-buffer (find-file-noselect f)
>   (mark-whole-buffer)
>   (unfill-paragraph t)
>   (org-forward-paragraph)
>   (save-buffer)))
>
> It turns out that this works just on the FIRST paragraph of the file! Only this one is edited. The script does not move to the next paragraph.

Right, I guess the mark-whole-buffer thing doesn't have the intended
effect because it relies on transient-mark-mode, which is probably
tied up with the command loop.

> So, what works well in a kbd macro (moving point forward to next paragraph via org-forward-paragraph) does not in this script.

It makes sense that the paragraph movement doesn't work, because you
only call it once per file, so you unfill the 1st paragraph, move to
the 2nd, and then leave.

> Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.

Perhaps this?

(let ((fill-column most-positive-fixnum))
  (dolist (f (directory-files-recursively
              "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
    (with-current-buffer (find-file-noselect f)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))

Re: How to do a massive unfill paragraph operation over several hundred files?

[Gerald Wildgruber]

On Do, Okt 04 2018, Noam Postavsky <npostavs@gmail.com> wrote:

>> Main problem seems to be: how to iterate through ALL paragraphs of the buffer programmatically, applying "fill-paragraph" each time anew.
>
> Perhaps this?
>
> (let ((fill-column most-positive-fixnum))
>   (dolist (f (directory-files-recursively
>               "dir/with/my/files/" (rx (or ".tex" ".org") eos)))
>     (with-current-buffer (find-file-noselect f)
>       (while (not (eobp))
>         (fill-paragraph)
>         (org-forward-paragraph))
>       (save-buffer))))

Thanks Noam, this almost did it! but I couldn't see through a last problem, it sometimes worked, sometimes didn't. Here's the solution someone from the emacs Org mode mailing list pointed me to:

http://lists.gnu.org/archive/html/emacs-orgmode/2018-10/msg00070.html

I had to UNFOLD the entries before working on them, even programmatically, via "(org-show-all)".

I do this now from a lisp buffer and both functions work exactly as expected:

single file:
============

(let ((fill-column most-positive-fixnum))
  (dolist (f (list "~/lorem.org"))
    (with-current-buffer (find-file-noselect f)
      (org-show-all)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))


recursively through dir tree:
=============================

(let ((fill-column most-positive-fixnum))
  (dolist (f (directory-files-recursively
              "/dirs/with/org/files/" (rx (or ".org" ".outl") eos)))
    (with-current-buffer (find-file-noselect f)
      (org-show-all)
      (while (not (eobp))
        (fill-paragraph)
        (org-forward-paragraph))
      (save-buffer))))


That's great; I'm glad it works now from within emacs itself (and not via sed, awk or tr as I tried before), harnessing all the knowledge the editor has of its own constructs, especially with some of the more complicated list and enumeration structures, -- all of them are correctly unfilled. This is solved now.

Thanks again to everyone who contributed!

Gerald.


-- 
Sent with mu4e

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> It turns out that this works just on the
> FIRST paragraph of the file! Only this one is
> edited. The script does not move to the
> next paragraph.

Yeah, told you so :)

For this to work, one way would be to put
a loop inside the loop, where the outer loop
will iterate the files and the inner loop will
iterate the paragraphs.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> (dolist (f argv) (find-file f)
> (mark-whole-buffer) (unfill-paragraph t)
> (org-forward-paragraph) (save-buffer))

This looks a bit wierd but I'm unsure because
I'm not as close to whatever it is you're
trying to do. What does `org-forward-paragraph'
do? Shouldn't you have a loop for that as well
if you you want to do this to several
paragraphs? But then, isn't `mark-whole-buffer'
something that runs contradictory to this?
Even so, something should happen what I can
see. Like the FIRST paragraph getting unfilled
at the very least.

Try to replace it with something simpler.
Like adding a single char to each file:

    (insert ?a)

If this works, your problem is with the
`dolist' stuff and you just need the details
figured out.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: How to do a massive unfill paragraph operation over several hundred files?

[Emanuel Berg]

Gerald Wildgruber wrote:

> Yes, that is the whole point I didn't
> understand! What does "ARGS" have to be when
> called non-interactively in a script.
> From the doc string of the function:
>
> (defun unfill-paragraph (&optional region)
>   "Takes a multi-line paragraph and makes it into a single line of text."
>   (interactive (progn (barf-if-buffer-read-only) '(t)))
>   (let ((fill-column (point-max)))
>     (fill-paragraph nil region)))
>
> I cannot see what to put here.

When you call that interactively what happens
is `barf-if-buffer-read-only', which

    [s]ignal[s] a `buffer-read-only' error if
    the current buffer is read-only

but if that doesn't hold (i.e. the buffer isn't
read-only), then "region", the optional
argument, is assigned `t'.

However when you call it from Lisp, NONE of
this happens so "region" remains nil!

Observe:

(defun test-interactive-optional-args (&optional arg)
  (interactive (progn (barf-if-buffer-read-only) '(t)))
  (message "arg: %s" arg) )
  
(call-interactively
#'test-interactive-optional-args)     ; arg: t

 (test-interactive-optional-args)     ; arg: nil
 
 (test-interactive-optional-args t)   ; arg: t

and of course:

 (test-interactive-optional-args nil) ; arg: nil

So in so many words, why don't you pass `t' as
the argument? :)

-- 
underground experts united
http://user.it.uu.se/~embe8573