New feature? Remove duplicate subheadings, preserving order

New feature? Remove duplicate subheadings, preserving order

[Allen Li]

I wrote a command to remove duplicate subheadings, which I use to
remove duplicate captured links among other things.  Would this be a
useful addition to Org mode?

I have included it below for reference.  I will clean it up a bit if
it's a worthy feature.

(defun mir-org-uniq ()
  "Remove duplicate subheadings, preserving order."
  (interactive)
  (let ((seen (make-hash-table :test 'equal))
        (removed 0))
    (save-excursion
      (org-map-entries (lambda ()
                         (let ((heading (org-get-heading t t t t)))
                           (if (not (gethash heading seen))
                               (puthash heading t seen)
                             (org-cut-subtree)
                             (org-backward-heading-same-level 1)
                             (setq removed (1+ removed)))))
                       (format "LEVEL=%s" (1+ (org-current-level)))
                       'tree))
    (message "Removed %d duplicates" removed)))

Re: New feature? Remove duplicate subheadings, preserving order

[Marcin Borkowski]

On 2018-01-01, at 03:42, Allen Li <vianchielfaura@gmail.com> wrote:

> I wrote a command to remove duplicate subheadings, which I use to
> remove duplicate captured links among other things.  Would this be a
> useful addition to Org mode?

IMHO yes.

-- 
Marcin Borkowski

Re: New feature? Remove duplicate subheadings, preserving order

[Nicolas Goaziou]

Hello,

Allen Li <vianchielfaura@gmail.com> writes:

> I wrote a command to remove duplicate subheadings, which I use to
> remove duplicate captured links among other things.  Would this be a
> useful addition to Org mode?
>
> I have included it below for reference.  I will clean it up a bit if
> it's a worthy feature.
>
> (defun mir-org-uniq ()
>   "Remove duplicate subheadings, preserving order."
>   (interactive)
>   (let ((seen (make-hash-table :test 'equal))
>         (removed 0))
>     (save-excursion
>       (org-map-entries (lambda ()
>                          (let ((heading (org-get-heading t t t t)))
>                            (if (not (gethash heading seen))
>                                (puthash heading t seen)
>                              (org-cut-subtree)
>                              (org-backward-heading-same-level 1)
>                              (setq removed (1+ removed)))))
>                        (format "LEVEL=%s" (1+ (org-current-level)))
>                        'tree))
>     (message "Removed %d duplicates" removed)))

Duplicates headings are not necessarily wrong. I think this is too
specific to be integrated in Org proper.

Maybe we could add a check for duplicates headings in Org Lint instead,
and add this to Worg, in a "tools" page.

Or we could check for duplicate headings _including contents_, which are
more likely to be an error.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Mon, Jan 1, 2018 at 2:04 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Duplicates headings are not necessarily wrong. I think this is too
> specific to be integrated in Org proper.
>
> Maybe we could add a check for duplicates headings in Org Lint instead,
> and add this to Worg, in a "tools" page.
>
> Or we could check for duplicate headings _including contents_, which are
> more likely to be an error.
>
> WDYT?

Org mode is fundamentally an outliner, and one often makes lists with
an outliner.  Filtering out duplicates from a list seems to me like a
common need.  I wrote such a command to support some of my work flows,
and I posted this here because I think there is a possibility that
other Org users might also find it useful.

If this is not so, I’m perfectly okay just keeping this in my personal
config.

Re: New feature? Remove duplicate subheadings, preserving order

[Nicolas Goaziou]

Allen Li <vianchielfaura@gmail.com> writes:

> Org mode is fundamentally an outliner, and one often makes lists with
> an outliner.  Filtering out duplicates from a list seems to me like a
> common need.

AFAIK, this is the first time this need is expressed on this ML. There
is no equivalent in "org-list.el" either.

Anyway, I'm not questioning the usefulness of the feature in your
workflow. AIUI, in your implementation, duplicates are headlines with
the same title, but without considering TODO keyword, priority, comment
status, tags or contents. So,

  * DONE Summary :Alice:
  * TODO Summary :Bob:

are duplicates. Isn't it a bit too tolerant? We may be able to find
a more general function that still suits you.

> I wrote such a command to support some of my work flows, and I posted
> this here because I think there is a possibility that other Org users
> might also find it useful.

You didn't answer to any of my suggestions, tho. Are they fundamentally
wrong? I.e., wouldn't warning instead of deleting more useful? Or would
it make more sense to include contents when looking for duplicates ? In
the latter case, maybe a prefix argument could toggle headline check and
full check.

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Mon, Jan 1, 2018 at 10:26 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Allen Li <vianchielfaura@gmail.com> writes:
>
>> Org mode is fundamentally an outliner, and one often makes lists with
>> an outliner.  Filtering out duplicates from a list seems to me like a
>> common need.
>
> AFAIK, this is the first time this need is expressed on this ML. There
> is no equivalent in "org-list.el" either.
>
> Anyway, I'm not questioning the usefulness of the feature in your
> workflow. AIUI, in your implementation, duplicates are headlines with
> the same title, but without considering TODO keyword, priority, comment
> status, tags or contents. So,
>
>   * DONE Summary :Alice:
>   * TODO Summary :Bob:
>
> are duplicates. Isn't it a bit too tolerant? We may be able to find
> a more general function that still suits you.

I see this feature as primarily being useful on lists.  So for example:

* Things to buy
** cabbage
** milk
** carrots
** milk

I don’t know if a more intelligent way of handling tags and todo
keywords is worth the extra complexity, but in the use case that I
imagine it makes sense to match using only the heading/list item:

* Things to buy
** TODO cabbage
** DONE milk :store1:
   Maybe I forgot a tag here.  Oh well, I already bought the milk.
** TODO carrots
** TODO milk :store1:store2:

>
>> I wrote such a command to support some of my work flows, and I posted
>> this here because I think there is a possibility that other Org users
>> might also find it useful.
>
> You didn't answer to any of my suggestions, tho. Are they fundamentally
> wrong? I.e., wouldn't warning instead of deleting more useful? Or would
> it make more sense to include contents when looking for duplicates ? In
> the latter case, maybe a prefix argument could toggle headline check and
> full check.

Since the point would be remove duplicates from lists, I don’t think
warning is very useful.  I would want to remove the duplicate list
items, not get a warning about it and delete them manually.  Perhaps
that would be a useful additional feature however (like uniq -d).

It doesn’t make sense to include the contents because I see this as
primarily being useful for list items.  In particular, we would want
to ignore log entries and properties for the sake of matching
(intelligent property or logbook merging might be useful, but adds
complexity).

I don’t think doing a full text check is useful, but if someone has a
use case for that, please speak up.

Re: New feature? Remove duplicate subheadings, preserving order

[Adam Porter]

Allen Li <vianchielfaura@gmail.com> writes:

> I don’t know if a more intelligent way of handling tags and todo
> keywords is worth the extra complexity, but in the use case that I
> imagine it makes sense to match using only the heading/list item:
>
> * Things to buy
> ** TODO cabbage
> ** DONE milk :store1:
>    Maybe I forgot a tag here.  Oh well, I already bought the milk.
> ** TODO carrots
> ** TODO milk :store1:store2:
>
> ...
>
> It doesn’t make sense to include the contents because I see this as
> primarily being useful for list items.  In particular, we would want
> to ignore log entries and properties for the sake of matching
> (intelligent property or logbook merging might be useful, but adds
> complexity).

I think such a command should check all heading data by default,
because that's the safest option.  A user who commonly needs to ignore
one or more types of data could use a custom function that calls the
command with arguments to disable checking of certain types.

> Since the point would be remove duplicates from lists, I don’t think
> warning is very useful.  I would want to remove the duplicate list
> items, not get a warning about it and delete them manually.  Perhaps
> that would be a useful additional feature however (like uniq -d).

I think warning or asking for confirmation should be the default action,
because it's the safest option.  Users who want to skip that could use a
prefix argument or call it from a custom command.

> I don’t think doing a full text check is useful, but if someone has a
> use case for that, please speak up.

An example where this would be useful is if the user has copied and
pasted subtrees and accidentally pasted one more than once.

I argue here for the safest behavior by default because I've found that,
in very large Org buffers, it's easy to lose my place in the file, and
it's easy to accidentally do something that I didn't mean to, without
noticing.  IMO this is simply a consequence of Org buffers still being
plain-text.

So it is quite conceivable to me that a user might intentionally give
two headings the same name (e.g. a user who captures quotations to an
inbox file might have two "Quote" headings that are completely
different), or might accidentally duplicate a subtree and then make a
desired change to one of them without realizing there was a
duplicate--then he might use this deduplication command and accidentally
delete a subtree he didn't mean to, resulting in data loss.

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Mon, Jan 1, 2018 at 8:07 PM, Adam Porter <adam@alphapapa.net> wrote:
> Allen Li <vianchielfaura@gmail.com> writes:
>
>> I don’t know if a more intelligent way of handling tags and todo
>> keywords is worth the extra complexity, but in the use case that I
>> imagine it makes sense to match using only the heading/list item:
>>
>> * Things to buy
>> ** TODO cabbage
>> ** DONE milk :store1:
>>    Maybe I forgot a tag here.  Oh well, I already bought the milk.
>> ** TODO carrots
>> ** TODO milk :store1:store2:
>>
>> ...
>>
>> It doesn’t make sense to include the contents because I see this as
>> primarily being useful for list items.  In particular, we would want
>> to ignore log entries and properties for the sake of matching
>> (intelligent property or logbook merging might be useful, but adds
>> complexity).
>
> I think such a command should check all heading data by default,
> because that's the safest option.  A user who commonly needs to ignore
> one or more types of data could use a custom function that calls the
> command with arguments to disable checking of certain types.

I don’t see a use case for checking all heading data.

>> Since the point would be remove duplicates from lists, I don’t think
>> warning is very useful.  I would want to remove the duplicate list
>> items, not get a warning about it and delete them manually.  Perhaps
>> that would be a useful additional feature however (like uniq -d).
>
> I think warning or asking for confirmation should be the default action,
> because it's the safest option.  Users who want to skip that could use a
> prefix argument or call it from a custom command.

There is always undo and automatic Emacs file backups.

>> I don’t think doing a full text check is useful, but if someone has a
>> use case for that, please speak up.
>
> An example where this would be useful is if the user has copied and
> pasted subtrees and accidentally pasted one more than once.

In that case, the user should use undo instead of a remove duplicates
command.

> I argue here for the safest behavior by default because I've found that,
> in very large Org buffers, it's easy to lose my place in the file, and
> it's easy to accidentally do something that I didn't mean to, without
> noticing.  IMO this is simply a consequence of Org buffers still being
> plain-text.

I don’t agree with this philosophy.  Org and Emacs already has lots of
commands that can cause damage, for example org-sort-entries which my
remove duplicates command is modeled after (both modify the direct children
under the heading at point irreversibly ignoring undo).  If this command should
warn, then org-sort-entries should also warn.  If org-sort-entries does not need
to warn, then this command does not need to warn.

Emacs makes backups by default and supports undo, which under my
philosophy is good enough; we shouldn’t be constantly asking for
confirmation to prevent user error.  That just causes pop-up dialog fatigue.
For example, everyone clicks OK on pop-up confirmation boxes without
reading them.
These kinds of confirmation prompts are worse than useless; they slow
down and annoy the user without providing any protection.  Undo is the
better solution.

> So it is quite conceivable to me that a user might intentionally give
> two headings the same name (e.g. a user who captures quotations to an
> inbox file might have two "Quote" headings that are completely
> different), or might accidentally duplicate a subtree and then make a
> desired change to one of them without realizing there was a
> duplicate--then he might use this deduplication command and accidentally
> delete a subtree he didn't mean to, resulting in data loss.

I think it would be more useful for list members to post actual use
cases than hypothesize about what people want.

For me, the use case is filtering out duplicates from a list,
e.g. groceries to buy or links to read captured with timestamps and
other metadata, so checking the tags, todo, or body text is not useful,
warning is not useful.

Based on the responses I have gotten, it sounds like this feature is
too specialized to be worth including in Org mode, so I will stop
pursuing this unless people post actual use cases/desire for
the feature.

Re: New feature? Remove duplicate subheadings, preserving order

[Robert Horn]

Allen Li writes:

> On Mon, Jan 1, 2018 at 8:07 PM, Adam Porter <adam@alphapapa.net> wrote:
>
> I don’t see a use case for checking all heading data.
>

I can see such cases arising from templates and time tracking.  I can
have a template that captures telephone calls.  The call comes in and I
start the template.  At this point the heading is just "Received Phone
Call" and a time tracking start.  Time tracking is eventually kept in a
drawer, not in the headline.

I might eventually go back an revise the headline based on notes from
the call, but that will not happen during the call.  It's quite likely
that sorting out the calls will happen at the end of the day or the next
day.

Similarly, I receive lab results.  These will initially be a headline
with just "Lab Result", a time tag like CLOCK, and a tag to indicate
that it is a lab result.  Some time later I might move them around to
attach them to a patient or project, but often by just moving them as
element into the right section for that patient or project.  So these
also have the same headline contents and different headline data.

R Horn

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Tue, Jan 2, 2018 at 6:36 AM, Robert Horn <rjhorniii@gmail.com> wrote:
>
> Allen Li writes:
>
>> On Mon, Jan 1, 2018 at 8:07 PM, Adam Porter <adam@alphapapa.net> wrote:
>>
>> I don’t see a use case for checking all heading data.
>>
>
> I can see such cases arising from templates and time tracking.  I can
> have a template that captures telephone calls.  The call comes in and I
> start the template.  At this point the heading is just "Received Phone
> Call" and a time tracking start.  Time tracking is eventually kept in a
> drawer, not in the headline.
>
> I might eventually go back an revise the headline based on notes from
> the call, but that will not happen during the call.  It's quite likely
> that sorting out the calls will happen at the end of the day or the next
> day.
>
> Similarly, I receive lab results.  These will initially be a headline
> with just "Lab Result", a time tag like CLOCK, and a tag to indicate
> that it is a lab result.  Some time later I might move them around to
> attach them to a patient or project, but often by just moving them as
> element into the right section for that patient or project.  So these
> also have the same headline contents and different headline data.

It doesn’t sound like you end up with duplicates though?  How do you
envision using duplicate removal for your workflows?

In any case, it sounds like you want to assign unique IDs to each entry
as suggested by Florian and remove duplicates using IDs instead of
matching against heading data, which could collide through sheer luck.

Re: New feature? Remove duplicate subheadings, preserving order

[Nick Dokos]

Allen Li <vianchielfaura@gmail.com> writes:

>
> I don’t see a use case for checking all heading data.
>
>>> Since the point would be remove duplicates from lists, I don’t think
>>> warning is very useful.  I would want to remove the duplicate list
>>> items, not get a warning about it and delete them manually.  Perhaps
>>> that would be a useful additional feature however (like uniq -d).
>>
>> I think warning or asking for confirmation should be the default action,
>> because it's the safest option.  Users who want to skip that could use a
>> prefix argument or call it from a custom command.
>
> There is always undo and automatic Emacs file backups.
>

There be dragons.

The problem is that some things happen invisibly and far away from
where you are, so you don't know about it and you don't find out for a
couple of weeks.  Undo and automatic backups are useless in that case.

That *has* happened: there have been multiple postings in the ML about
such problems. Whenever it has happened, the devs have always modified
org to make it safer: that is the prudent thing to do and the correct
course of action IMO.

Hell hath no fury like an orgmode user who lost part of his/her
precious org file because of an errant keystroke a month ago and was
not aware of the loss until it was too late.

-- 
Nick

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Tue, Jan 2, 2018 at 8:36 AM, Nick Dokos <ndokos@gmail.com> wrote:
> Allen Li <vianchielfaura@gmail.com> writes:
>>
>> There is always undo and automatic Emacs file backups.
>>
>
> There be dragons.
>
> The problem is that some things happen invisibly and far away from
> where you are, so you don't know about it and you don't find out for a
> couple of weeks.  Undo and automatic backups are useless in that case.
>
> That *has* happened: there have been multiple postings in the ML about
> such problems. Whenever it has happened, the devs have always modified
> org to make it safer: that is the prudent thing to do and the correct
> course of action IMO.
>
> Hell hath no fury like an orgmode user who lost part of his/her
> precious org file because of an errant keystroke a month ago and was
> not aware of the loss until it was too late.

I can see where you're coming from, but for me there are various reasons
why I don’t think warning is right:

1. org-sort-entries, which performs an action of similar scope and
   destructiveness, does not need to warn so far.

2. Since I see the only use case for warning is checking beforehand, a
   user that uses this command frequently is not going to type C-c d C-u
   C-c d every time (assuming the user has bound this command to C-c d),
   they’re just going to type C-u C-c d or get frustrated and just bind
   the actual command without warning to C-c d.  So warning provides
   zero safety in practice.

   Another possibility is using a y or n confirmation prompt before
   removing duplicates, however this falls into the same trap that a
   user who uses this frequently is just going to bind the command to a
   key and disable this check.

3. I don’t propose binding this command to any key by default, and I
   don’t think M-x org-remove-duplicates RET is a very common typo.

4. The only commands in Emacs that warn beforehand are truly
   irreversible commands, like deleting in Dired or killing a buffer.
   Everything else in Emacs follows the philosophy of using undo if the
   user makes a mistake, including lots of commands that could have
   unintentional, low visibility effects.  I would prefer following this
   policy unless it proves to actually be a problem.  It seems like
   org-sort-entries in practice has not suffered from this problem, so I
   believe a remove duplicates command will similarly not suffer from
   this problem in practice.

5. Everyone should be keeping reliable backups.  This is reiterated all
   the time, yet no one seems to follow it? =)

>
> --
> Nick
>
>

Re: New feature? Remove duplicate subheadings, preserving order

[Adam Porter]

Allen Li <vianchielfaura@gmail.com> writes:

> Designing around actual use cases that users have an immediate use for
> is better than trying to predict what users might need in the far
> future, especially if adding those features requires extra complexity.

You seem to be approaching this from a "use case" perspective.  I and
several other posters on the list are concerned from the "potential
damage" perspective.  It's vitally important as software developers to
anticipate potential user *actions*, regardless of their intentions or
needs, and to proactively defend against mistakes that may cause data
loss.

>  Everyone should be keeping reliable backups.  This is reiterated all
>  the time, yet no one seems to follow it? =)

As I mentioned, I am not merely hypothesizing: I have experienced such
data loss myself, which I only recovered months after the fact when I
noticed and was able to track it down in the git repository I
automatically commit most of my Org files to.  This was a
time-consuming, laborious, manual process which we cannot expect most
Org users to be able to do; how many Org users do you think know how to
use git and commit all their Org files to it?  On top of that, as I and
others mentioned, due to the nature of Emacs, Org, and plain-text
buffers, it is very easy for such changes to happen outside the visible
portion of the buffer, in which case even the most experienced user is
unlikely to notice such data loss.  In that case, it might go
permanently unnoticed.  For example, I have some large Org files that I
capture data into, with hundreds or thousands of top-level headings.  If
I accidentally cut one of those subtrees, outside the visible portion of
the buffer, which I had captured weeks or months earlier, how would I
even know that it was missing?  Most likely I would not, because the
whole point of storing them in Org is that I will forget about them if I
don't.

An analogy is, if you ran an "rm -rf" command with a wildcard, and
accidentally left off a slash somewhere without realizing it, how long
would it be before you noticed that you had deleted the wrong data?
What if you were operating on a directory deep in a hierarchy that is
essentially an archive of rarely accessed files?  How long would it be
before you noticed the mistake?  Would you still be able to recover from
it?  What if this happened in a script you wrote, rather than an
interactive command?  Have you ever made a mistake in a Bash script that
had undesirable effects?  I can't help but be reminded of this recent
classic:

https://github.com/ValveSoftware/steam-for-linux/issues/3671

Another anecdote: I once almost lost my GPG private key, because somehow
(I still have no idea what happened) it was truncated.  This went
unnoticed by me for a long time, and the truncated file was *backed up*
over and over again.  When I finally noticed, all of my readily
available backups had the truncated version of the file.  I was
only--very luckily!--able to recover it by digging out old CD-R backups
I had made years earlier, one of which had an intact copy of the file.
That experience taught me some lessons, among which are 1. Keep your old
backups, and 2. Respect Murphy's Law.

> If you had an immediate use case in mind, I would love to hear it.
> There's no need to suggest your use case as conceivable if it's
> something you could benefit from immediately.

So the "use case" here is simply "using Org, without losing data by
accidentally activating a command in Emacs that operates on data that
may or may not be currently visible--because Emacs and Org are
incredibly complex software that is extensively customized by users in
ways that the developers often do not anticipate--therefore we should
use extra caution in code that manipulates user data in destructive
ways."

I hope that these explanations help clarify our perspectives.  We are
not attacking you, your code, or its potential usefulness.  We are
simply concerned about implementing such things in ways that are best
for users.

Thanks,
Adam

Re: New feature? Remove duplicate subheadings, preserving order

[Adam Porter]

Nick Dokos <ndokos@gmail.com> writes:

> There be dragons.
>
> The problem is that some things happen invisibly and far away from
> where you are, so you don't know about it and you don't find out for a
> couple of weeks.  Undo and automatic backups are useless in that case.
>
> That *has* happened: there have been multiple postings in the ML about
> such problems. Whenever it has happened, the devs have always modified
> org to make it safer: that is the prudent thing to do and the correct
> course of action IMO.
>
> Hell hath no fury like an orgmode user who lost part of his/her
> precious org file because of an errant keystroke a month ago and was
> not aware of the loss until it was too late.

Indeed.  Maybe I'm just paranoid, but having worked with Org code a bit,
I still wonder sometimes if I have ever accidentally wiped out a subtree
without noticing.  Would I ever notice that it's missing?  Even if it's
stored in git or a backup, how can I restore something that I don't know
needs restoring?

Some of this is simply the nature of computers, I think--a keystroke
here, a blink of the eye there, and poof, the data is gone.  If the
point is in one buffer when my fingers press C-c C-x C-w, but my eyes
are in another buffer, does the subtree still get deleted?  :)

And despite how great Emacs and Org are, this is one area in which their
power may make them more vulnerable to such issues.  Their use of global
state and special variables also makes unintended consequences easier to
achieve.

This is why I think we should always be very careful.  Org is nothing if
we can't trust it to keep our data safe!  :)

Re: New feature? Remove duplicate subheadings, preserving order

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2254 bytes --]

Is there any possible way to prevent it more reliably?

I am aware of org-catch-invisible-edits, but this is obviously not
enough. Does it make sense to generate some kind of subtree based diff
after each change, so that user can review all recent changes in org
files?

Ihor

Adam Porter <adam@alphapapa.net> writes:

> Nick Dokos <ndokos@gmail.com> writes:
>
>> There be dragons.
>>
>> The problem is that some things happen invisibly and far away from
>> where you are, so you don't know about it and you don't find out for a
>> couple of weeks.  Undo and automatic backups are useless in that case.
>>
>> That *has* happened: there have been multiple postings in the ML about
>> such problems. Whenever it has happened, the devs have always modified
>> org to make it safer: that is the prudent thing to do and the correct
>> course of action IMO.
>>
>> Hell hath no fury like an orgmode user who lost part of his/her
>> precious org file because of an errant keystroke a month ago and was
>> not aware of the loss until it was too late.
>
> Indeed.  Maybe I'm just paranoid, but having worked with Org code a bit,
> I still wonder sometimes if I have ever accidentally wiped out a subtree
> without noticing.  Would I ever notice that it's missing?  Even if it's
> stored in git or a backup, how can I restore something that I don't know
> needs restoring?
>
> Some of this is simply the nature of computers, I think--a keystroke
> here, a blink of the eye there, and poof, the data is gone.  If the
> point is in one buffer when my fingers press C-c C-x C-w, but my eyes
> are in another buffer, does the subtree still get deleted?  :)
>
> And despite how great Emacs and Org are, this is one area in which their
> power may make them more vulnerable to such issues.  Their use of global
> state and special variables also makes unintended consequences easier to
> achieve.
>
> This is why I think we should always be very careful.  Org is nothing if
> we can't trust it to keep our data safe!  :)
>
>

-- 
Ihor Radchenko,
PhD Student
Singapore University of Technology and Design,
8 Somapah Road Singapore 487372
Email: yantar92@gmail.com, ihor_radchenko@mymail.sutd.edu.sg
Tel: +6584017977

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: New feature? Remove duplicate subheadings, preserving order

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

> Is there any possible way to prevent it more reliably?
>
> I am aware of org-catch-invisible-edits, but this is obviously not
> enough. Does it make sense to generate some kind of subtree based diff
> after each change, so that user can review all recent changes in org
> files?

That's a good question.  If you want to be max-paranoid, I guess you
should put all your Org files in git, and review and commit all changes
with magit.  For certain things that might make sense (e.g. I do that
with readme files in published projects), but for my personal Org files,
that would feel like a burden to me.

I do store my personal Org files in git, but I don't review the changes
manually.  I commit the changes automatically with a cron job and when my
"emacs-raise-or-run" script raises or minimizes the Emacs window.  If I
ever need to review the changes, I can use magit or gitk.

If you add this to your .git/config file in a git repo of Org files, it
uses Org heading lines as diff headers, which helps when reviewing
changes:

[diff "org"]
    xfuncname = "^\\*+ +.*$"

Re: New feature? Remove duplicate subheadings, preserving order

[Florian Beck]

> AFAIK, this is the first time this need is expressed on this ML. There
> is no equivalent in "org-list.el" either.

A way to handle duplicates would be useful, indeed. But a basic function 
should only remove duplicates that are truly identical (same properties, 
same tags, same/no content). Still, removing true duplicates from 
subtrees (AND lists) would be useful.

More useful would be a slightly more general approach. I have three 
kinds of duplicates:
  - duplicate IDs (which are handled rather poorly),
  - duplicate content (which often is only almost identical) and
  - duplicate headings (which usually I want to rectify when they are on
    the same level of the same subtree)

As you can see, a fixed concept of duplication is probably not going to 
work.

What I'd like is a function finds duplicates according to scope, match 
(as in `org-map-entries') and a user defined function. This function 
should then display the problem cases (via agenda view?). Then we need a 
couple of convenience functions like
  - delete all duplicates but the one at point,
  - mark duplicates I want to keep,
  - uniquify entries (tricky; for headlines maybe prompt the user; for
    IDs, we should check if the ID is referenced from somewhere)
  - merge entries.

But then, I also have duplicates (in content) I want to keep, e.g. one 
in my notes and in a writing project. So, we'd need a property like 
"DUPLICATE_OF".

Re: New feature? Remove duplicate subheadings, preserving order

[Allen Li]

On Tue, Jan 2, 2018 at 7:28 AM, Florian Beck <fb@fbeck.net> wrote:
>
>> AFAIK, this is the first time this need is expressed on this ML. There
>> is no equivalent in "org-list.el" either.
>
>
> A way to handle duplicates would be useful, indeed. But a basic function
> should only remove duplicates that are truly identical (same properties,
> same tags, same/no content). Still, removing true duplicates from subtrees
> (AND lists) would be useful.
>
> More useful would be a slightly more general approach. I have three kinds of
> duplicates:
>  - duplicate IDs (which are handled rather poorly),
>  - duplicate content (which often is only almost identical) and
>  - duplicate headings (which usually I want to rectify when they are on
>    the same level of the same subtree)
>
> As you can see, a fixed concept of duplication is probably not going to
> work.

It sounds like this problem might vary too much across use cases to
generalize and include in Org mode because...

>
> What I'd like is a function finds duplicates according to scope, match (as
> in `org-map-entries') and a user defined function. This function should then
> display the problem cases (via agenda view?). Then we need a couple of
> convenience functions like
>  - delete all duplicates but the one at point,
>  - mark duplicates I want to keep,
>  - uniquify entries (tricky; for headlines maybe prompt the user; for
>    IDs, we should check if the ID is referenced from somewhere)
>  - merge entries.
>
> But then, I also have duplicates (in content) I want to keep, e.g. one in my
> notes and in a writing project. So, we'd need a property like
> "DUPLICATE_OF".

You’re requesting a different feature, removing duplicates across all
agenda files.  My initial proposal was just for removing duplicate
direct children (whether by heading or full text pending discussion)
under the heading at point.