[BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Carlos Pita]

Hi all,

I don't see any clear reason why org-occur should hide unmatching
entries except at the top level, where it merely folds them. For files
containing lots of top-level entries this requires tree demoting in
order to hide irrelevant information, but this imposes constraints on
the tree structure that are not related to its author's goals.

It has been asked in the past:

- https://stackoverflow.com/questions/7602787

- https://emacs-orgmode.gnu.narkive.com/ogvwJnHD

I've been playing with org-show-context-detail and it doesn't seem to be
any option which actually hides top levels, though the description of
some options seem to imply that they should, I believe. This is why I'm
reporting this as a bug and not a feature request.

Best regards,
Carlos

---

Emacs  : GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin20.6.0, NS appkit-2022.60 Version 11.6 (Build 20G165))
 of 2021-11-14
Package: Org mode version 9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)

current state:
==============
(setq
 org-link-elisp-confirm-function 'yes-or-no-p
 org-directory "~/Documents"
 org-bibtex-headline-format-function '(closure
                                       (org-id-locations org-agenda-search-view-always-boolean org-agenda-overriding-header
                                        t)
                                       (entry) (cdr (assq :title entry)))
 org-persist-after-read-hook '(org-element--cache-persist-after-read)
 org-export-before-parsing-hook '(org-attach-expand-links)
 org-default-notes-file "~/Documents/Capturas.org"
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers org-cycle-show-empty-lines
                  org-optimize-window-after-visibility-change)
 org-persist-before-read-hook '(org-element--cache-persist-before-read)
 org-mode-hook '((closure
                  (org-agenda-skip-regexp org-table1-hline-regexp org-table-tab-recognizes-table\.el
                   org-table-dataline-regexp org-table-any-border-regexp org-agenda-restriction-lock-overlay
                   org-agenda-overriding-restriction org-agenda-diary-file org-complex-heading-regexp t)
                  nil (setq imenu-create-index-function 'org-imenu-get-tree))
                 (closure
                  (org--rds reftex-docstruct-symbol org-element-greater-elements visual-fill-column-width org-clock-history
                   org-agenda-current-date org-with-time org-defdecode org-def org-read-date-inactive org-ans2 org-ans1
                   org-columns-current-fmt-compiled org-clock-current-task org-clock-effort org-agenda-skip-function
                   org-agenda-skip-comment-trees org-agenda-archives-mode org-end-time-was-given org-time-was-given
                   org-log-note-extra org-log-note-purpose org-log-post-message org-last-inserted-timestamp
                   org-last-changed-timestamp org-entry-property-inherited-from org-blocked-by-checkboxes org-state
                   org-agenda-headline-snapshot-before-repeat org-agenda-buffer-name org-agenda-start-on-weekday
                   org-agenda-buffer-tmp-name org-priority-regexp org-mode-abbrev-table org-mode-syntax-table
                   org-element-use-cache org-element-cache-persistent buffer-face-mode-face org-tbl-menu org-org-menu
                   org-struct-menu org-entities org-last-state org-id-track-globally org-clock-start-time texmathp-why
                   remember-data-file org-agenda-tags-todo-honor-ignore-options iswitchb-temp-buflist
                   calc-embedded-open-mode calc-embedded-open-formula calc-embedded-close-formula align-mode-rules-list
                   org-emphasis-alist org-emphasis-regexp-components org-export-registered-backends org-modules
                   crm-separator org-babel-load-languages org-id-overriding-file-name org-indent-indentation-per-level
                   org-element-paragraph-separate org-agenda-buffer-name org-inlinetask-min-level t)
                  nil (add-hook 'change-major-mode-hook 'org-show-all 'append 'local))
                 (closure
                  (org-src-window-setup *this* org-babel-confirm-evaluate-answer-no org-babel-tangle-uncomment-comments
                   org-src-preserve-indentation org-src-lang-modes org-edit-src-content-indentation
                   org-babel-library-of-babel t)
                  nil (add-hook 'change-major-mode-hook #'org-babel-show-result-all 'append 'local))
                 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-confirm-shell-link-function 'yes-or-no-p
 org-reveal-start-hook '(org-decrypt-entry)
 outline-isearch-open-invisible-function 'outline-isearch-open-invisible
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-show-context-detail '((agenda . local) (bookmark-jump . lineage) (isearch . lineage) (occur-tree . minimal)
                           (default . ancestors))
 org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-cycle-separator-lines 1
 org-todo-keywords '((sequence "TODO" "NEXT" "|" "DONE"))
 outline-minor-mode-buttons '(("▶" "▼" outline--valid-char-p))
 org-support-shift-select t
 org-speed-command-hook '(org-speed-command-activate org-babel-speed-command-activate)
 org-preview-latex-default-process 'dvisvgm
 org-persist-before-write-hook '(org-element--cache-persist-before-write)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 org-link-shell-confirm-function 'yes-or-no-p
 org-babel-pre-tangle-hook '(save-buffer)
 org-agenda-loop-over-headlines-in-active-region nil
 org-occur-hook '(org-first-headline-recenter)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-link-parameters '(("attachment" :follow org-attach-follow :complete org-attach-complete-link)
                       ("id" :follow org-id-open) ("eww" :follow org-eww-open :store org-eww-store-link)
                       ("rmail" :follow org-rmail-open :store org-rmail-store-link)
                       ("mhe" :follow org-mhe-open :store org-mhe-store-link)
                       ("irc" :follow org-irc-visit :store org-irc-store-link :export org-irc-export)
                       ("info" :follow org-info-open :export org-info-export :store org-info-store-link)
                       ("gnus" :follow org-gnus-open :store org-gnus-store-link)
                       ("docview" :follow org-docview-open :export org-docview-export :store org-docview-store-link)
                       ("bibtex" :follow org-bibtex-open :store org-bibtex-store-link)
                       ("bbdb" :follow org-bbdb-open :export org-bbdb-export :complete org-bbdb-complete-link :store
                        org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link) ("doi" :follow org-link-doi-open :export org-link-doi-export)
                       ("file+sys") ("file+emacs") ("shell" :follow org-link--open-shell)
                       ("news" :follow
                        (closure
                         ((scheme . "news") (--dolist-tail--) org-ts-regexp org-time-stamp-formats org-src-source-file-name
                          org-outline-regexp-bol org-inhibit-startup org-id-link-to-org-use-id org-highlight-links
                          org-comment-string org-agenda-buffer-name clean-buffer-list-kill-buffer-names t)
                         (url arg) (browse-url (concat scheme ":" url) arg))
                        )
                       ("mailto" :follow
                        (closure
                         ((scheme . "mailto") (--dolist-tail--) org-ts-regexp org-time-stamp-formats
                          org-src-source-file-name org-outline-regexp-bol org-inhibit-startup org-id-link-to-org-use-id
                          org-highlight-links org-comment-string org-agenda-buffer-name clean-buffer-list-kill-buffer-names
                          t)
                         (url arg) (browse-url (concat scheme ":" url) arg))
                        )
                       ("https" :follow
                        (closure
                         ((scheme . "https") (--dolist-tail--) org-ts-regexp org-time-stamp-formats org-src-source-file-name
                          org-outline-regexp-bol org-inhibit-startup org-id-link-to-org-use-id org-highlight-links
                          org-comment-string org-agenda-buffer-name clean-buffer-list-kill-buffer-names t)
                         (url arg) (browse-url (concat scheme ":" url) arg))
                        )
                       ("http" :follow
                        (closure
                         ((scheme . "http") (--dolist-tail--) org-ts-regexp org-time-stamp-formats org-src-source-file-name
                          org-outline-regexp-bol org-inhibit-startup org-id-link-to-org-use-id org-highlight-links
                          org-comment-string org-agenda-buffer-name clean-buffer-list-kill-buffer-names t)
                         (url arg) (browse-url (concat scheme ":" url) arg))
                        )
                       ("ftp" :follow
                        (closure
                         ((scheme . "ftp") (--dolist-tail--) org-ts-regexp org-time-stamp-formats org-src-source-file-name
                          org-outline-regexp-bol org-inhibit-startup org-id-link-to-org-use-id org-highlight-links
                          org-comment-string org-agenda-buffer-name clean-buffer-list-kill-buffer-names t)
                         (url arg) (browse-url (concat scheme ":" url) arg))
                        )
                       ("help" :follow org-link--open-help :store org-link--store-help)
                       ("file" :complete org-link-complete-file) ("elisp" :follow org-link--open-elisp))
 org-metaup-hook '(org-babel-load-in-session-maybe)
 )

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Ihor Radchenko]

Carlos Pita <carlosjosepita2@gmail.com> writes:

> Hi all,
>
> I don't see any clear reason why org-occur should hide unmatching
> entries except at the top level, where it merely folds them. For files
> containing lots of top-level entries this requires tree demoting in
> order to hide irrelevant information, but this imposes constraints on
> the tree structure that are not related to its author's goals.

The reason is because nobody implemented this feature.
Note that you can also use org-agenda for the same purpose.

> I've been playing with org-show-context-detail and it doesn't seem to be
> any option which actually hides top levels, though the description of
> some options seem to imply that they should, I believe.

Can you elaborate? If the docstring is not clear enough, we can easily
improve it.

Best,
Ihor

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Carlos Pita]

Hi Ihor,

> Can you elaborate? If the docstring is not clear enough, we can easily
> improve it.

Consider for example:

    minimal   show current headline; if point is not on headline, also
show entry

So if you have:

* a
* b
** b1
** b2
** b3
* c

I see no reason why a match inside b2 will hide b1 and b3 but not a
and c (I'm referring to the headings, not the contents). The only way
to change the documentation to convey that is to state an explicit
exception for the top level, but that seems extremely ad hoc. Perhaps
there is a rationale, some presumption about what is in the top-level
take makes it a valuable context no matter what the preferred context
detail is set to, but if that's the case I find it too opinionated.

Best regards,
Carlos

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Carlos Pita]

As a concrete example:

* a
ca
* b
** ba
cba
** bb
cbb
** bc
cbc
* c
cc

Then C-c / / cbb:

* a...
* b...
** bb
cbb
...
* c...

This is also true going deeper into the hierarchy:

* a
ca
* b
** ba
cba
** bb
*** bba
cbba
*** bbb
cbbb
*** bbc
cbbc
** bc
cbc
* c
cc

Then C-c / / cbbb:

* a...
* b...
** bb...
*** cbbb
cbbb
...
* c...

It's just the top-level that gets a special treatment.

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Ihor Radchenko]

Carlos Pita <carlosjosepita2@gmail.com> writes:

> I see no reason why a match inside b2 will hide b1 and b3 but not a
> and c (I'm referring to the headings, not the contents).

I think you misread the docstring for org-show-context-detail:

>> Alist between context and visibility span when **revealing** a ;; <-- revealing!
>> location.
>> 
>> Some actions may move point into invisible
>> locations.  As a consequence, Org always exposes a neighborhood
>> around point.  How much is shown depends on the initial action,
>> or context.

This variable does not control how much text is hidden, but rather how
much context is revealed around a folded text.

Your misunderstanding may come from the fact that you are interested in
org-occur, which first folds everything inside lowest-level headings.

Consider agenda views. The relevant default value in
org-show-context-detail is (agenda . local). If you press <RET> on an
agenda entry which is located _inside folded subtree_, only the headline
will be revealed. However, if e.g. the headline and it's siblings are
already revealed, org-show-context-detail will not cause siblings to be
hidden.

Best,
Ihor

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Carlos Pita]

Hi Ihor,

On Thu, Nov 18, 2021 at 7:28 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Carlos Pita <carlosjosepita2@gmail.com> writes:
>
> > I see no reason why a match inside b2 will hide b1 and b3 but not a
> > and c (I'm referring to the headings, not the contents).
>
> I think you misread the docstring for org-show-context-detail:

Sorry, I don't concur here.

This is in the docstring I'm interested in (org-occur):

    The tree will show the lines where the regexp matches, and any other context
    defined in ‘org-show-context-detail’, which see.

It might be that org-show-context-detail is used for org-reveal, but
this says it's also used for org-occur, which makes sense.

Besides, there is

    occur-tree     when using the command ‘org-occur’ (‘C-c / /’)

as a context for org-show-context-detail and it indeed makes a
difference in what is shown and what is hidden.

> This variable does not control how much text is hidden, but rather how
> much context is revealed around a folded text.

> Your misunderstanding may come from the fact that you are interested in
> org-occur, which first folds everything inside lowest-level headings.

It's not very relevant to my concern if things are first hidden and
then revealed, because that will just change my question to why
top-levels are not hidden to begin with. Again:

    The tree will show the lines where the regexp matches, and any other context
    defined in ‘org-show-context-detail’, which see.

The top-level headings in my example don't match the regexp and are
not part of the context that org-show-context-detail should reveal
with my current configuration. I cannot help concluding that this fact
contradicts the documentation.

Moreover, even if a escape clause were found in the documentation, the
examples I've presented show a behavior that is clearly different for
the top-level compared to any other level and that alone calls for
some kind of explanation IMO, as it is now it seems arbitrary, perhaps
there is an obvious reason for this that I'm failing to grasp.

> Consider agenda views. The relevant default value in

I indeed considered the agenda, but I prefer using a sparse tree. I
have a file with a large number of brief notes in top level headings
and it's useful to see the expanded matching notes, the behaviour of
org-occur is ideal in this regard, instead the agenda only shows the
headings and even in follow mode it's more cumbersome than merely
pressing M-g n/p directly in the target buffer. The problem I have now
is that hundreds of unmatching headings are still shown just because
they happen to be at the top-level. I see I could demote them or just
use the agenda, it's no big deal after all. But because of all the
reasons above I judge that the current behavior is not right, that's
the reason for my report. Nevertheless, thank you for your
suggestions, as always.

Best regards,
Carlos

Re: [BUG] org-occur doesn't hide unmatching top-levels [9.5 (release_9.5-225-g494c20.dirty @ /Users/carlos/Install/Source/org-mode/lisp/)]

[Ihor Radchenko]

Carlos Pita <carlosjosepita2@gmail.com> writes:

>> I think you misread the docstring for org-show-context-detail:
>
> Sorry, I don't concur here.
>
> This is in the docstring I'm interested in (org-occur):

I agree that org-occur could have a better docstring.

> It's not very relevant to my concern if things are first hidden and
> then revealed, because that will just change my question to why
> top-levels are not hidden to begin with. Again:
>
>     The tree will show the lines where the regexp matches, and any other context
>     defined in ‘org-show-context-detail’, which see.
>
> The top-level headings in my example don't match the regexp and are
> not part of the context that org-show-context-detail should reveal
> with my current configuration. I cannot help concluding that this fact
> contradicts the documentation.

Currently, Org never ever hides headlines without parent and the top
section of the Org buffers.

Note that org-occur's docstring never explicitly states that all
non-matching lines will be hidden, just that matching lines will be
revealed and that the tree will be "compact". Though I understand your
confusion and I do agree that we should either clarify the docstring or
change org-occur to hide non-matching subtrees.

Changing org-occur is actually trivial (just one LOC), but this would be
a major change and can catch old Org users by surprise. I would prefer
to hear other's opinions before pushing such patch upsteam.

>> Consider agenda views. The relevant default value in
>
> I indeed considered the agenda, but I prefer using a sparse tree. I
> have a file with a large number of brief notes in top level headings
> and it's useful to see the expanded matching notes, the behaviour of
> org-occur is ideal in this regard, instead the agenda only shows the
> headings and even in follow mode it's more cumbersome than merely
> pressing M-g n/p directly in the target buffer.

FYI, there is org-agenda-entry-text-mode ("E" in agenda buffers).

Best,
Ihor