[org-save-all-org-buffers] Saving is not reliable?

[org-save-all-org-buffers] Saving is not reliable?

[Mikhail Skorzhisnkii]

[-- Attachment #1: Type: text/html, Size: 4663 bytes --]

[-- Attachment #2: Type: text/plain, Size: 2070 bytes --]

Hello forum,

I start noticing some time ago that saving org-mode buffers works
unreliably in my setup. Most of the time I am using function
`org-save-all-org-buffers' from core org. Unfortunately I don't have a
good reproduction scenarios of this bug. In fact I don't have
reproduction scenario at all. It just happens sometimes: I am sure I
saved all org buffers I had, then restart emacs and then I see that some
changes were not actually saved.

Possibly there is something wrong in my customisations. But without a
reproduction scenario, I don't see a way to prove it. However, after I
made a tiny change to the function, I stopped seeing these problems at
all. Here is the fix I have applied:

,----
| diff --git a/lisp/org.elf b/lisp/org.el
| index df3f377f6..448dc4a88 100644
| --- a/lisp/org.el
| +++ b/lisp/org.el
| @@ -15229,7 +15229,9 @@ The value is a list, with zero or more of the symbols `effort', `appt',
|    "Save all Org buffers without user confirmation."
|    (interactive)
|    (message "Saving all Org buffers...")
| -  (save-some-buffers t (lambda () (derived-mode-p 'org-mode)))
| +  (save-some-buffers t (lambda ()
| +                         (and (derived-mode-p 'org-mode)
| +                              (not (buffer-base-buffer)))))
|    (when (featurep 'org-id) (org-id-locations-save))
|    (message "Saving all Org buffers... done"))
`----

My theory was that `save-some-buffers' may work unreliably with indirect
buffers, so I've excluded them from the saving. Again, I have tried to
prove it by using indirect buffer and saving it instead of base buffer.
But it worked without a problem. So even if my theory is correct, bug
not reproducing every time. Nevertheless I am having this change already
for two weeks and I don't have reproduction of this bug. Previously I've
noticed loosing data every day or so.

I don't suggest to apply this patch, but may be someone have\had the
same problem or have a deeper insight how indirect buffers work and why
my fix may be a working solution?

      Kind regards,
      Mikhail Skorzhinskii

Re: [org-save-all-org-buffers] Saving is not reliable?

[Kyle Meyer]

Mikhail Skorzhisnkii writes:

> Hello forum,
>
> I start noticing some time ago that saving org-mode buffers works
> unreliably in my setup. Most of the time I am using function
> `org-save-all-org-buffers' from core org.

Unreliable in that some Org buffers are left in a modified state?

[...]
> Possibly there is something wrong in my customisations. But without a
> reproduction scenario, I don't see a way to prove it. However, after I
> made a tiny change to the function, I stopped seeing these problems at
> all. Here is the fix I have applied:
>
> ,----
> | diff --git a/lisp/org.elf b/lisp/org.el
> | index df3f377f6..448dc4a88 100644
> | --- a/lisp/org.el
> | +++ b/lisp/org.el
> | @@ -15229,7 +15229,9 @@ The value is a list, with zero or more of the symbols `effort', `appt',
> |    "Save all Org buffers without user confirmation."
> |    (interactive)
> |    (message "Saving all Org buffers...")
> | -  (save-some-buffers t (lambda () (derived-mode-p 'org-mode)))
> | +  (save-some-buffers t (lambda ()
> | +                         (and (derived-mode-p 'org-mode)
> | +                              (not (buffer-base-buffer)))))
> |    (when (featurep 'org-id) (org-id-locations-save))
> |    (message "Saving all Org buffers... done"))
> `----
>
> My theory was that `save-some-buffers' may work unreliably with indirect
> buffers, so I've excluded them from the saving. Again, I have tried to
> prove it by using indirect buffer and saving it instead of base buffer.
> But it worked without a problem. So even if my theory is correct, bug
> not reproducing every time. Nevertheless I am having this change already
> for two weeks and I don't have reproduction of this bug. Previously I've
> noticed loosing data every day or so.

Hmm, I may be completely missing something, but for what it's worth, I'd
be surprised if indirect buffers are the culprit.  When you save an
indirect buffer directly, it should just save the base buffer.  And in
any case, save-some-buffers should skip indirect buffers.  Here is the
relevant handling from save-some-buffers, with the key line marked:

    (setq files-done
          (map-y-or-n-p
           (lambda (buffer)
             (and (buffer-live-p buffer)
                  (buffer-modified-p buffer)
                  (not (buffer-base-buffer buffer))  ; <- skip indirect buffers
                  (or
                   (buffer-file-name buffer)
                   (with-current-buffer buffer
                     (or (eq buffer-offer-save 'always)
                         (and pred buffer-offer-save
                              (> (buffer-size) 0)))))
                  (or (not (functionp pred))
                      (with-current-buffer buffer (funcall pred)))
                  (if arg
                      t
                    (setq queried t)
                    (if (buffer-file-name buffer)
                        (format "Save file %s? "
                                (buffer-file-name buffer))
                      (format "Save buffer %s? "
                              (buffer-name buffer))))))
           (lambda (buffer)
             (with-current-buffer buffer
               (save-buffer)))
           (buffer-list)
           '("buffer" "buffers" "save")
           save-some-buffers-action-alist))

Re: [org-save-all-org-buffers] Saving is not reliable?

[Mikhail Skorzhisnkii]

Hi, Kyle,

Thank you for finding time to take a look at this. I have 
experienced data loss once again, so you're right. This is not 
indirect buffers, i.e. my fix is not working. I was just lucky.

Fortunately I managed to capture the moment in emacs when it 
happens. It's kind of reproduction scenario. Basically I need to 
modify buffer from search-type agenda. For example, search for 
tags and then change TODO state or add a note. Then use 
`org-save-all-org-buffers`. Function will report that it is done 
its job and buffers will be marked as non-modified. But in fact 
they are modified and unsaved.

I can't reproduce this in emacs without my configuration (i.e. 
only emacs and most recent org-mode). So it must be something that 
interfere with buffer "modified" state. I guess I need to review 
every hook related to buffer saving. I think (but unsure) it is 
start happening when I have start enabling follow-mode in agenda 
from start up.

For the time being I have applied advice from this stack overflow 
question: 
https://stackoverflow.com/questions/3215866/how-to-force-emacs-to-save-even-if-it-thinks-no-changes-need-to-be-saved

  (defadvice save-buffer (before save-buffer-always activate)
    "always save buffer"
    (set-buffer-modified-p t))

It looks like it helps me. I'll report back to this thread when I 
find the offender. I guess I'm calling (set-buffer-modified-p nil) 
somewhere unknowingly.

Mikhail Skorzhinskii

Kyle Meyer <kyle@kyleam.com> writes:

> Mikhail Skorzhisnkii writes:
>
>> Hello forum,
>>
>> I start noticing some time ago that saving org-mode buffers 
>> works
>> unreliably in my setup. Most of the time I am using function
>> `org-save-all-org-buffers' from core org.
>
> Unreliable in that some Org buffers are left in a modified 
> state?
>
> [...]
>> Possibly there is something wrong in my customisations. But 
>> without a
>> reproduction scenario, I don't see a way to prove it. However, 
>> after I
>> made a tiny change to the function, I stopped seeing these 
>> problems at
>> all. Here is the fix I have applied:
>>
>> ,----
>> | diff --git a/lisp/org.elf b/lisp/org.el
>> | index df3f377f6..448dc4a88 100644
>> | --- a/lisp/org.el
>> | +++ b/lisp/org.el
>> | @@ -15229,7 +15229,9 @@ The value is a list, with zero or 
>> more of the symbols `effort', `appt',
>> |    "Save all Org buffers without user confirmation."
>> |    (interactive)
>> |    (message "Saving all Org buffers...")
>> | -  (save-some-buffers t (lambda () (derived-mode-p 
>> 'org-mode)))
>> | +  (save-some-buffers t (lambda ()
>> | +                         (and (derived-mode-p 'org-mode)
>> | +                              (not (buffer-base-buffer)))))
>> |    (when (featurep 'org-id) (org-id-locations-save))
>> |    (message "Saving all Org buffers... done"))
>> `----
>>
>> My theory was that `save-some-buffers' may work unreliably with 
>> indirect
>> buffers, so I've excluded them from the saving. Again, I have 
>> tried to
>> prove it by using indirect buffer and saving it instead of base 
>> buffer.
>> But it worked without a problem. So even if my theory is 
>> correct, bug
>> not reproducing every time. Nevertheless I am having this 
>> change already
>> for two weeks and I don't have reproduction of this bug. 
>> Previously I've
>> noticed loosing data every day or so.
>
> Hmm, I may be completely missing something, but for what it's 
> worth, I'd
> be surprised if indirect buffers are the culprit.  When you save 
> an
> indirect buffer directly, it should just save the base buffer. 
> And in
> any case, save-some-buffers should skip indirect buffers.  Here 
> is the
> relevant handling from save-some-buffers, with the key line 
> marked:
>
>     (setq files-done
>           (map-y-or-n-p
>            (lambda (buffer)
>              (and (buffer-live-p buffer)
>                   (buffer-modified-p buffer)
>                   (not (buffer-base-buffer buffer))  ; <- skip 
>                   indirect buffers
>                   (or
>                    (buffer-file-name buffer)
>                    (with-current-buffer buffer
>                      (or (eq buffer-offer-save 'always)
>                          (and pred buffer-offer-save
>                               (> (buffer-size) 0)))))
>                   (or (not (functionp pred))
>                       (with-current-buffer buffer (funcall 
>                       pred)))
>                   (if arg
>                       t
>                     (setq queried t)
>                     (if (buffer-file-name buffer)
>                         (format "Save file %s? "
>                                 (buffer-file-name buffer))
>                       (format "Save buffer %s? "
>                               (buffer-name buffer))))))
>            (lambda (buffer)
>              (with-current-buffer buffer
>                (save-buffer)))
>            (buffer-list)
>            '("buffer" "buffers" "save")
>            save-some-buffers-action-alist))


--
---
Mikhail Skorzhinskii

Re: [org-save-all-org-buffers] Saving is not reliable?

[Eric S Fraga]

On Wednesday,  9 Dec 2020 at 11:16, Mikhail Skorzhisnkii wrote:
> It's kind of reproduction scenario. Basically I need to
> modify buffer from search-type agenda. 

In the past, anecdotally I have seen something similar: adjust the
scheduled date for an entry via the agenda view and ask to save all org
buffers.  The change to the scheduled date is sometimes forgotten.  I
haven't tried with emacs -Q so it could, as in Mikhail's case, be
configuration dependent and it's also not entirely reproducible (i.e. it
sometimes happens, sometimes doesn't).

But I've not seen this happen recently so maybe it was a bug along the
way.  Sorry for vagueness but I thought I'd chime in just in case it
helps.
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-160-g7c8dce

Re: [org-save-all-org-buffers] Saving is not reliable?

[Samuel Wales]

an undo-boundary bug can make something unexpected get undone as part
of a batch or make an org operation require two undos.  the agenda is
one place where these bugs have existed.


On 12/9/20, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> On Wednesday,  9 Dec 2020 at 11:16, Mikhail Skorzhisnkii wrote:
>> It's kind of reproduction scenario. Basically I need to
>> modify buffer from search-type agenda.
>
> In the past, anecdotally I have seen something similar: adjust the
> scheduled date for an entry via the agenda view and ask to save all org
> buffers.  The change to the scheduled date is sometimes forgotten.  I
> haven't tried with emacs -Q so it could, as in Mikhail's case, be
> configuration dependent and it's also not entirely reproducible (i.e. it
> sometimes happens, sometimes doesn't).
>
> But I've not seen this happen recently so maybe it was a bug along the
> way.  Sorry for vagueness but I thought I'd chime in just in case it
> helps.
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-160-g7c8dce
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: [org-save-all-org-buffers] Saving is not reliable?

[Mikhail Skorzhisnkii]

I found the bug. The bug was mostly my own creation. There was a 
small (but very destructive) bug in doom emacs distribution. Fixed 
that bug and sent them a pull requested: 
https://github.com/hlissner/doom-emacs/pull/4424

Basically I made a small snippet for agenda that makes side window 
of agenda staticly sized, really small and includes only very 
small portion of context related to agenda entry. And to correctly 
possition it in frame, I've used wonderful doom emacs popup 
syntax:

  (set-popup-rule! "^*org-agenda-side\*"
    :side 'top
    :size 0.15
    :autosave t
    :quit t)

Turns out :autosave ignores indirect buffers and sets 
buffer-modified-p state to nil. Hope this is the end of my quest, 
My previous solution was also very clunky.

Mikhail Skorzhinskii


Samuel Wales <samologist@gmail.com> writes:

> an undo-boundary bug can make something unexpected get undone as 
> part
> of a batch or make an org operation require two undos.  the 
> agenda is
> one place where these bugs have existed.
>
>
> On 12/9/20, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>> On Wednesday,  9 Dec 2020 at 11:16, Mikhail Skorzhisnkii wrote:
>>> It's kind of reproduction scenario. Basically I need to
>>> modify buffer from search-type agenda.
>>
>> In the past, anecdotally I have seen something similar: adjust 
>> the
>> scheduled date for an entry via the agenda view and ask to save 
>> all org
>> buffers.  The change to the scheduled date is sometimes 
>> forgotten.  I
>> haven't tried with emacs -Q so it could, as in Mikhail's case, 
>> be
>> configuration dependent and it's also not entirely reproducible 
>> (i.e. it
>> sometimes happens, sometimes doesn't).
>>
>> But I've not seen this happen recently so maybe it was a bug 
>> along the
>> way.  Sorry for vagueness but I thought I'd chime in just in 
>> case it
>> helps.
>> --
>> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-160-g7c8dce
>>
>>


--
---
Mikhail Skorzhinskii