org-caldav can't find org-prepare-agenda-buffers

org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

As of today, org-cadav-syn fails with

--8<---------------cut here---------------start------------->8---
org-export-icalendar: Symbol's function definition is void: org-prepare-agenda-buffers
--8<---------------cut here---------------end--------------->8---

Actually, I can't find this function but org-agenda-prepare-buffers
exist. Are there two different functions or is there a confusion
somewhere ?

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi Julien,

Julien Cubizolles <j.cubizolles@free.fr> writes:

> As of today, org-cadav-syn fails with
>
> org-export-icalendar: Symbol's function definition is void: org-prepare-agenda-buffers
>
> Actually, I can't find this function but org-agenda-prepare-buffers
> exist. Are there two different functions or is there a confusion
> somewhere ?

The function has been renamed a while ago.

`org-agenda-prepare-buffers' is the correct name.

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Bastien writes:
> Hi Julien,
>
> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> As of today, org-cadav-syn fails with
>>
>> org-export-icalendar: Symbol's function definition is void:
>> org-prepare-agenda-buffers
>>
>> Actually, I can't find this function but org-agenda-prepare-buffers
>> exist. Are there two different functions or is there a confusion
>> somewhere ?
>
> The function has been renamed a while ago.
>
> `org-agenda-prepare-buffers' is the correct name.

org-caldav does not call this function. It however requires
org-icalendar, and that was renamed to ox-icalendar in org git. So I
guess it pulls org-icalendar from the Org that is included with Emacs,
which calls the obsoleted function.

I don't follow org-development very closely. I realize there's a new
exporter, but renaming the exporters in this way is asking for
trouble. In this case, we've actually been lucky that an error like the
above is thrown; much more subtle things can happen when new and old Org
functions interact.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi David,

David Engster <deng@randomsample.de> writes:

> org-caldav does not call this function. It however requires
> org-icalendar, and that was renamed to ox-icalendar in org git. So I
> guess it pulls org-icalendar from the Org that is included with Emacs,
> which calls the obsoleted function.

Can you fix this by requiring ox-icalendar.el when it's available,
org-icalendar.el otherwise?

Thanks!

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Bastien writes:
> Hi David,
>
> David Engster <deng@randomsample.de> writes:
>
>> org-caldav does not call this function. It however requires
>> org-icalendar, and that was renamed to ox-icalendar in org git. So I
>> guess it pulls org-icalendar from the Org that is included with Emacs,
>> which calls the obsoleted function.
>
> Can you fix this by requiring ox-icalendar.el when it's available,
> org-icalendar.el otherwise?

Of course I can fix this. But I hope you realize that any third-party
code out there that requires an exporter will load the old one from
Emacs proper. I'm wondering why you felt the need to rename them. If the
new exporters are compatible with the old ones, why not keep the names?
This would also avoid that the provided feature differs from the used
name prefix (ox-icalendar != org-icalendar).

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi David,

David Engster <deng@randomsample.de> writes:

> Of course I can fix this. But I hope you realize that any third-party
> code out there that requires an exporter will load the old one from
> Emacs proper. 

Yes, I'm well aware of this.  The change now lives in the master
branch, and will happen when we release 8.0, hopefully in a not so
distant future.

We will document all incompatible changes in the release notes, as we
usually do.  I expect third-party maintainers to read these notes.

> I'm wondering why you felt the need to rename them. If the
> new exporters are compatible with the old ones, why not keep the names?
> This would also avoid that the provided feature differs from the used
> name prefix (ox-icalendar != org-icalendar).

There are several good reasons for this:

1) conflicting library names: we now have org-man.el (for links to man
   pages) and ox-man.el (for exporting);

2) using the dedicated prefix ox- makes it clear that the library is
   an export backend, the same way that the ob- prefix makes it clear
   it is to support a language for Org Babel.

In general, the change is incompatible for third-part libraries by is 
clearly useful for future maintainance, so the trade-off was in favor
of making it, and 8.0 is a good time for it.

Best,

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Bastien writes:
> David Engster <deng@randomsample.de> writes:
>
>> Of course I can fix this. But I hope you realize that any third-party
>> code out there that requires an exporter will load the old one from
>> Emacs proper. 
>
> Yes, I'm well aware of this.  The change now lives in the master
> branch, and will happen when we release 8.0, hopefully in a not so
> distant future.
>
> We will document all incompatible changes in the release notes, as we
> usually do.  I expect third-party maintainers to read these notes.

Well, I don't expect it.

>> I'm wondering why you felt the need to rename them. If the
>> new exporters are compatible with the old ones, why not keep the names?
>> This would also avoid that the provided feature differs from the used
>> name prefix (ox-icalendar != org-icalendar).
>
> There are several good reasons for this:
>
> 1) conflicting library names: we now have org-man.el (for links to man
>    pages) and ox-man.el (for exporting);
>
> 2) using the dedicated prefix ox- makes it clear that the library is
>    an export backend, the same way that the ob- prefix makes it clear
>    it is to support a language for Org Babel.
>
> In general, the change is incompatible for third-part libraries by is 
> clearly useful for future maintainance, so the trade-off was in favor
> of making it, and 8.0 is a good time for it.

I'm afraid I remain unconvinced. 1) is just one name clash, so just
rename one of it. Also, like all the other ox-* packages now, ox-man
uses 'org-man' as a prefix for its names; what will 'org-man' use, then?

2) is nice, but IMO not a good enough reason to break compatibility in
such a major way.

Anyway, you've made a decision and did the rename. Let's just agree to
disagree here.

The most serious issue is that things will often seem to work because
old exporters are pulled in from Emacs, possibly *very* old
exporters. They might work, or they might fail for various reasons,
making it difficult for users and developers to see what went wrong.
Just look at the other bug report by Karl Voit from yesterday; I guess
it's the same issue.

So if you change your API in an incompatible way (and packages names are
part of that), you should at least throw a clear error when the old API
is used, and tell the user what happened and how it can be fixed. Hence
I would suggest to use something like

(eval-after-load "org-icalendar"
  '(error "The old org-icalendar exporter is deprecated; use ox-icalendar instead."))

An alternative would be to remove the bundled Org from load-path when a
newer version is loaded. We do that with CEDET, but it is difficult to
do right (because of autoloading, for instance), so I think the
eval-after-load hack is better.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Achim Gratz]

David Engster writes:
> An alternative would be to remove the bundled Org from load-path when a
> newer version is loaded. We do that with CEDET, but it is difficult to
> do right (because of autoloading, for instance), so I think the
> eval-after-load hack is better.

That part is actually relatively easy, I have posted code to do that
here.  The part that gives me headaches is the defcustom definitions,
which are nowhere near as easy to defeat and provide yet another way of
"autoloading" stuff in Emacs.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Achim Gratz writes:
> David Engster writes:
>> An alternative would be to remove the bundled Org from load-path when a
>> newer version is loaded. We do that with CEDET, but it is difficult to
>> do right (because of autoloading, for instance), so I think the
>> eval-after-load hack is better.
>
> That part is actually relatively easy, I have posted code to do that
> here.  The part that gives me headaches is the defcustom definitions,
> which are nowhere near as easy to defeat and provide yet another way of
> "autoloading" stuff in Emacs.

You mean this cus-load thingie? CEDET is actually excluded from that, so
we don't have to deal with it. But wouldn't it be enough to remove all
properties beginning with 'org-' from custom-loads?

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Achim Gratz]

David Engster writes:
> You mean this cus-load thingie? CEDET is actually excluded from that, so
> we don't have to deal with it. But wouldn't it be enough to remove all
> properties beginning with 'org-' from custom-loads?

First you have to find all symbols, then remove the property and then
there are one or two undocumented internal variables that I don't really
know yet what they're used for.  I don't know if they're any functions
for doing this.  I'll get around to it eventually.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi David,

David Engster <deng@randomsample.de> writes:

> The most serious issue is that things will often seem to work because
> old exporters are pulled in from Emacs, possibly *very* old
> exporters.

I've added (provide 'org-icalendar) to ox-icalendar.el so that
a user will load the correct file instead of the old file when
ox-icalendar.el takes precedence over org-icalendar.el in the
load-path.  Not the most elegant solution, I agree.

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi David,

David Engster <deng@randomsample.de> writes:

> (eval-after-load "org-icalendar"
>   '(error "The old org-icalendar exporter is deprecated; use ox-icalendar instead."))

I'm not sure about this one: where are you suggesting to add this?
In org.el?  Will the users get the warning if org-icalendar.el has
been loaded from a previous install?

Thanks,

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Bastien writes:
> Hi David,
>
> David Engster <deng@randomsample.de> writes:
>
>> The most serious issue is that things will often seem to work because
>> old exporters are pulled in from Emacs, possibly *very* old
>> exporters.
>
> I've added (provide 'org-icalendar) to ox-icalendar.el so that
> a user will load the correct file instead of the old file when
> ox-icalendar.el takes precedence over org-icalendar.el in the
> load-path.  Not the most elegant solution, I agree.

Did you actually try that? How should Emacs possibly know that the file
ox-icalendar provides the feature org-icalendar? This will only work if
ox-icalendar is already loaded.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Bastien writes:
> Hi David,
>
> David Engster <deng@randomsample.de> writes:
>
>> (eval-after-load "org-icalendar"
>>   '(error "The old org-icalendar exporter is deprecated; use
>> ox-icalendar instead."))
>
> I'm not sure about this one: where are you suggesting to add this?
> In org.el?

Your call. Anywhere where it's guaranteed to be loaded upon Org startup.

>  Will the users get the warning if org-icalendar.el has been loaded
> from a previous install?

The user will get the error as soon as org-icalendar is loaded (from
wherever). org-icalendar *will* be loaded then, and the user can ignore
the error and continue to use it if he likes, but at least he got a
clear error that this is not a supported.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Hi David,

David Engster <deng@randomsample.de> writes:

> Did you actually try that? How should Emacs possibly know that the file
> ox-icalendar provides the feature org-icalendar? This will only work if
> ox-icalendar is already loaded.

Of course, you're right.  I reverted the commit.

So the problems stay.  For third-party libraries developers,
we cannot do anything else now than to ask them to update their
code.  For the problem of Emacs autoloaded functions, org.el
provides (load "org-loaddefs.el t t t) which should load
the correct autoloads from the correct files...  but that's
unstable.

It seems the ox- prefix is a bad idea, you're right.
I'll think about it again.

Thanks,

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 1439 bytes --]

Hi Bastien, 
I didn't follow this thread in detail.  But shouldn't it be enough to symlink e.g.  org-icalendar against ox-icalendar. As far as I understood emacs would prioritize those local symlinks over the system wide installation.  This would be a temporary solution until a new emacs release.  
Actually,  under Linux, this is a pretty common way to bend dependencies towards the newest version of a lib. 
Not sure for windows users.  

Instead of a simple symlink, the current dev head could have wrappers for those "old" files which bend the calls to the new files and issue a warning.  
That would help to identify 3 party code which needs some rework. 

Torsten 

Bastien <bzg@altern.org> wrote:

>Hi David,
>
>David Engster <deng@randomsample.de> writes:
>
>> Did you actually try that? How should Emacs possibly know that the
>file
>> ox-icalendar provides the feature org-icalendar? This will only work
>if
>> ox-icalendar is already loaded.
>
>Of course, you're right.  I reverted the commit.
>
>So the problems stay.  For third-party libraries developers,
>we cannot do anything else now than to ask them to update their
>code.  For the problem of Emacs autoloaded functions, org.el
>provides (load "org-loaddefs.el t t t) which should load
>the correct autoloads from the correct files...  but that's
>unstable.
>
>It seems the ox- prefix is a bad idea, you're right.
>I'll think about it again.
>
>Thanks,
>
>-- 
> Bastien

[-- Attachment #2: Type: text/html, Size: 2042 bytes --]

Re: org-caldav can't find org-prepare-agenda-buffers

[Achim Gratz]

Bastien writes:
> It seems the ox- prefix is a bad idea, you're right.
> I'll think about it again.

Please, no.  The prefix is good, you would just need to obsolete all
previously exported interfaces (autoloads and defcustoms) to this code
to either give a warning or silently "do the right thing" where
possible.  Of course this code should not be in Emacs proper.  I have
code that kills the defcustoms from old Emacs. I need to figure out how
to replace it with something sensible and also how to kill the autoloads
that belong to older versions of Org (I think I know how to do it, I've
just not tested it).  I hope to put this together in the next week or
so, then it will be possible to nuke all traces fr4om an old Org and
then start from a clean slate with a new Org installation.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: org-caldav can't find org-prepare-agenda-buffers

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> I hope to put this together in the next week or
> so, then it will be possible to nuke all traces fr4om an old Org and
> then start from a clean slate with a new Org installation.

Okay, thanks.  Let us know how it goes,

-- 
 Bastien

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Torsten Wagner writes:
> I didn't follow this thread in detail. But shouldn't it be enough to symlink
> e.g. org-icalendar against ox-icalendar. As far as I understood emacs would
> prioritize those local symlinks over the system wide installation. This would
> be a temporary solution until a new emacs release.

Why temporary? What about people installing Org 8.x on older Emacsen?

> Actually, under Linux, this is a pretty common way to bend
> dependencies towards the newest version of a lib.  Not sure for
> windows users.

Won't work on MS-DOS, and on Windows it is highly problematic for
various reasons (they're called "junctions" there; you need
administrator privileges to create them, and the upcoming Emacs 24.3
will be the first version to even support them).

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

Hi David,

Well it would be a temporary solution for two reasons.
1. New emacs releases would come with the new org-files.
2. All third-party code might by time move to the new files.
Thus, I thought this is a (maybe on a long perspective) temporary solution.

Other non-unix like OSes:
In that case, a small wrapper file, which replaces the old files might be
the best solution, since it would work under all OSes. This wrapper could
call the right functions within the new file and issues a warning/error/log
that the call is deprecated. Hence it gives third-party maintainers (or
anyone who jumps in) enough time to change the code.


On 3 March 2013 13:26, David Engster <deng@randomsample.de> wrote:

> Torsten Wagner writes:
> > I didn't follow this thread in detail. But shouldn't it be enough to
> symlink
> > e.g. org-icalendar against ox-icalendar. As far as I understood emacs
> would
> > prioritize those local symlinks over the system wide installation. This
> would
> > be a temporary solution until a new emacs release.
>
> Why temporary? What about people installing Org 8.x on older Emacsen?
>
> > Actually, under Linux, this is a pretty common way to bend
> > dependencies towards the newest version of a lib.  Not sure for
> > windows users.
>
> Won't work on MS-DOS, and on Windows it is highly problematic for
> various reasons (they're called "junctions" there; you need
> administrator privileges to create them, and the upcoming Emacs 24.3
> will be the first version to even support them).
>
> -David
>

[-- Attachment #2: Type: text/html, Size: 2119 bytes --]

Re: org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

Bastien <bzg@altern.org> writes:

> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> I hope to put this together in the next week or
>> so, then it will be possible to nuke all traces fr4om an old Org and
>> then start from a clean slate with a new Org installation.
>
> Okay, thanks.  Let us know how it goes,

What temporary (as in every user fixing it locally) solution would you
recommend to get org-caldav running in the mean time ?

Julien.

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Julien Cubizolles writes:
> Bastien <bzg@altern.org> writes:
>
>> Achim Gratz <Stromeko@nexgo.de> writes:
>>
>>> I hope to put this together in the next week or
>>> so, then it will be possible to nuke all traces fr4om an old Org and
>>> then start from a clean slate with a new Org installation.
>>
>> Okay, thanks.  Let us know how it goes,
>
> What temporary (as in every user fixing it locally) solution would you
> recommend to get org-caldav running in the mean time ?

I just pushed a change which should fix this (if the new exporter is
compatible; I don't use Org from git).

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Nicolas Goaziou]

Hello,

David Engster <deng@randomsample.de> writes:

> Julien Cubizolles writes:
>> Bastien <bzg@altern.org> writes:
>>
>>> Achim Gratz <Stromeko@nexgo.de> writes:
>>>
>>>> I hope to put this together in the next week or
>>>> so, then it will be possible to nuke all traces fr4om an old Org and
>>>> then start from a clean slate with a new Org installation.
>>>
>>> Okay, thanks.  Let us know how it goes,
>>
>> What temporary (as in every user fixing it locally) solution would you
>> recommend to get org-caldav running in the mean time ?
>
> I just pushed a change which should fix this (if the new exporter is
> compatible; I don't use Org from git).

Interactive functions are usually not compatible between old export
back-ends and new ones.

What do you need from org-icalendar.el?


Regards,

-- 
Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Nicolas Goaziou writes:
> Hello,
>
> David Engster <deng@randomsample.de> writes:
>
>> Julien Cubizolles writes:
>>> Bastien <bzg@altern.org> writes:
>>>
>>>> Achim Gratz <Stromeko@nexgo.de> writes:
>>>>
>>>>> I hope to put this together in the next week or
>>>>> so, then it will be possible to nuke all traces fr4om an old Org and
>>>>> then start from a clean slate with a new Org installation.
>>>>
>>>> Okay, thanks.  Let us know how it goes,
>>>
>>> What temporary (as in every user fixing it locally) solution would you
>>> recommend to get org-caldav running in the mean time ?
>>
>> I just pushed a change which should fix this (if the new exporter is
>> compatible; I don't use Org from git).
>
> Interactive functions are usually not compatible between old export
> back-ends and new ones.
>
> What do you need from org-icalendar.el?

Only org-export-icalendar. As long as that one is producing the same
output, everything should be fine.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

David Engster <deng@randomsample.de> writes:


> I just pushed a change which should fix this (if the new exporter is
> compatible; 

It doesn't seem to be, I get 

--8<---------------cut here---------------start------------->8---
apply: Symbol's function definition is void: org-export-icalendar
--8<---------------cut here---------------end--------------->8---

when running org-caldav-sync, which is weird since org-export-icalendar
is present in both org-icalendar and ox-icalendar.

> I don't use Org from git).

I'm thinking about reverting to an older Org since org-caldav is more
important to me than the new exporter. Does it work with the latest
stable release (7.9.3f) ?

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Julien Cubizolles writes:
> David Engster <deng@randomsample.de> writes:
>> I just pushed a change which should fix this (if the new exporter is
>> compatible; 
>
> It doesn't seem to be, I get 
>
> apply: Symbol's function definition is void: org-export-icalendar
>
> when running org-caldav-sync, which is weird since org-export-icalendar
> is present in both org-icalendar and ox-icalendar.

No, it's not defined in ox-icalendar. I guess it must be generated by
the new exporter first, probably through some additional require.

>> I don't use Org from git).
>
> I'm thinking about reverting to an older Org since org-caldav is more
> important to me than the new exporter. Does it work with the latest
> stable release (7.9.3f) ?

I'm still on 7.9.2, but I wouldn't know why it shouldn't work with that.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Nicolas Goaziou]

David Engster <deng@randomsample.de> writes:

>> What do you need from org-icalendar.el?
>
> Only org-export-icalendar. As long as that one is producing the same
> output, everything should be fine.

Unfortunately, there is no `org-export-icalendar' anymore.

There is:

  (org-icalendar--combine-files nil FILES)

which is equivalent to:

  (org-export-icalendar t FILES)

Also

  (org-export-icalendar nil FILES)

should be equivalent to something like:

  (dolist (file files)
    (find-file file)
    (org-mode)
    (org-icalendar-export-to-ics))


Regards,

-- 
Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

David Engster <deng@randomsample.de> writes:

> Julien Cubizolles writes:
>> David Engster <deng@randomsample.de> writes:
>>> I just pushed a change which should fix this (if the new exporter is
>>> compatible; 
>>
>> It doesn't seem to be, I get 
>>
>> apply: Symbol's function definition is void: org-export-icalendar
>>
>> when running org-caldav-sync, which is weird since org-export-icalendar
>> is present in both org-icalendar and ox-icalendar.
>
> No, it's not defined in ox-icalendar. I guess it must be generated by
> the new exporter first, probably through some additional require.

I have the following definition in ox-icalendar.el. 

--8<---------------cut here---------------start------------->8---
(defgroup org-export-icalendar nil
  "Options specific for iCalendar export back-end."
  :tag "Org Export iCalendar"
  :group 'org-export)
--8<---------------cut here---------------end--------------->8---

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Nicolas Goaziou writes:
> David Engster <deng@randomsample.de> writes:
>
>>> What do you need from org-icalendar.el?
>>
>> Only org-export-icalendar. As long as that one is producing the same
>> output, everything should be fine.
>
> Unfortunately, there is no `org-export-icalendar' anymore.
>
> There is:
>
>   (org-icalendar--combine-files nil FILES)
>
> which is equivalent to:
>
>   (org-export-icalendar t FILES)

Thanks, that's what I need.

After skimming through the new exporter, it also seems that
org-combined-agenda-icalendar-file was replaced with
org-icalendar-combined-agenda-file. Is that correct?

I'm also using org-icalendar-timezone, org-icalendar-store-UID, and
org-icalendar-date-time-format, but it seems those are still there.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Nicolas Goaziou]

David Engster <deng@randomsample.de> writes:

> After skimming through the new exporter, it also seems that
> org-combined-agenda-icalendar-file was replaced with
> org-icalendar-combined-agenda-file. Is that correct?
>
> I'm also using org-icalendar-timezone, org-icalendar-store-UID, and
> org-icalendar-date-time-format, but it seems those are still there.

Correct.


Regards,

-- 
Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Julien Cubizolles writes:
> David Engster <deng@randomsample.de> writes:
>> No, it's not defined in ox-icalendar. I guess it must be generated by
>> the new exporter first, probably through some additional require.
>
> I have the following definition in ox-icalendar.el. 
>
> (defgroup org-export-icalendar nil
>   "Options specific for iCalendar export back-end."
>   :tag "Org Export iCalendar"
>   :group 'org-export)

That's just defining a group for the customization system. It's not a
function.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Nicolas Goaziou writes:
> David Engster <deng@randomsample.de> writes:
>
>> After skimming through the new exporter, it also seems that
>> org-combined-agenda-icalendar-file was replaced with
>> org-icalendar-combined-agenda-file. Is that correct?
>>
>> I'm also using org-icalendar-timezone, org-icalendar-store-UID, and
>> org-icalendar-date-time-format, but it seems those are still there.
>
> Correct.

Thanks. I've now pushed a further fix which should hopefully make things
work with the new exporter. Julien, please let me know if it works.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

David Engster <deng@randomsample.de> writes:

> Thanks. I've now pushed a further fix which should hopefully make things
> work with the new exporter. Julien, please let me know if it works.

Sorry, it doesn't. 

I get the following when running org-caldav with the latest org-mode and
org-caldav from git.

--8<---------------cut here---------------start------------->8---
byte-code: Wrong type argument: number-or-marker-p, nil
--8<---------------cut here---------------end--------------->8---

Let me know what additional information you need to debug it, a
backtrace...

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

Hi, 

Same here. But the error is not in org-caldav, because it is triggered like this:

(org-icalendar--combine-files nil "~/Desktop/ECM.org")

where ~/Desktop/ECM.org is a file with nothing but one headline.

/v 

-- 
Vincent Beffara


On Thursday, March 7, 2013 at 10:13 , Julien Cubizolles wrote:

> David Engster <deng@randomsample.de (mailto:deng@randomsample.de)> writes:
> 
> > Thanks. I've now pushed a further fix which should hopefully make things
> > work with the new exporter. Julien, please let me know if it works.
> 
> 
> 
> Sorry, it doesn't. 
> 
> I get the following when running org-caldav with the latest org-mode and
> org-caldav from git.
> 
> --8<---------------cut here---------------start------------->8---
> byte-code: Wrong type argument: number-or-marker-p, nil
> --8<---------------cut here---------------end--------------->8---
> 
> Let me know what additional information you need to debug it, a
> backtrace...

Re: org-caldav can't find org-prepare-agenda-buffers

[Nicolas Goaziou]

Hello,

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Same here. But the error is not in org-caldav, because it is triggered like this:
>
> (org-icalendar--combine-files nil "~/Desktop/ECM.org")
>
> where ~/Desktop/ECM.org is a file with nothing but one headline.

I cannot reproduce it. Maybe a larger backtrace would help.


Regards,

-- 
Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

Here is an ECM.el file, run with emacs -Q triggers the crash: 

(custom-set-variables
'(org-icalendar-store-UID t)
)
(setq-default debug-on-error t)
(add-to-list 'load-path "~/.emacs.d/el-get/org-mode/lisp")
(require 'ox-icalendar)
(org-icalendar--combine-files nil "~/Desktop/ECM.org")


without org-icalendar-store-UID it works fine. Not sure the option is needed anymore, but at least the crash can be reproduced. BTW my org-mode is at commit 516f0df.

HTH,

/v

-- 
Vincent Beffara


On Thursday, March 7, 2013 at 11:54 , Nicolas Goaziou wrote:

> Hello,
> 
> Vincent Beffara <vbeffara@ens-lyon.fr (mailto:vbeffara@ens-lyon.fr)> writes:
> 
> > Same here. But the error is not in org-caldav, because it is triggered like this:
> > 
> > (org-icalendar--combine-files nil "~/Desktop/ECM.org (http://ECM.org)")
> > 
> > where ~/Desktop/ECM.org (http://ECM.org) is a file with nothing but one headline.
> 
> I cannot reproduce it. Maybe a larger backtrace would help.
> 
> 
> Regards,
> 
> -- 
> Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[Nicolas Goaziou]

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> Here is an ECM.el file, run with emacs -Q triggers the crash: 
>
> (custom-set-variables
> '(org-icalendar-store-UID t)
> )
> (setq-default debug-on-error t)
> (add-to-list 'load-path "~/.emacs.d/el-get/org-mode/lisp")
> (require 'ox-icalendar)
> (org-icalendar--combine-files nil "~/Desktop/ECM.org")
>
>
> without org-icalendar-store-UID it works fine. Not sure the option is
> needed anymore, but at least the crash can be reproduced. BTW my
> org-mode is at commit 516f0df.

The problem should be fixed in master.  Could anyone confirm it?

Thank you for reporting this problem.


Regards,

-- 
Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

It doesn't crash anymore, thanks!

OTOH when I ran org-caldav, it removed everything from the online calendar, and it won't put anything back there ... 

-- 
Vincent Beffara


On Thursday, March 7, 2013 at 13:56 , Nicolas Goaziou wrote:

> Vincent Beffara <vbeffara@ens-lyon.fr (mailto:vbeffara@ens-lyon.fr)> writes:
> 
> > Here is an ECM.el file, run with emacs -Q triggers the crash: 
> > 
> > (custom-set-variables
> > '(org-icalendar-store-UID t)
> > )
> > (setq-default debug-on-error t)
> > (add-to-list 'load-path "~/.emacs.d/el-get/org-mode/lisp")
> > (require 'ox-icalendar)
> > (org-icalendar--combine-files nil "~/Desktop/ECM.org (http://ECM.org)")
> > 
> > 
> > without org-icalendar-store-UID it works fine. Not sure the option is
> > needed anymore, but at least the crash can be reproduced. BTW my
> > org-mode is at commit 516f0df.
> 
> 
> 
> The problem should be fixed in master. Could anyone confirm it?
> 
> Thank you for reporting this problem.
> 
> 
> Regards,
> 
> -- 
> Nicolas Goaziou

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

> OTOH when I ran org-caldav, it removed everything from the online calendar, and it won't put anything back there ...

Mmmkay, something weird is happening. I create a headline with a date, and call org-caldav. An UID is created, and the file looks like this:

*** Gayet, Damien : Une majoration de l'espérance des nombres de Betti
:PROPERTIES:
:ID: BD783419-0D10-4B88-8540-730ACF03B42E
:END:

<2013-03-06 Wed 14:00>  

So far so good. But on trying to sync, I get the error that "Could not find UID TS1-BD783419-0D10-4B88-8540-730ACF03B42E." (Which is the UID in the generated ICS file.) So now, iCalendar export works, all that remains is fixing the sync code in org-caldav ...

/v
>  
>  
> --  
> Vincent Beffara
>  
>  
> On Thursday, March 7, 2013 at 13:56 , Nicolas Goaziou wrote:
>  
> > Vincent Beffara <vbeffara@ens-lyon.fr (mailto:vbeffara@ens-lyon.fr)> writes:
> >  
> > > Here is an ECM.el file, run with emacs -Q triggers the crash:  
> > >  
> > > (custom-set-variables
> > > '(org-icalendar-store-UID t)
> > > )
> > > (setq-default debug-on-error t)
> > > (add-to-list 'load-path "~/.emacs.d/el-get/org-mode/lisp")
> > > (require 'ox-icalendar)
> > > (org-icalendar--combine-files nil "~/Desktop/ECM.org (http://ECM.org)")
> > >  
> > >  
> > > without org-icalendar-store-UID it works fine. Not sure the option is
> > > needed anymore, but at least the crash can be reproduced. BTW my
> > > org-mode is at commit 516f0df.
> >  
> >  
> >  
> >  
> >  
> > The problem should be fixed in master. Could anyone confirm it?
> >  
> > Thank you for reporting this problem.
> >  
> >  
> > Regards,
> >  
> > --  
> > Nicolas Goaziou
>  

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

... sorry for replying to myself like this - here is a patch that seems to work:  

diff --git a/org-caldav.el b/org-caldav.el
index 0383366..14cca8f 100644
--- a/org-caldav.el
+++ b/org-caldav.el
@@ -786,7 +786,7 @@ is no UID to rewrite. Returns the UID."
((re-search-forward "^UID:\\(orgsexp-[0-9]+\\)" nil t)
;; This is a sexp entry, so do nothing.
(match-string 1))
- ((re-search-forward "^UID:\\(\\s-*\\)\\([A-Z][A-Z]-\\)?\\(.+\\)\\s-*$"
+ ((re-search-forward "^UID:\\(\\s-*\\)\\([A-Z][A-Z][0-9]?-\\)?\\(.+\\)\\s-*$"
nil t)
(when (match-string 1)
(replace-match "" nil nil nil 1))



I made a pull request to org-caldav, hoping I got it right!

/v  

--  
Vincent Beffara


On Thursday, March 7, 2013 at 14:31 , Vincent Beffara wrote:

> > OTOH when I ran org-caldav, it removed everything from the online calendar, and it won't put anything back there ...
>  
>  
>  
> Mmmkay, something weird is happening. I create a headline with a date, and call org-caldav. An UID is created, and the file looks like this:
>  
> *** Gayet, Damien : Une majoration de l'espérance des nombres de Betti
> :PROPERTIES:
> :ID: BD783419-0D10-4B88-8540-730ACF03B42E
> :END:
>  
> <2013-03-06 Wed 14:00>  
>  
> So far so good. But on trying to sync, I get the error that "Could not find UID TS1-BD783419-0D10-4B88-8540-730ACF03B42E." (Which is the UID in the generated ICS file.) So now, iCalendar export works, all that remains is fixing the sync code in org-caldav ...
>  
> /v
> >  
> >  
> > --  
> > Vincent Beffara
> >  
> >  
> > On Thursday, March 7, 2013 at 13:56 , Nicolas Goaziou wrote:
> >  
> > > Vincent Beffara <vbeffara@ens-lyon.fr (mailto:vbeffara@ens-lyon.fr)> writes:
> > >  
> > > > Here is an ECM.el file, run with emacs -Q triggers the crash:  
> > > >  
> > > > (custom-set-variables
> > > > '(org-icalendar-store-UID t)
> > > > )
> > > > (setq-default debug-on-error t)
> > > > (add-to-list 'load-path "~/.emacs.d/el-get/org-mode/lisp")
> > > > (require 'ox-icalendar)
> > > > (org-icalendar--combine-files nil "~/Desktop/ECM.org (http://ECM.org)")
> > > >  
> > > >  
> > > > without org-icalendar-store-UID it works fine. Not sure the option is
> > > > needed anymore, but at least the crash can be reproduced. BTW my
> > > > org-mode is at commit 516f0df.
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > > The problem should be fixed in master. Could anyone confirm it?
> > >  
> > > Thank you for reporting this problem.
> > >  
> > >  
> > > Regards,
> > >  
> > > --  
> > > Nicolas Goaziou
> >  
>  

Re: org-caldav can't find org-prepare-agenda-buffers

[Julien Cubizolles]

Vincent Beffara <vbeffara@ens-lyon.fr> writes:

> ... sorry for replying to myself like this - here is a patch that seems to work:  

I'm still running into problems even with this patch, but I'm not very
sure where they lie. Basically I get some messages about duplicate
entries, no org-caldav-results buffer pops up and a subsequent run of
org-caldav-sync informs me that the last sync was aborted. 

For now, I'm back with org-7.9.4: sync works without problem from
org->google and google->org. I'll be willing to test any changes updates
to org or org-caldav though.

Julien.

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

Hi, 

Yes, I had the very same problem, I had to restart org-caldav from scratch a few times (org-caldav-delete-everything, wait a bit for caches to empty, and then eventually it worked).

In the meantime I also tweaked the way org-caldav manages UIDs to exploit the fact that multiply occurring events are now exported multiple times in ox-icalendar, everything works out well if only org->ical is used. One thing remaining, on ical->org sync, there is an issue that makes the next sync erase the event on ical and upload it again, under another id (and with the info not generated by org-caldav missing, of course). I think I know how to fix it, just need to find some time ...

I will send what works to the list. (In the meantime, David, don't pull my second pull request, it is not right.) 

/v

-- 
Vincent Beffara


On Tuesday, March 12, 2013 at 07:21 , Julien Cubizolles wrote:

> Vincent Beffara <vbeffara@ens-lyon.fr (mailto:vbeffara@ens-lyon.fr)> writes:
> 
> > ... sorry for replying to myself like this - here is a patch that seems to work: 
> 
> I'm still running into problems even with this patch, but I'm not very
> sure where they lie. Basically I get some messages about duplicate
> entries, no org-caldav-results buffer pops up and a subsequent run of
> org-caldav-sync informs me that the last sync was aborted. 
> 
> For now, I'm back with org-7.9.4: sync works without problem from
> org->google and google->org. I'll be willing to test any changes updates
> to org or org-caldav though.
> 
> Julien. 

Re: org-caldav can't find org-prepare-agenda-buffers

[David Engster]

Vincent Beffara writes:
> In the meantime I also tweaked the way org-caldav manages UIDs to
> exploit the fact that multiply occurring events are now exported
> multiple times in ox-icalendar, everything works out well if only
> org->ical is used. One thing remaining, on ical->org sync, there is an
> issue that makes the next sync erase the event on ical and upload it
> again, under another id (and with the info not generated by org-caldav
> missing, of course). I think I know how to fix it, just need to find
> some time ...
>
> I will send what works to the list. (In the meantime, David, don't
> pull my second pull request, it is not right.)

I appreciate your work, of course, but let me add a word of warning. I
think if you give up the one-to-one correspondence that one Org event
has exactly *one* event in the remote calendar, you are entering a world
of pain.

Remember that there are three things: new items, changed items and
deleted items. Since these can happen on both sides, you have to figure
out six different cases which must be handled properly. As you've
already experienced, the changes on the calendar side are more
difficult. What happens if you delete one of the events that stems from
one Org entry?  You would first have to figure out which timestamp
created it, but how? Timestamps don't have UIDs, only the full entry has
one. So you would have to somehow add this information to the event
database which gets stored to disk. You might get this to work, but I
have a gut feeling that you'll loose robustness. For example, the Google
calendar CalDAV interface is pretty slow and not very reliable, so a
robust resume functionality is essential.

Also, I deliberately shoved conflict handling at the side for the
moment, but this is a feature which will have to be added one day, and
will complicate things much more.

-David

Re: org-caldav can't find org-prepare-agenda-buffers

[Vincent Beffara]

[-- Attachment #1: Type: text/plain, Size: 2249 bytes --]

Hi,

Agreed ... I believe the only problem will occur when one of a multiply occurring event is edited / deleted on the cal side. I have nothing more constructive to propose than just "don't do that" ... My use-case is just as a way to push org changes to cal and nothing more, and for bidirectional people who would create events on cal and sync, not much bad would happen I believe. But you're right, conflict resolution (which is what that would be) will be a problem. Would have to manage reference counts etc, for little benefit. Who is using multiply occurring events anyway?

Attached is what I am proposing: absent multiply occurring events, the org and cal IDs are the same (which is as it was except for the missing 1 in a regexp). For the second occurrence, the TS2- prefix is conserved, and so on - but it is trimmed when fed to org-id etc. As you say, it is not correct and will lead to problems down the road, but I really wanted all occurrences to appear in iCal ;-)

Cheers,

/v 
> I appreciate your work, of course, but let me add a word of warning. I
> think if you give up the one-to-one correspondence that one Org event
> has exactly *one* event in the remote calendar, you are entering a world
> of pain.
> 
> Remember that there are three things: new items, changed items and
> deleted items. Since these can happen on both sides, you have to figure
> out six different cases which must be handled properly. As you've
> already experienced, the changes on the calendar side are more
> difficult. What happens if you delete one of the events that stems from
> one Org entry? You would first have to figure out which timestamp
> created it, but how? Timestamps don't have UIDs, only the full entry has
> one. So you would have to somehow add this information to the event
> database which gets stored to disk. You might get this to work, but I
> have a gut feeling that you'll loose robustness. For example, the Google
> calendar CalDAV interface is pretty slow and not very reliable, so a
> robust resume functionality is essential.
> 
> Also, I deliberately shoved conflict handling at the side for the
> moment, but this is a feature which will have to be added one day, and
> will complicate things much more.
> 
> -David 


[-- Attachment #2: org-caldav-multi.patch --]
[-- Type: application/octet-stream, Size: 3409 bytes --]

commit 24b8a666f9d7b0b12a0d4cb5937d17952a644b0a
Author: Vincent Beffara <vbeffara@ens-lyon.fr>
Date:   Thu Mar 14 00:50:12 2013 +0100

    Allow multiply occurring events

diff --git a/org-caldav.el b/org-caldav.el
index 0383366..73445ee 100644
--- a/org-caldav.el
+++ b/org-caldav.el
@@ -413,7 +413,7 @@ Are you really sure? ")))
 
 (defun org-caldav-generate-md5-for-org-entry (uid)
   "Find Org entry with UID and calculate its MD5."
-  (let ((marker (org-id-find uid t)))
+  (let ((marker (org-id-find (org-caldav-trim-uid uid) t)))
     (when (null marker)
       (error "Could not find UID %s." uid))
     (with-current-buffer (marker-buffer marker)
@@ -606,7 +606,7 @@ which can only be synced to calendar. Ignoring." uid))
 	;; This is a changed event.
 	(org-caldav-debug-print
 	 1 (format "Event UID %s: Changed in Cal --> Org" uid))
-	(let ((marker (org-id-find (car cur) t)))
+	(let ((marker (org-id-find (org-caldav-trim-uid (car cur)) t)))
 	  (when (null marker)
 	    (error "Could not find UID %s." (car cur)))
 	  (with-current-buffer (marker-buffer marker)
@@ -648,7 +648,7 @@ which can only be synced to calendar. Ignoring." uid))
   ;; (Maybe) delete entries which were deleted in calendar.
   (unless (eq org-caldav-delete-org-entries 'never)
     (dolist (cur (org-caldav-filter-events 'deleted-in-cal))
-      (org-id-goto (car cur))
+      (org-id-goto (org-caldav-trim-uid (car cur)))
       (when (or (eq org-caldav-delete-org-entries 'always)
 		(and (eq org-caldav-delete-org-entries 'ask)
 		     (y-or-n-p "Delete this entry? ")))
@@ -775,9 +775,15 @@ Returns nil if there are no more events."
 		      (forward-line 1)
 		      (point))))
 
+(defun org-caldav-trim-uid (uid)
+  "Remove extra export prefixes from a UID.
+This is to take care of multiply occurring events, the UID kept
+by org-caldav contains TS2- and so on but the org ID doesn't."
+  (replace-regexp-in-string "^[A-Z][A-Z][0-9]?-" "" uid))
+
 (defun org-caldav-rewrite-uid-in-event ()
   "Rewrite UID in current buffer.
-This will strip prefixes like 'DL' or 'TS' the Org exporter puts
+This will strip prefixes like 'DL1' or 'TS1' the Org exporter puts
 in the UID and also remove whitespaces. Throws an error if there
 is no UID to rewrite. Returns the UID."
   (save-excursion
@@ -786,7 +792,7 @@ is no UID to rewrite. Returns the UID."
      ((re-search-forward "^UID:\\(orgsexp-[0-9]+\\)" nil t)
       ;; This is a sexp entry, so do nothing.
       (match-string 1))
-     ((re-search-forward "^UID:\\(\\s-*\\)\\([A-Z][A-Z]-\\)?\\(.+\\)\\s-*$"
+     ((re-search-forward "^UID:\\(\\s-*\\)\\([A-Z][A-Z]1?-\\)?\\(.+\\)\\s-*$"
 			 nil t)
       (when (match-string 1)
 	(replace-match "" nil nil nil 1))
@@ -962,7 +968,7 @@ If COMPLEMENT is non-nil, return all item without errors."
 
 (defun org-caldav-get-heading-from-uid (uid)
   "Get org heading from entry with UID."
-  (let ((marker (org-id-find uid t)))
+  (let ((marker (org-id-find (org-caldav-trim-uid uid) t)))
     (if (null marker)
 	"(Could not find UID)"
       (with-current-buffer (marker-buffer marker)
@@ -983,7 +989,7 @@ If COMPLEMENT is non-nil, return all item without errors."
 	       '(face link))
     (beginning-of-line)
     (looking-at "UID: \\(.+\\)$")
-    (org-id-goto (match-string 1))))
+    (org-id-goto (org-caldav-trim-uid (match-string 1)))))
 
 ;; The following is taken from icalendar.el, written by Ulf Jasper.