org-caldav: New version with proper two-way sync

org-caldav: New version with proper two-way sync

[David Engster]

I just pushed a pretty big update to org-caldav. Get it at

https://github.com/dengste/org-caldav

The short story: org-caldav now does proper two-way syncing. It's pretty
much a rewrite, actually. If you're already using org-caldav, you will
have to start from scratch after updating.

Please read the README before using this version. Most notably,
org-caldav will set org-icalendar-store-UID when doing the iCalendar
export, so every entry with an activate timestamp will get an ID
property like this:

  :PROPERTIES:
  :ID:       6cdf8805-8d1a-46ac-94fc-d225cac5f098
  :END:

If you don't want this, do not use this package.

Please report bugs here or in the github tracker. And please have
backups.

-David

Re: org-caldav: New version with proper two-way sync

[Rasmus]

David Engster <deng@randomsample.de> writes:

> I just pushed a pretty big update to org-caldav. 
> [...]
> The short story: org-caldav now does proper two-way syncing. 

Wow, that's amazing.  I'm looking forward to trying this, although I
switched to an Org-only calendar now.  But it would be great with
smart phones, I guess!

–Rasmus

-- 
Don't panic!!!

Re: org-caldav: New version with proper two-way sync

[Vincent Beffara]

Hi,  

Works great for me, thanks! One thing that changed (but it was kind of a glitch anyway, even if a nice one to have): events with several timestamps now appear only once rather than one per timestamp. Not sure if this is due to org-caldav or to recent evolutions in the org->ics export. Anyway, not a big deal.

One thing hinted at in the docs, but I couldn't figure out how to do it: how do you use an authinfo file? Mine seems to be ignored totally ...

Best,

/vincent  

--  
Vincent Beffara


On Monday, January 14, 2013 at 23:29 , Rasmus wrote:

> David Engster <deng@randomsample.de (mailto:deng@randomsample.de)> writes:
>  
> > I just pushed a pretty big update to org-caldav.  
> > [...]
> > The short story: org-caldav now does proper two-way syncing.  
>  
>  
>  
> Wow, that's amazing. I'm looking forward to trying this, although I
> switched to an Org-only calendar now. But it would be great with
> smart phones, I guess!
>  
> –Rasmus
>  
> --  
> Don't panic!!!

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

David Engster <deng@randomsample.de> writes:

> I just pushed a pretty big update to org-caldav. Get it at 
> 
> https://github.com/dengste/org-caldav 
> 
> The short story: org-caldav now does proper two-way 
> syncing. It's pretty much a rewrite, actually. If you're already 
> using org-caldav, you will have to start from scratch after 
> updating. 

David,
 
this is fantastic!  I've just tried this out and it works 
brilliantly, at least for org -> Google.  I've not tried synching 
yet but looking good so far.  It's enough for me to stop using my 
own system which did two way transfers but no synching.

I did have two sync errors.  Is there any way to diagnose what 
went wrong?  The output from org-caldav is very nice in that it 
provides a link to the entries that were processed, including 
those with errors, but I cannot see anything strange in either of 
the entries.
 
Off now to try two way synching!

Thanks for this,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

David Engster <deng@randomsample.de> writes:
> Please report bugs here or in the github tracker. And please 
> have backups. 
 
Dear David,

First of all, ignore my previous request for info on how to debug 
failed synchronisation operations: I've seen that you create an 
*org-caldav-debug* buffer and I've seen that the errors are 404 
errors.  Trying again has resolved those failed synchronisations. 

I am unable to get entries any entries created in Google's 
calendar to appear in my org inbox file, whether old or new.  The 
output from org-caldav indicates nothing has been done yet the ics 
events file from Google does have the relevant entries. 

Further, I seem to have run into strange problem: every time I 
start a new sync operation, I get the following error:

,---- | Debugger entered--Lisp error: (error "Could not find UID 
emacs20726839125514526.")  |   signal(error ("Could not find UID 
emacs20726839125514526."))  |   error("Could not find UID %s." 
"emacs20726839125514526") |   (progn (error "Could not find UID 
%s." uid)) |   (if (null marker) (progn (error "Could not find UID 
%s." uid))) |   (let ((marker (org-id-find uid t))) (if (null 
marker) [...]  | 
org-caldav-generate-md5-for-org-entry("emacs20726839125514526") | 
(let* ((uid (org-caldav-rewrite-uid-in-event)) [...]  |   (while 
(org-caldav-narrow-next-event) [...]  |   (save-current-buffer 
(set-buffer buf) [...]  | 
org-caldav-update-eventdb-from-org(#<buffer org-caldav-4496C1y>) | 
(if (and org-caldav-event-list [...]  |   org-caldav-sync() | 
call-interactively(org-caldav-sync record nil) | 
command-execute(org-caldav-sync record) | 
execute-extended-command(nil "org-caldav-sync") | 
call-interactively(execute-extended-command nil nil) `---- 

I've truncated the lines for posting but I'm happy to send you the 
full output from the debug trace.

I cannot find the UID anywhere in my files.

However, if I try org-caldav-sync again, it asks whether I want to 
resume the interrupted sync operation and it works.  This 
behaviour is consistent. 

I wonder whether the two things are related?

Thanks,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87

Re: org-caldav: New version with proper two-way sync

[JBash]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

It's working great for me to.  I was able to test 2 way synch, so far with
a very simple case, and it worked fine.

Thanks very much.

Jerry


On Mon, Jan 14, 2013 at 4:53 PM, David Engster <deng@randomsample.de> wrote:

> I just pushed a pretty big update to org-caldav. Get it at
>
> https://github.com/dengste/org-caldav
>
> The short story: org-caldav now does proper two-way syncing. It's pretty
> much a rewrite, actually. If you're already using org-caldav, you will
> have to start from scratch after updating.
>
> Please read the README before using this version. Most notably,
> org-caldav will set org-icalendar-store-UID when doing the iCalendar
> export, so every entry with an activate timestamp will get an ID
> property like this:
>
>   :PROPERTIES:
>   :ID:       6cdf8805-8d1a-46ac-94fc-d225cac5f098
>   :END:
>
> If you don't want this, do not use this package.
>
> Please report bugs here or in the github tracker. And please have
> backups.
>
> -David
>
>

[-- Attachment #2: Type: text/html, Size: 1564 bytes --]

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

David,

I'm following up on my own posts...  how uncool!  ;-)

I've tracked down the root of the various problems I have 
encountered with the synchronisation.  It all comes down to 
character sets.  I have some entries that have UTF-8 characters 
that are not present in ASCII, specifically accented characters 
and similar (latin1, basically, but not exclusively).  Any such 
entries cause the synchronisation to fail. 

Resuming a failed synchronisation seems to forget to try to bring 
in the external calendar entries into org?  In other words, 
resuming doesn't really resume but seems to start at some later 
point.  Not sure.

In any case, temporarily, I have commented out the problem entries 
from my org agenda files and everything now works very well 
indeed, including two way synchronisation of creating, deleting 
and changing.

The fact that descriptions are not synchronised for changed 
entries is something I understand but probably need to think about 
some more (in terms of default behaviour).  Is there a practical 
limitation on the size of the description entry that could be 
synchronised?  The default is 100 but would there be any harm in 
having this 1, 2 or even 3 orders of magnitude larger?  Or even 
unlimited?  Just wondering.

Anyway, with respect to my problems, would you please have a look 
to see if you are assuming ASCII or similar and whether there is 
anything you can do about this?  I (and many others, I assume) 
really do need to be able to work in UTF-8 at the very least.  I 
am hooked on org-caldav already and would hate to have to go back 
to my old system! 

Thanks again,
eric

-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87

Re: org-caldav: New version with proper two-way sync

[Detlef Steuer]

Thank you a lot!

I ran into the issue that repeated syncs have putting and deleting 
wrong (it seems).


I use a script in crontab to sync different calendars:

-----
#!/bin/bash

### Es werden die Inhalte der Org-Mode Dateien in verschiedene Calender
### in der owncloud exportiert.

for calendar in Ferien Geburtstage Todo WAB;  do \
emacs --batch \
       --load ${HOME}/Working-Copies/Scripts/Org-Multicalendar/${calendar}.el ; \
done
##

-----

${calendar}.el looking like


-----
(add-to-list 'load-path "/home/steuer/GIT/org-mode/lisp/")
(require 'org)

(setq org-icalendar-include-todo t)
(setq org-icalendar-use-deadline '(todo-due))
(setq org-icalendar-use-scheduled '(event-if-not-todo))

(require 'url)
(require 'auth-source)
(setq auth-sources '((:source "~/.netrc" :host t :protocol t)))
(setq auth-source-debug t)

(add-to-list 'load-path "/home/steuer/GIT/org-caldav/")
(setq org-caldav-url "http://steuer-geilke.de/owncloud/remote.php/caldav/calendars/detlef")
(setq org-caldav-calendar-id "ferien")
(setq org-caldav-files '("/home/steuer/.pim/ferien.org") )
(setq org-caldav-inbox "/home/steuer/.pim/caldav-inbox.org")
(require 'org-caldav)
(org-caldav-sync)
-----


That worked for me, but now, after a first successful sync I get:

(2. and subsequent syncs!)
-----
steuer@vknecht-intel:~> /home/steuer/bin/orgdavexport.sh
Contacting host: ######
Reading [application/xml; charset=utf-8]... 306 bytes of 291 bytes (105%)
auth-source-user-or-password: get login for ######:80 (http)
auth-source-user-or-password: found (login)=(#####) for ######:80 (http)
auth-source-user-or-password: get password for #####:80 (http)
auth-source-user-or-password: found (password)=SECRET for #####:80 (http)
Reading [application/xml; charset=utf-8]... 1k of 1k (101%)
OVERVIEW
OVERVIEW
Saving file /tmp/orgics11019cvx...
Wrote /tmp/orgics11019cvx
Saving file /tmp/orgics11019O5A...
Wrote /tmp/orgics11019O5A
Saving file /tmp/org-caldav-11019TUn...
Wrote /tmp/org-caldav-11019TUn
Reading [application/xml; charset=utf-8]... 1k of 1k (101%)
Reading [application/xml; charset=utf-8]... 1k of 1k (101%)
Putting event 1 of 5
Putting event 2 of 5
Putting event 3 of 5
Putting event 4 of 5
Putting event 5 of 5
Deleting event 1 from 5
Deleting event 2 from 5
Deleting event 3 from 5
Deleting event 4 from 5
Deleting event 5 from 5
Wrote /home/steuer/.emacs.d/org-caldav-fb95cae.el
Symbol's function definition is void: pop-to-buffer-same-window

###

So it seems all events get deleted immediately after loading them up.
Indeed my calendars show up empty again in owncloud :-)

Any hints?

org from today, emacs 23.2, org-caldav from yesterday


Deltlef 


On Mon, 14 Jan 2013 22:53:53 +0100
David Engster <deng@randomsample.de> wrote:

> I just pushed a pretty big update to org-caldav. Get it at
> 
> https://github.com/dengste/org-caldav
> 
> The short story: org-caldav now does proper two-way syncing. It's pretty
> much a rewrite, actually. If you're already using org-caldav, you will
> have to start from scratch after updating.
> 
> Please read the README before using this version. Most notably,
> org-caldav will set org-icalendar-store-UID when doing the iCalendar
> export, so every entry with an activate timestamp will get an ID
> property like this:
> 
>   :PROPERTIES:
>   :ID:       6cdf8805-8d1a-46ac-94fc-d225cac5f098
>   :END:
> 
> If you don't want this, do not use this package.
> 
> Please report bugs here or in the github tracker. And please have
> backups.
> 
> -David
> 
> 

Re: org-caldav: New version with proper two-way sync

[David Engster]

Detlef Steuer writes:
> So it seems all events get deleted immediately after loading them up.
> Indeed my calendars show up empty again in owncloud :-)
>
> Any hints?

Could you post the contents of the *org-caldav-debug* buffer when this
happens?

-David

Re: org-caldav: New version with proper two-way sync

[David Engster]

Eric S. Fraga writes:
> I've tracked down the root of the various problems I have encountered
> with the synchronisation.  It all comes down to character sets.  I
> have some entries that have UTF-8 characters that are not present in
> ASCII, specifically accented characters and similar (latin1,
> basically, but not exclusively).  Any such entries cause the
> synchronisation to fail. 

Could you post an example entry where this happens?

> Resuming a failed synchronisation seems to forget to try to bring in
> the external calendar entries into org? 

That happens last, so I wouldn't know why this would be skipped on
resume. It's kinda hard to debug, though.

> The fact that descriptions are not synchronised for changed entries is
> something I understand but probably need to think about some more (in
> terms of default behaviour).  Is there a practical limitation on the
> size of the description entry that could be synchronised?  The default
> is 100 but would there be any harm in having this 1, 2 or even 3
> orders of magnitude larger?  Or even unlimited?  Just wondering.

I was wondering about that, too, but couldn't find any definite
statements on the maximum length of the DESCRIPTION field. I only
skimmed RFC 2445, though. Even if there is such a definite limit, I
wouldn't count on servers to correctly implement that. I guess you just
have to try what works.

> Anyway, with respect to my problems, would you please have a look to
> see if you are assuming ASCII or similar and whether there is anything
> you can do about this?  I (and many others, I assume) really do need
> to be able to work in UTF-8 at the very least.

I will look into that. I'm pretty sure this is fixable.

-David

Re: org-caldav: New version with proper two-way sync

[David Engster]

Vincent Beffara writes:
> Works great for me, thanks! One thing that changed (but it was kind of
> a glitch anyway, even if a nice one to have): events with several
> timestamps now appear only once rather than one per timestamp. Not
> sure if this is due to org-caldav or to recent evolutions in the
> org->ics export. Anyway, not a big deal.

That's a consequence of org-caldav now doing proper syncing. Every Org
entry is linked through the UID to exactly one event in the calendar. If
I would allow an entry to produce several events in the calendar, we
would get duplicate UIDs there, which is not allowed. One could code
around that (in fact, the iCalendar export already does that by putting
stuff like 'DL' or 'TS' in front of the UIDs), but it would make syncing
from the Calendar to Org much more difficult.

> One thing hinted at in the docs, but I couldn't figure out how to do
> it: how do you use an authinfo file? Mine seems to be ignored totally
> ...

The example given in the Readme

machine www.google.com:443 port https login username password secret

should work. If it doesn't, set `auth-source-debug' to 't' and look into
the Messages buffer. Maybe it doesn't work properly on older Emacs
versions; a lot has changed in auth-source recently.

-David

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

David Engster <deng@randomsample.de> writes:

> Eric S. Fraga writes: 
>> I've tracked down the root of the various problems I have 
>> encountered with the synchronisation.  It all comes down to 
>> character sets.  I 
> 
>> have some entries that have UTF-8 characters that are not 
>> present in ASCII, specifically accented characters and similar 
>> (latin1, basically, but not exclusively).  Any such entries 
>> cause the synchronisation to fail.  
> 
> Could you post an example entry where this happens? 

Sure but the problem does not appear to be what I had originally 
thought.  It turns out that all of my entries with non-ASCII 
characters happened to be SEXP based entries, of this form:

#+begin_src org
* Test entries %%(diary-anniversary 1971 03 13) Somebody's 
birthday (%d años) #+end_src 

(see the N TILDE character in the spanish word for years)

The problem, however, seems related to the use SEXP entries and 
not the character set.

Having said this, my org diary file has had the encoding changed 
out from under me so that all my UTF-8 characters have been 
mangled.  I've not quite figured out how this happened or when it 
happened between yesterday and today.  I cannot reproduce the 
problem at the moment.  This may be coincidental and have nothing 
to do with org-caldav.  However, it may be something to do with 
using org-caldav in emacs -batch mode and whether files get loaded 
in the same way.  Still playing with this.
 
>> Resuming a failed synchronisation seems to forget to try to 
>> bring in the external calendar entries into org?  
> 
> That happens last, so I wouldn't know why this would be skipped 
> on resume. It's kinda hard to debug, though. 

Okay.
 
>> The fact that descriptions are not synchronised for changed 
>> entries is something I understand but probably need to think 
>> about some more (in terms of default behaviour).  Is there a 
>> practical limitation on the size of the description entry that 
>> could be synchronised?  The default is 100 but would there be 
>> any harm in having this 1, 2 or even 3 orders of magnitude 
>> larger?  Or even unlimited?  Just wondering. 
> 
> I was wondering about that, too, but couldn't find any definite 
> statements on the maximum length of the DESCRIPTION field. I 
> only skimmed RFC 2445, though. Even if there is such a definite 
> limit, I wouldn't count on servers to correctly implement 
> that. I guess you just have to try what works. 

Thanks.  I'll stick to the default behaviour you have chosen; it's 
good enough for most of my use cases and seems safest.
 
>> Anyway, with respect to my problems, would you please have a 
>> look to see if you are assuming ASCII or similar and whether 
>> there is anything you can do about this?  I (and many others, I 
>> assume) really do need to be able to work in UTF-8 at the very 
>> least. 
> 
> I will look into that. I'm pretty sure this is fixable. 

As mentioned above, I am now not sure if there are problems with 
ASCII vs non-ASCII encodings.  I will keep playing.
 
Thanks,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> Sure but the problem does not appear to be what I had originally 
> thought.  It turns out that all of my entries with non-ASCII 
> characters happened to be SEXP based entries, of this form: 
> 
> #+begin_src org * Test entries %%(diary-anniversary 1971 03 13) 
> Somebody's birthday (%d años) #+end_src  

Ummm, something has screwed up my email...  I was playing with
=use-hard-newlines= for email.  Sorry about this.  The example should
have been (fingers crossed hoping this comes through fine this time):

#+begin_src org
* Test entries
%%(diary-anniversary 1971 03 13) Somebody's birthday (%d años)
#+end_src 

Hope it comes through correctly this time...

-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87

Re: org-caldav: New version with proper two-way sync

[Detlef Steuer]

> Having said this, my org diary file has had the encoding changed 
> out from under me so that all my UTF-8 characters have been 
> mangled.  I've not quite figured out how this happened or when it 
> happened between yesterday and today.  I cannot reproduce the 
> problem at the moment.  This may be coincidental and have nothing 
> to do with org-caldav.  However, it may be something to do with 
> using org-caldav in emacs -batch mode and whether files get loaded 
> in the same way.  Still playing with this.

FWIW: here UTF-8 is respected and stays UTF-8, in -batch, too.

Detlef

Re: org-caldav: New version with proper two-way sync

[Detlef Steuer]

> Detlef Steuer writes:
> > So it seems all events get deleted immediately after loading them up.
> > Indeed my calendars show up empty again in owncloud :-)
> >
> > Any hints?
> 
> Could you post the contents of the *org-caldav-debug* buffer when this
> happens?

As soon as it happens again. 
I can't reproduce the emptying of calendars at the moment. 
Whatever that was I saw yesterday.

Sorry for the noise.

Detlef


> 
> -David
> 
> 

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

Detlef Steuer <detlef.steuer@gmx.de> writes:
> 
> FWIW: here UTF-8 is respected and stays UTF-8, in -batch, too. 
> 
> Detlef 

Thanks for the data point.  What Emacs version are you using?  I 
seem to have run into a coding problem elsewhere (in gnus within 
Emacs) so maybe I'm hitting a recently introduced bug in 
Emacs.  One of the risks of tracking Emacs development...

-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 + Ma Gnus v0.6
: BBDB version 3.02 ($Date: 2013/01/13 22:41:36 $)

Re: org-caldav: New version with proper two-way sync

[Detlef Steuer]

On Thu, 17 Jan 2013 19:05:15 +1030
Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> Detlef Steuer <detlef.steuer@gmx.de> writes:
> > 
> > FWIW: here UTF-8 is respected and stays UTF-8, in -batch, too. 
> > 
> > Detlef 
> 
> Thanks for the data point.  What Emacs version are you using?  


emacs 23.2 under linux
Detlef 

> I 
> seem to have run into a coding problem elsewhere (in gnus within 
> Emacs) so maybe I'm hitting a recently introduced bug in 
> Emacs.  One of the risks of tracking Emacs development...
> 
> -- 
> : Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
> : in Emacs 24.3.50.1 + Ma Gnus v0.6
> : BBDB version 3.02 ($Date: 2013/01/13 22:41:36 $)
> 
> 
> 

Re: org-caldav: New version with proper two-way sync

[Eric S Fraga]

Detlef Steuer <detlef.steuer@gmx.de> writes:

> On Thu, 17 Jan 2013 19:05:15 +1030 Eric S Fraga 
> <e.fraga@ucl.ac.uk> wrote: 
> 
>> Detlef Steuer <detlef.steuer@gmx.de> writes: 
>> >  FWIW: here UTF-8 is respected and stays UTF-8, in -batch, 
>> > too.    Detlef  
>>  Thanks for the data point.  What Emacs version are you using?   
>  
> emacs 23.2 under linux 

Thanks.  I'm definitely running a much newer version.  I'll keep 
playing around to see if I can find the source of my encoding 
problems.
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-826-gbe0d87