Timezone handling problem in icalendar.el/icalendar-import-*

Timezone handling problem in icalendar.el/icalendar-import-*

[Christian]

I have detected a problem with timezone handling in icalendar.el;
unfortunately I know far too little about timezones to be able to say
what the correct fix is.

The problem was discovered in trying to use icalendar to parse some
Microsoft Exchange 2007 appointments on a SuSE 10.1 Linux system. I am
using emacs version "23.1.50.1" (from CVS) and the icalendar distributed
with that.

The appointments are recorded in timezone GMT+2 whereas I am in timezone
GMT+1. But when extracting the appointment using `icalendar-import-buffer',
all times where an hour off. Basically, it was as if the appointment was
registered in GMT rather than GMT+2.

After a lot of huffing and puffing, I narrowed it down to the following
piece of code:

    (format-time-string  "%Y-%m-%dT%T%z" (encode-time 0 30 11 16 12 2009 "(STD?)-02:00(DST?)-03:00,M3.-1.0/03:00:00,M10.-1.0/04:00:00"))
    ;=> "2009-12-16T09:30:00+0100"

When evaluated, this returns a time of 09:30 but it should have reported
10:30. Checking the man page for `tzset', however, I read the following:

       The second format is used when there is daylight saving time:

              std offset dst [offset],start[/time],end[/time]
       ...
              Mm.w.d This  specifies  day  d (0 <= d <= 6) of week w (1 <= w <= 5) of
              month m (1 <= m <= 12).  Week 1 is the first week in which day d
              occurs and week 5 is the last week in which day d occurs.  Day 0
              is a Sunday.

and sure enough, if I change the week specification from "-1" to "5" as in:

    (format-time-string  "%Y-%m-%dT%T%z" (encode-time 0 30 11 16 12 2009 "(STD?)-02:00(DST?)-03:00,M3.5.0/03:00:00,M10.5.0/04:00:00"))
    ;=> "2009-12-16T10:30:00+0100"

I now get the correct start time of 10:30. In other words, at least on
this particular Linux system, a week specification for DST start/end of
"-1" is not supported, making `encode-time' falling back to GMT. Instead
one should use "5" to indicate the last week of the month. I have no
idea whether this is a bug in icalendar, SuSE 10.1 or Linux as such.

One fix (but I doubt it is the correct one) is to change the function
`icalendar--convert-tz-offset' into the following (one `if' form has
been inserted following the comment of "FIX"):

    (defun icalendar--convert-tz-offset (alist dst-p)
      "Return a cons of two strings representing a timezone start.
    ALIST is an alist entry from a VTIMEZONE, like STANDARD.
    DST-P is non-nil if this is for daylight savings time.
    The strings are suitable for assembling into a TZ variable."
      (let ((offset (car (cddr (assq 'TZOFFSETTO alist))))
            (rrule-value (car (cddr (assq 'RRULE alist))))
            (dtstart (car (cddr (assq 'DTSTART alist)))))
        ;; FIXME: for now we only handle RRULE and not RDATE here.
        (when (and offset rrule-value dtstart)
          (let* ((rrule (icalendar--split-value rrule-value))
                 (freq (cadr (assq 'FREQ rrule)))
                 (bymonth (cadr (assq 'BYMONTH rrule)))
                 (byday (cadr (assq 'BYDAY rrule))))
            ;; FIXME: we don't correctly handle WKST here.
            (if (and (string= freq "YEARLY") bymonth)
                (cons
                 (concat
                  ;; Fake a name.
                  (if dst-p "(DST?)" "(STD?)")
                  ;; For TZ, OFFSET is added to the local time.  So,
                  ;; invert the values.
                  (if (eq (aref offset 0) ?-) "+" "-")
                  (substring offset 1 3)
                  ":"
                  (substring offset 3 5))
                 ;; The start time.
                 (let* ((day (icalendar--get-weekday-number (substring byday -2)))
                        (week (if (eq day -1)
                                  byday
                                (substring byday 0 -2))))
                   ;; FIX for bad week spec
                   (if (equal week "-1")
                       (setq week "5"))
                   (concat "M" bymonth "." week "." (if (eq day -1) "0"
                                                      (int-to-string day))
                           ;; Start time.
                           "/"
                           (substring dtstart -6 -4)
                           ":"
                           (substring dtstart -4 -2)
                           ":"
                           (substring dtstart -2)))))))))


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Andreas Schwab]

"Christian" <christian.lynbech@tieto.com> writes:

>     (format-time-string  "%Y-%m-%dT%T%z" (encode-time 0 30 11 16 12 2009 "(STD?)-02:00(DST?)-03:00,M3.-1.0/03:00:00,M10.-1.0/04:00:00"))
>     ;=> "2009-12-16T09:30:00+0100"

This timezone string is clearly outside of the specs.  The timezone
names shall consist only of alphabetic characters in the portable
character set.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Christian Lynbech]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> "Christian" <christian.lynbech@tieto.com> writes:
>> (format-time-string  "%Y-%m-%dT%T%z" (encode-time 0 30 11 16 12 2009 "(STD?)-02:00(DST?)-03:00,M3.-1.0/03:00:00,M10.-1.0/04:00:00"))
>> ;=> "2009-12-16T09:30:00+0100"

Andreas> This timezone string is clearly outside of the specs.  The timezone
Andreas> names shall consist only of alphabetic characters in the portable
Andreas> character set.

I assume you refer to the use of "(STD?)" and "(DST?)". It actually
works since it appears that mktime() will assume GMT if it encounters
something it does not understand, but that is admittedly not a wise
thing to count on, Why icalendar didn't just use "GMT" or "UTC" is not
clear to me.

I can also report that when using emacs version "22.3.1" on Mac OSX 10.5.8,
one also needs to replace the "-1" with "5".


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Ulf Jasper]

"Christian" <christian.lynbech@tieto.com> writes:

> I have detected a problem with timezone handling in icalendar.el;
> unfortunately I know far too little about timezones to be able to say
> what the correct fix is.
>
> The problem was discovered in trying to use icalendar to parse some
> Microsoft Exchange 2007 appointments on a SuSE 10.1 Linux system. I am
> using emacs version "23.1.50.1" (from CVS) and the icalendar distributed
> with that.
>
> The appointments are recorded in timezone GMT+2 whereas I am in timezone
> GMT+1. But when extracting the appointment using `icalendar-import-buffer',
> all times where an hour off. Basically, it was as if the appointment was
> registered in GMT rather than GMT+2.

Could you please provide an icalendar example? 

Thanks,
Ulf

-- 
Herzlichen Glückwunsch noch einmal, vergessen Sie Ihre Angaben,und dies 
für die Art und Weise von der öffentlichen Bekanntmachung.

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Christian]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

I have attached an example below. I have crossed out names and email
addresses but otherwise this is how it appears from Exchange.

The appointment in question would appear in Outlook as occuring between
13:00 and 14:30 (GMT+01:00) on December the 9.

Let me know if you need more examples.

                              -- Christian


[-- Attachment #2: AppHandled.16.1 --]
[-- Type: application/octet-stream, Size: 1582 bytes --]

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft Exchange Server 2007
VERSION:2.0
BEGIN:VTIMEZONE
TZID:FLE Standard Time
BEGIN:STANDARD
DTSTART:16010101T040000
TZOFFSETFROM:+0300
TZOFFSETTO:+0200
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T030000
TZOFFSETFROM:+0200
TZOFFSETTO:+0300
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ORGANIZER;CN=xxxxxxxxxx xxxxxxxxxxxx:MAILTO:xxxxxxxxxxxxxxxxxxxxxxxx@xxxxx.
 xxx
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE;CN=xxxxxxx xx
 xxxxxxx:MAILTO:xxxxxxxxxxxxxxxxx@xxxxx.xxx
DESCRIPTION;LANGUAGE=en-US:When: 9. december 2009 13:00-14:30 (GMT+01:00) A
 msterdam\, Berlin\, Bern\, Rome\, Stockholm\, Vienna.\nWhere: RES DK Aarhu
 s Room 0.20 CA (12 seats)\n\n*~*~*~*~*~*~*~*~*~*\n\n\n\n
SUMMARY;LANGUAGE=en-US:Mødelokale
DTSTART;TZID=FLE Standard Time:20091209T140000
DTEND;TZID=FLE Standard Time:20091209T153000
UID:040000008200E00074C5B7101A82E00800000000A07D9B07CD78CA01000000000000000
 0100000004F696FB697648242B59C852413F0CE2C
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20091209T114212Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:0
LOCATION;LANGUAGE=en-US:RES DK Aarhus Room 0.20 CA (12 seats)
X-MICROSOFT-CDO-APPT-SEQUENCE:0
X-MICROSOFT-CDO-OWNERAPPTID:-750139431
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:0
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:REMINDER
TRIGGER;RELATED=START:-PT15M
END:VALARM
END:VEVENT
END:VCALENDAR

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Ulf Jasper]

"Christian" <christian.lynbech@tieto.com> writes:

> I have attached an example below. I have crossed out names and email
> addresses but otherwise this is how it appears from Exchange.
>
> The appointment in question would appear in Outlook as occuring between
> 13:00 and 14:30 (GMT+01:00) on December the 9.

I checked the specs: Your idea to replace "-1" with "5" appears to be
absolutely correct.

RFC2445 (ical):

   Each BYDAY value can also be preceded by a positive (+n) or negative
   (-n) integer. If present, this indicates the nth occurrence of the
   specific day within the MONTHLY or YEARLY RRULE. For example, within
   a MONTHLY rule, +1MO (or simply 1MO) represents the first Monday
   within the month, whereas -1MO represents the last Monday of the
   month. If an integer modifier is not present, it means all days of
   this type within the specified frequency. For example, within a
   MONTHLY rule, MO represents all Mondays within the month.

man tzset:

   Mm.w.d This specifies day d (0 <= d <= 6) of week w (1 <= w <= 5) of
   month m (1 <= m <= 12).  Week 1 is the first week in which day d
   occurs and week 5 is the last week in which day d occurs.  Day 0 is a
   Sunday.


I'll fix it in CVS (and add some unit tests).

Thanks for investigating!
Ulf

-- 
Herzlichen Glückwunsch noch einmal, vergessen Sie Ihre Angaben,und dies 
für die Art und Weise von der öffentlichen Bekanntmachung.

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Christian]

>>>>> "Ulf" == Ulf Jasper <ulf.jasper@web.de> writes:

Ulf> I'll fix it in CVS (and add some unit tests).

Thanks. 

While you are at it, perhaps you should also act on Andreas' observation
that icalendar.el uses illegal timezone names (ie. "(STD?)"  and
"(DST?)"). As I said, it happens to work because mktime() sensibly
enough will assume GMT when nothing better can be deduced but why not
just use "GMT" directly?

                              -- Christian

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Ulf Jasper]

"Christian" <christian.lynbech@tieto.com> writes:

>>>>>> "Ulf" == Ulf Jasper <ulf.jasper@web.de> writes:
>
> Ulf> I'll fix it in CVS (and add some unit tests).
>
> Thanks. 

Fixed.

> While you are at it, perhaps you should also act on Andreas' observation
> that icalendar.el uses illegal timezone names (ie. "(STD?)"  and
> "(DST?)"). As I said, it happens to work because mktime() sensibly
> enough will assume GMT when nothing better can be deduced but why not
> just use "GMT" directly?

Changed timezone names to "STD" and "DST". As I understand these are
just names and should consist of alphabetic characters only.

Ulf

-- 
Herzlichen Glückwunsch noch einmal, vergessen Sie Ihre Angaben,und dies 
für die Art und Weise von der öffentlichen Bekanntmachung.

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Christian Lynbech]

>>>>> "Ulf" == Ulf Jasper <ulf.jasper@web.de> writes:

Ulf> Changed timezone names to "STD" and "DST". As I understand these are
Ulf> just names and should consist of alphabetic characters only.

Hm, perhaps. 

My understanding is/was that some names are recognized (such as CET for
central europe or PST for pacific standard time). These maps to (the
equivalent of) GMT+1 (for CET). Ie. I thought we had a synctactic requirement
(must start with a letter and be at least three letters long) and a
semantic (must belong to this particular set). If a function such as
mktime() encounters something it cannot parse it wil just assume GMT.

If this is correct, why not just use "GMT" rather than "STD" and "DST"?

                              -- Christian

Re: Timezone handling problem in icalendar.el/icalendar-import-*

[Ulf Jasper]

Christian Lynbech <christian@defun.dk> writes:

>>>>>> "Ulf" == Ulf Jasper <ulf.jasper@web.de> writes:
>
> Ulf> Changed timezone names to "STD" and "DST". As I understand these are
> Ulf> just names and should consist of alphabetic characters only.
>
> Hm, perhaps. 
>
> My understanding is/was that some names are recognized (such as CET for
> central europe or PST for pacific standard time). These maps to (the
> equivalent of) GMT+1 (for CET). Ie. I thought we had a synctactic requirement
> (must start with a letter and be at least three letters long) and a
> semantic (must belong to this particular set). If a function such as
> mktime() encounters something it cannot parse it wil just assume GMT.
>
> If this is correct, why not just use "GMT" rather than "STD" and "DST"?

The timezone names "DST" and "STD" together with the time-offsets and
daylight saving time definition are passed as ZONE argument to
encode-time. This in turn fills the environment variable TZ with this
data and calls tzset. The tzset manpage says:

       The  value of TZ can be one of three formats.  The first format is used
       when there is no daylight saving time in the local timezone:

              std offset

       The std string specifies the name of the timezone and must be three  or
       more  alphabetic characters.  The offset string immediately follows std
       and specifies the time value to be added to the local time to get Coor‐
       dinated  Universal  Time  (UTC).   The  offset is positive if the local
       timezone is west of the Prime Meridian and negative if it is east.  The
       hour must be between 0 and 24, and the minutes and seconds 0 and 59.

       The second format is used when there is daylight saving time:

              std offset dst [offset],start[/time],end[/time]

       There  are  no spaces in the specification.  The initial std and offset
       specify the standard timezone, as described above.  The dst string  and
       offset  specify the name and offset for the corresponding daylight sav‐
       ing timezone.  If the offset is omitted, it default to one  hour  ahead
       of standard time.

icalendar.el uses the "second format". It sets the timezone names std
and dst to "STD" and "DST", but could use any other strings as
well. These names are irrelevant -- only the offsets and start/end times
matter.

The VTIMEZONE element in the icalendar object *defines* a timezone which
is translated and passed to encode-time/tzset. It does not reference a
timezone defined elsewhere. The name of this timezone (std) is
irrelevant and so is the name of the daylight saving time (dst).

Ulf

-- 
Herzlichen Glückwunsch noch einmal, vergessen Sie Ihre Angaben,und dies 
für die Art und Weise von der öffentlichen Bekanntmachung.