From mboxrd@z Thu Jan 1 00:00:00 1970 From: Allen Li Subject: Re: Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)] Date: Wed, 1 Nov 2017 17:26:53 -0700 Message-ID: References: <87bmknkwhe.fsf@nicolasgoaziou.fr> <87tvyfjgjk.fsf@nicolasgoaziou.fr> <87tvyelb60.fsf@gmail.com> <87shdykuxc.fsf@gmail.com> <87r2thlewz.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:54987) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eA3LU-00027m-Ur for emacs-orgmode@gnu.org; Wed, 01 Nov 2017 20:26:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eA3LS-00060N-N3 for emacs-orgmode@gnu.org; Wed, 01 Nov 2017 20:26:56 -0400 Received: from mail-qt0-x22d.google.com ([2607:f8b0:400d:c0d::22d]:55272) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eA3LS-00060C-GF for emacs-orgmode@gnu.org; Wed, 01 Nov 2017 20:26:54 -0400 Received: by mail-qt0-x22d.google.com with SMTP id z19so4828213qtg.11 for ; Wed, 01 Nov 2017 17:26:54 -0700 (PDT) In-Reply-To: <87r2thlewz.fsf@gmail.com> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Tim Cross Cc: emacs-orgmode@gnu.org, Nicolas Goaziou On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross wrote: > > Let me try to clarify and see if that helps. > > Org is only forcing time into UTC format for functions which deal with > time comparisons. i.e. those which use org-2ft. The reason for doing > this is to try and avoid issues relating to daylight savings timezone > changes when performing comparison operations. > > It is not clear to me exactly how this is actually creating a problem > for your scenario as there does not seem to be anywhere in org where > values created by org-2ft are converted back to human readable time > strings. If you can provide more details on how this is a practical > issue in org-mode that might help. I think I made the problem quite clear multiple times. Assuming today is 2017-10-31, SCHEDULED<"" will match results that have SCHEDULED=<2017-11-01> depending on your local time zone. > > I do think there was an error in your example analysis. > > In your examples of > > (current-time-string (float-time)) > "Mon Oct 30 21:21:31 2017" ; right > > (current-time-string (org-time-today)) > "Mon Oct 30 00:00:00 2017" ; right > > (current-time-string (org-2ft "<2017-10-31>")) > "Mon Oct 30 17:00:00 2017" ; wrong > > the last one is only wrong because you have failed to tell > current-time-string that the value you are converting is a UTC time. If > you change the code to > > (current-time-string (org-2ft "<2017-10-31>") "UTC") > "Mon Oct 30 00:00:00 2017" ; correct No, that is wrong. Mon Oct 30 00:00:00 2017 UTC != Mon Oct 30 00:00:00 2017 UTC-7 (format-time-string "%FT%T%z" (org-2ft "<2017-10-31>")) "2017-10-30T17:00:00+0700" ; wrong (format-time-string "%FT%T%z" (org-2ft "<2017-10-31>") "UTC") "2017-10-31T00:00:00+0000" ; wrong In fact, you haven't solved a thing, since (current-time-string (org-2ft "<2017-10-31>") "UTC") == (current-time-string (org-2ft "<2017-10-31>")) The time is still the same, and still wrong, just formatted differently if you silently drop the TZ info from the time string. > > Now, considering your suggestion to change org-2ft to not use UTC. > > If we change org-2ft to not use UTC time, we end up with time > related calculations which cross daylight savings boarders becoming > incorrect. So, if you have a clock table where you want to calculate the > number of hours between two timestamps and one is within daylgiht > savings time and the other is outside it, your calculation will be out > by an hour. Or rather, we end up with time related calculations which cross daylight savings becoming correct. But I am beginning to see the picture; someone wanted time related calculations crossing DST to be incorrect to appear "right" to time zone naive applications. > > As org-2ft is not used in conversion of 'epoch' time to times strings by > org, I'm not sure reverting the change gains us anything and will likely > adversely impact things like clock table calculations. What may be > justified is to add a note to the manual which clearly states that if > you are using org-2ft in your own code and plan to convert the result > back to a timestamp string, you need to ensure you explicitly tell the > function performing the conversion the input value is in UTC and not > local time. Yes, org-2ft documentation should be updated at the very least, assuming that this behavior is desired. > > I agree that enhancing org timestamps to include TZ information would be > a non-trivial chunk of work, I do feel it is the only solid way to > address all of these issues. I suspect there are still edge cases in > time related computations which fail even with the UTC approach and > there are certainly extensions which could benefit from explicit TZ > information (for example, taskgjuggler, which does time calculations for > planning and where such calculations can easily cross multiple daylight > savings TZ changes or include operations covering different timezones > etc). > > Tim > > Allen Li writes: > >> Apologies, my reply was partially omitted. My full reply follows >> >> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross wrote: >>> >>> Allen Li writes: >>> >>>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross wrote: >>>>> >>>>> My preferences would be >>>>> >>>>> 1. If a timestamp does not include the TZ, then assume the local TZ >>>>> 2. If a timestamp does include the TZ, honour that TZ >>>> >>>> Org mode does not support TZ in time strings and adding support isn't >>>> the topic of the current bug. >>>> >>> >>> I disagree. The root cause of the bug is due to a lack of clarity >>> regarding timezones. I also suspect this is also the basic cause of >>> issues relating to the use of timestamps/time strings and daylight >>> savings, especially when it comes to performing calculations etc. >> >> Unless I missed something, Org mode does not support explicit >> timezones in Org timestamps, and adding such support would take a >> large effort [1]. >> >> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html >> >> Unfortunately, I do not have time for the task. Thus, Org mode >> will have to make due with only Org timestamps without an >> explicit timezone unless a savior appears. >> >>> >>>> I want to emphasize the distinction between timestamps and time >>>> strings. Timestamps are a >>>> float value, Unix timestamps. Time strings are Org mode specific, >>>> like <2017-10-30 12:34:56> >>> >>> Org mode refers to what you are calling time strings as timestamps. In >>> reality, there is no difference - one is just a numeric representation >>> and the other is a string representation. It is good you have clarified >>> your definition to reduce confusion. However, I think the problems are >>> arising because of a lack of explicit TZ handling. >> >> There's a pretty significant difference. Unix timestamps are >> standardized and used widely by most computers and Emacs itself. >> Unix timestamps are time zone agnostics. >> >> Org timestamps (time strings, whatever) are used exclusively by >> Org mode. As Org timestamps do not support explicit timezones, >> they are not time zone agnostic. The interpretation of an Org >> timestamp depends on the time zone of the local machine, whereas >> the interpretation of a Unix timestamp does not depend on the >> time zone of the local machine. >> >>> >>>> >>>> Timestamps do not have timezone information since they describe an >>>> exact (well, minus leap seconds) >>>> point in time, the number of seconds after the epoch. >>>> >>> >>> A point in time is measured in number of seconds since epoch. However, >>> if you want to use local (wall) time to display that point, you have to >>> include TZ information when converting that number to a more >>> meaningful/usable (for humans) format. >> >> I'm not sure what you mean. Time zone information is provided by >> the local machine OS. If I pass a Unix timestamp to Emacs/the OS >> and tell it to format it in human readable format in the local >> time zone (the default behavior), then it will be done, without >> my having to attach time zone information anywhere. >> >>> The point 'now' for me is UTC+1100 >>> and for you (based on your previous post) UTC-0700, so our >>> representations in string format of this value will be different. Even >>> on a single machine, it is also relevant. For example, if I have two org >>> timestamps (your times strings) and I want to calculate the number of >>> hours between the two, I need to include TZ information as one timestamp >>> might be during daylight savings time and the other outside daylight >>> savings time. >> >> Ideally, except Org mode does not support including explicit TZ >> information in Org timestamps. Thus, Org mode can only current >> interpret Org timestamps according to some blessed time zone. >> Four months ago, that blessed time zone changed from the local >> machine's time zone to UTC. >> >>> Any calculation which fails to consider TZ information in >>> this case will be out by an hour. >>> >>> If we just take the simple solution and ignore TZ information, >> >> Org timestamps do not have TZ information. It's not a matter of >> not ignoring TZ information, support for TZ information would >> have to be added, which as I understand is non-trivial. >> >>> we will >>> break calculations which cross daylight savings 'boarders'. I also >>> suspect we will get inconsistent behaviour when integrating with other >>> systems (such as external calendars, ical integration or sharing >>> timestamp data with others etc). >> >> As long as we use Unix timestamps when integrating with other >> systems, there is no ambiguity. >> >>> >>>>> 3. If the timestamp does not include a time component, default to 0:00:0 >>>> >>>> This is what Org mode does >>>> >>>>> for the local TZ >>>> >>>> This is what Org mode used to do, now it interprets it as 00:00:00 UTC. >>>> >>>>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect. >>>>> >>>>> Rationale is that the user should have ability to fully control how >>>>> their timestamps are represented. However, adding the TZ is probably >>>>> just a pain for the majority of use cases, so defaulting to the local >>>>> (wall) TZ is OK provided you can override this consistently by adding >>>>> explicit TZ values. >>>>> >>>>> However, there is some devil in the details we need to work out. For >>>>> example, should we support both TZ names (like AEDT or Australia/Sydney) >>>>> and POSIX style +11/+11:00/+1100, should we add an option to tell org to >>>>> always add TZ info in timestamps which include time components and if >>>>> so, how complex will this need to be e.g. handle setting a future/past >>>>> timestamp which is in a different (daylight savings) offset and what >>>>> about the additional complexity in dealing with timestamp calculations >>>>> where dates might be in different offsets due to daylight savings - >>>>> while all quite possible, it does add significant complexity and this >>>>> may have adverse impact on performance. Not to say we shouldn't do it, >>>>> just that it will take significant work. >>>>> >>>>> I suspect just the first part won't have major impact - at least no more >>>>> than enforcing UTC in org-2ft. >>>> >>>> Org mode does not support TZ in time strings currently. While I would >>>> like such a feature very much, >>>> adding TZ support isn't the topic of the current bug, just fixing how >>>> time strings are interpreted. >>>> >>> >>> We could make the decision that all org timestamps are relative to the >>> local TZ (ignoring the daylight savings issue) and that would probably >>> be OK for most users >> >> I agree, and think that this makes more sense than using UTC. >> >>> but I feel the whole issue is due to a lack of >>> adequate/consistent TZ interpretation. >> >> Also agreed, but I cannot volunteer for the task of adding that >> unfortunately. >> >>> The problem with org-2ft is that >>> it forces the conversion into UTC when the input time string may >>> actually be in local time, resulting in a number which is out by >>> whatever the local offset is. >> >> Exactly >> >>> When you then convert back, your string >>> representation will be out by at least that amount (possibly more >>> depending on whether TZ info is applied in the conversion back to a >>> string). >> >> TZ info is not applied in that way going back, since Unix >> timestamps are time zone agnostic. It may affect how the string >> representation is formatted, but the time will not be even more >> incorrect. >> >> e.g., 2017-10-30T00:00:00+0000 vs 2017-10-30T07:00:00-0700. >> Different strings due to the time zone used, but they both >> describe the exact point of time as their input Unix timestamp. >> >>> >>> If it wasn't for the benefits of org files being essentially plain text >>> files, the easiest solution would possibly be to represent timestamps as >>> a number and use an overlay or similar to convert it to a human readable >>> value based on whatever the user sets as the timezone for >>> display/export. Unfortunately, this breaks one of the big benefits of >>> org-mode over other solutions i.e. the data is just plain text and can >>> be accessed/used by anything able to read plain text. >>> >>> Tim >>> >>> >>>>> >>>>> Tim >>>>> >>>>> P.S. when you start to think about it, it is easy to see how Java >>>>> screwed up this stuff so badly! >>> >>> >>> -- >>> Tim Cross > > > -- > Tim Cross