bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[mail]

1. Start emacs with emacs -Q
2. Evaluate the following:

(require 'org-mode)

(defun bad-indent ()
  (interactive)
  (insert "* ")
  (org-insert-link nil "http://www.example.com/foo/bar/baz/qux/abc123f56789" "something")
  (org-set-tags-to '("foo")))

3. Switch to an empty buffer
4. M-x org-mode
5. M-x bad-indent
6. The result looks like the following (except with an actual hyperlink):

* something	:foo:

This indentation is incorrect. If you follow the steps for `bad-indent'
interactively (or even using edebug), the indentation looks like:

* something								:foo:

Digging in some more, it looks like the problem is the `current-column'
call in `org-set-tags' (where it says 'setq c0 (current-column)', on
line 14620 of org.el in my distribution). With some instrumentation,
`current-column' seems to be giving different values in interactive mode
than programatically; I'm guessing it has something to do with the long
link target causing the value to be different.



In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.6)
 of 2015-09-09 on foutrelis
Windowing system distributor `The X.Org Foundation', version 11.0.11704000
System Description:	Arch Linux

Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
bad-indent
<XF86PowerOff> is undefined
bad-indent
Quit
You can run the command `org-insert-link' with C-c C-l
Making completion list... [2 times]
nil

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util help-fns mail-prsvr
mail-utils help-mode tabify image-file org-element org-rmail org-mhe
org-irc org-info org-gnus gnus-util org-docview doc-view jka-compr
image-mode dired cl-loaddefs cl-lib org-bibtex bibtex org-bbdb org-w3m
org org-macro org-footnote org-pcomplete pcomplete org-list org-faces
org-entities noutline outline easy-mmode org-version ob-emacs-lisp ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint comint
ansi-color ring ob-core ob-eval org-compat org-macs org-loaddefs
format-spec find-func cal-menu easymenu calendar cal-loaddefs time-date
tooltip electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
x-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 133855 6759)
 (symbols 48 24865 0)
 (miscs 40 81 451)
 (strings 32 33885 4710)
 (string-bytes 1 1030329)
 (vectors 16 16015)
 (vector-slots 8 456618 6166)
 (floats 8 108 286)
 (intervals 56 332 0)
 (buffers 960 13)
 (heap 1024 52252 997))

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Eli Zaretskii]

> From: mail@emanuel.industries
> Date: Mon, 02 Nov 2015 21:40:43 -0800
> 
> 
> 1. Start emacs with emacs -Q
> 2. Evaluate the following:
> 
> (require 'org-mode)
> 
> (defun bad-indent ()
>   (interactive)
>   (insert "* ")
>   (org-insert-link nil "http://www.example.com/foo/bar/baz/qux/abc123f56789" "something")
>   (org-set-tags-to '("foo")))
> 
> 3. Switch to an empty buffer
> 4. M-x org-mode
> 5. M-x bad-indent
> 6. The result looks like the following (except with an actual hyperlink):
> 
> * something	:foo:
> 
> This indentation is incorrect. If you follow the steps for `bad-indent'
> interactively (or even using edebug), the indentation looks like:
> 
> * something								:foo:
> 
> Digging in some more, it looks like the problem is the `current-column'
> call in `org-set-tags' (where it says 'setq c0 (current-column)', on
> line 14620 of org.el in my distribution). With some instrumentation,
> `current-column' seems to be giving different values in interactive mode
> than programatically; I'm guessing it has something to do with the long
> link target causing the value to be different.

Are you sure this isn't a problem with Org mode?  If not, can you tell
why you think so?

Thanks.

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Eli Zaretskii]

> From: Emanuel Evans <mail@emanuel.industries>
> Date: Tue, 03 Nov 2015 10:14:33 -0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Are you sure this isn't a problem with Org mode?  If not, can you tell
> > why you think so?
> 
> Oh, I think it could definitely be a problem with Org mode, I just
> thought this would be the right place to file a bug because it's a
> built-in package. Is there a better place to send a report?

Org mode has its own bug tracker, AFAIR.

> (I also think it's possible that it's a lower-level bug because
> `current-column' is changing behavior when using edebug, which seems
> like a bug either in `current-column' or edebug.)

If the Org developers say it's a core bug, please come back here (or
they should), with data that shows how to reproduce the problem,
ideally without Org at all.

Thanks.

P.S. Your email bounces, so sending me a private email is not very
wise...

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Glenn Morris]

Eli Zaretskii wrote:

> Org mode has its own bug tracker, AFAIR.

Nope, just a mailing list AFAIK. Assigning something to the debbugs
org-mode package (which I already did for this report) sends stuff there.

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Glenn Morris]

Eli Zaretskii wrote:

> Org mode has its own bug tracker, AFAIR.

Nope, just a mailing list AFAIK. Assigning something to the debbugs
org-mode package (which I already did for this report) sends stuff there.

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Glenn Morris <rgm@gnu.org> writes:

> Nope, just a mailing list AFAIK. Assigning something to the debbugs
> org-mode package (which I already did for this report) sends stuff there.

Great, is there anything else I should do from my end to make sure it
gets to the org mailing list? (I looked for the original post there and
didn't see it.)

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Apologies if this is a dupe; I think some of my posts aren't getting
through to the list (I just subscribed, so hopefully that will help).

Glenn Morris <rgm@gnu.org> writes:

> Nope, just a mailing list AFAIK. Assigning something to the debbugs
> org-mode package (which I already did for this report) sends stuff there.

Great, is there anything else I should do from my end to make sure it
gets to the org mailing list? (I looked for the original post there and
didn't see it.)

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Apologies if this is a dupe; I think some of my posts aren't getting
through to the list (I just subscribed, so hopefully that will help).

Glenn Morris <rgm@gnu.org> writes:

> Nope, just a mailing list AFAIK. Assigning something to the debbugs
> org-mode package (which I already did for this report) sends stuff there.

Great, is there anything else I should do from my end to make sure it
gets to the org mailing list? (I looked for the original post there and
didn't see it.)

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Nicolas Goaziou]

Hello,

Emanuel Evans <mail@emanuel.industries> writes:

> Great, is there anything else I should do from my end to make sure it
> gets to the org mailing list? (I looked for the original post there and
> didn't see it.)

You report reached the Org mailing list. Thank you.

After a quick glance, I think you are right: invisible characters are
not treated the same way in both cases. I'll investigate deeper soon.

Regards,

-- 
Nicolas Goaziou

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Nicolas Goaziou]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> After a quick glance, I think you are right: invisible characters are
> not treated the same way in both cases. I'll investigate deeper soon.

Fixed in d5767ad.

Re: bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Fixed in d5767ad.

Awesome, thanks for looking at this!

Re: bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
> Fixed in d5767ad.

Hi! It looks like it's fixed for `org-insert-link', but I ran across a
similar bug with `org-element-interpret-data' that seems to be not
entirely fixed:

(defun good-indent ()
  (interactive)
  (insert (org-element-interpret-data
           `(headline (:level 1
                              :title "something"
                              :tags ("foo"))))))

(defun bad-indent ()
  (interactive)
  (insert (org-element-interpret-data
           `(headline (:level 1
                              :title ((link (:raw-link "http://www.example.com/foo/bar/baz/qux/abc123f56789")
                                            "something"))
                              :tags ("foo"))))))

`good-indent' looks like:

* something                                                             :foo:

With org-mode 8.2.10, `bad-indent' looks like:

* something    :foo:

With the latest master version of org-mode (d6aafd7), `bad-indent' looks
like:

* something                                                      :foo:

(i.e., "almost but still not quite right").

I'm not very familiar with the org-element internals, but is there any
way to work around this for 8.2.10? (I'm writing a library that I'd like
to be compatible with Emacs 24.5.)

Thanks so much!

Re: bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Nicolas Goaziou]

Hello,

Emanuel Evans <mail@emanuel.industries> writes:

> Hi! It looks like it's fixed for `org-insert-link', but I ran across a
> similar bug with `org-element-interpret-data' that seems to be not
> entirely fixed:
>
> (defun good-indent ()
>   (interactive)
>   (insert (org-element-interpret-data
>            `(headline (:level 1
>                               :title "something"
>                               :tags ("foo"))))))
>
> (defun bad-indent ()
>   (interactive)
>   (insert (org-element-interpret-data
>            `(headline (:level 1
>                               :title ((link (:raw-link "http://www.example.com/foo/bar/baz/qux/abc123f56789")
>                                             "something"))
>                               :tags ("foo"))))))
>
> `good-indent' looks like:
>
> * something                                                             :foo:
>
> With org-mode 8.2.10, `bad-indent' looks like:
>
> * something    :foo:
>
> With the latest master version of org-mode (d6aafd7), `bad-indent' looks
> like:
>
> * something                                                      :foo:
>
> (i.e., "almost but still not quite right").
>
> I'm not very familiar with the org-element internals, but is there any
> way to work around this for 8.2.10? (I'm writing a library that I'd like
> to be compatible with Emacs 24.5.)

I don't consider it to be a bug.

`org-element-interpret-data' produces a string, which is expected to be
syntactically correct, but doesn't guarantee aesthetics. In particular,
`org-element-headline-interpreter' tries to align tags as accurately as
possible, but, in this case, fails to succeed as it would require to
introduce fontification in the process. This is out of the scope of the
function.

I think it is the duty of the caller, here `bad-indent', to align the
tags properly.


Regards,

-- 
Nicolas Goaziou

Re: bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I don't consider it to be a bug.
>
> `org-element-interpret-data' produces a string, which is expected to be
> syntactically correct, but doesn't guarantee aesthetics. In particular,
> `org-element-headline-interpreter' tries to align tags as accurately as
> possible, but, in this case, fails to succeed as it would require to
> introduce fontification in the process. This is out of the scope of the
> function.

Hmmm, that doesn't really make sense to me. Shouldn't the flow of

1. Edit a document with org-mode
2. (org-element-interpret-data (org-element-parse-buffer))

be as close to the original as possible? Seems like the fontification
aspect is just an implementation detail. In any case, I looked more into
the code and would be happy to submit a patch to change it if it makes
sense to you.

- Emanuel

Re: bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Nicolas Goaziou]

Emanuel Evans <mail@emanuel.industries> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> I don't consider it to be a bug.
>>
>> `org-element-interpret-data' produces a string, which is expected to be
>> syntactically correct, but doesn't guarantee aesthetics. In particular,
>> `org-element-headline-interpreter' tries to align tags as accurately as
>> possible, but, in this case, fails to succeed as it would require to
>> introduce fontification in the process. This is out of the scope of the
>> function.
>
> Hmmm, that doesn't really make sense to me. Shouldn't the flow of
>
> 1. Edit a document with org-mode
> 2. (org-element-interpret-data (org-element-parse-buffer))
>
> be as close to the original as possible?

It is already as close to the original as reasonable.

The guarantee is that it is syntactically equivalent to the original.
Yet, some information is lost in the process. For example, global
indentation is not meaningful syntax-wise and not retained in the parse
tree. That information is lost. Case (for blocks, properties...) is
another example. As you noticed, the same goes for tags' column, under
some circumstances.

> Seems like the fontification aspect is just an implementation detail.
> In any case, I looked more into the code and would be happy to submit
> a patch to change it if it makes sense to you.

The problem is that the change has a cost. Basically, you need to create
a new buffer, switch to Org mode, copy local variables from the original
buffer, insert the headline and align the tags properly according to the
current fontification rules.

OTOH, all you need to from the caller is to re-align the tags after
insertion.

I suggest to do the latter.


Regards,

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Nicolas Goaziou]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> After a quick glance, I think you are right: invisible characters are
>> not treated the same way in both cases. I'll investigate deeper soon.
>
> Fixed in d5767ad.

Closing this bug report, since my past self apparently fixed it.

Regards,

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Eli Zaretskii]

> From: Emanuel Evans <mail@emanuel.industries>
> Date: Tue, 03 Nov 2015 10:14:33 -0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Are you sure this isn't a problem with Org mode?  If not, can you tell
> > why you think so?
> 
> Oh, I think it could definitely be a problem with Org mode, I just
> thought this would be the right place to file a bug because it's a
> built-in package. Is there a better place to send a report?

Org mode has its own bug tracker, AFAIR.

> (I also think it's possible that it's a lower-level bug because
> `current-column' is changing behavior when using edebug, which seems
> like a bug either in `current-column' or edebug.)

If the Org developers say it's a core bug, please come back here (or
they should), with data that shows how to reproduce the problem,
ideally without Org at all.

Thanks.

P.S. Your email bounces, so sending me a private email is not very
wise...

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Eli Zaretskii <eliz@gnu.org> writes:

> Org mode has its own bug tracker, AFAIR.

OK, I'll submit there. Thanks!

> P.S. Your email bounces, so sending me a private email is not very
> wise...

Oops, sorry about that and thanks for the heads up.

bug#21818: 24.5; org-set-tags-to indentation problems when called programmatically

[Emanuel Evans]

Eli Zaretskii <eliz@gnu.org> writes:

> Org mode has its own bug tracker, AFAIR.

OK, I'll submit there. Thanks!

> P.S. Your email bounces, so sending me a private email is not very
> wise...

Oops, sorry about that and thanks for the heads up.