Links Lost and Export-Report

Links Lost and Export-Report

[Stefan Vollmar]

[-- Attachment #1.1: Type: text/plain, Size: 3039 bytes --]

Hello,

I currently have a "showstopper-class" problem with links. I am using Org-mode release_7.01h-129-g3363a with Emacs 23.2.1. I have 

(require 'org-install)
(require 'org-id)

in my .emacs file.

This works fine:

(1) org-link-to-org-use-id is t and when I do a C-c l (org-store-link), a "a globally unique ID property" is generated for the current section - so far so good.
(2) I then change to another org-file and insert the link with C-c C-l (org-store-link), looking at the org source shows me that, indeed, the newly generated ID ("id:3A50..." in that example) has been used.
(3) When I do a C-c C-o (org-open-at-point) on the new link, the target org-file is opened at exactly the right place - nice!
(4) When I export the file as HTML, the link works exactly as expected: "<a href="some-file.html#ID-3A50...">

Here is the problem: I have two computers with very similar setups (Emacs and Org-mode: same versions; laptop and desktop) and synchronize org-files via a subversion repository. If I do a subversion update (sync the org-files to that computer) and then export to HTML on that computer, the previously working links do not work any more: there is no "#ID-..." in the link, in fact, the link is simply to the file that contains the link (not even the file I want to link to). Maybe Org-mode was not able to find the "id"?

I notice that there is no file referenced in the automatically generated 
"[[id:3A50...][My Link]]" link. How does Org-mode look for files that might contain a specific id? The org manual says "[...] and one that is globally unique and works even if the entry is moved from file to file" implying that there is a strategy for looking for suitable candidates. I assume that Org-mode will look in currently open buffers first, but even if I load the correct target file, HTML export will not create the proper link. Perhaps a caching problem (I use "global-auto-revert-mode")?

Maybe the above is a problem with my particular setup, but another and related problem is this:

Org-mode silently ignoring link problems and misspelled macro names is dangerous - it happened more than once in our case that HTML content went online with broken links and "{{{my-misspelled-macro()}}}" snippets. How about an (optional) report after generating more complex HTML contents: maybe it could be another org-file with links to offending Org code (unresolved links, unknown macro names) and an easy-to-parse summary line for batch jobs ("done in 5.6s, *** 2 errors found")? I am perfectly willing to help with this, unfortunately, my elisp skills are fairly basic and I fear this requires some intimate knowledge of the export process. 

Many thanks in advance.
Warm regards,
 Stefan
-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Head of IT group
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de







[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Links Lost and Export-Report

[Bernt Hansen]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> Hello,
>
> I currently have a "showstopper-class" problem with links. I am using Org-mode release_7.01h-129-g3363a with Emacs 23.2.1. I have 
>
> (require 'org-install)
> (require 'org-id)
>
> in my .emacs file.
>
> This works fine:
>
> (1) org-link-to-org-use-id is t and when I do a C-c l (org-store-link), a "a globally unique ID property" is generated for the current section - so far so good.
> (2) I then change to another org-file and insert the link with C-c C-l (org-store-link), looking at the org source shows me that, indeed, the newly generated ID ("id:3A50..." in that example) has been used.
> (3) When I do a C-c C-o (org-open-at-point) on the new link, the target org-file is opened at exactly the right place - nice!
> (4) When I export the file as HTML, the link works exactly as expected: "<a href="some-file.html#ID-3A50...">
>
> Here is the problem: I have two computers with very similar setups (Emacs and Org-mode: same versions; laptop and desktop) and synchronize org-files via a subversion repository. If I do a subversion update (sync the org-files to that computer) and then export to HTML on that computer, the previously working links do not work any more: there is no "#ID-..." in the link, in fact, the link is simply to the file that contains the link (not even the file I want to link to). Maybe Org-mode was not able to find the "id"?
>
> I notice that there is no file referenced in the automatically generated 
> "[[id:3A50...][My Link]]" link. How does Org-mode look for files that might contain a specific id? The org manual says "[...] and one that is globally unique and works even if the entry is moved from file to file" implying that there is a strategy for looking for suitable candidates. I assume that Org-mode will look in currently open buffers first, but even if I load the correct target file, HTML export will not create the proper link. Perhaps a caching problem (I use "global-auto-revert-mode")?
>
> Maybe the above is a problem with my particular setup, but another and related problem is this:
>
> Org-mode silently ignoring link problems and misspelled macro names is dangerous - it happened more than once in our case that HTML content went online with broken links and "{{{my-misspelled-macro()}}}" snippets. How about an (optional) report after generating more complex HTML contents: maybe it could be another org-file with links to offending Org code (unresolved links, unknown macro names) and an easy-to-parse summary line for batch jobs ("done in 5.6s, *** 2 errors found")? I am perfectly willing to help with this, unfortunately, my elisp skills are fairly basic and I fear this requires some intimate knowledge of the export process. 

I think you need to look in the ~/.emacs.d/.org-id-locations file.  This
maps link ids to files.

-Bernt

Re: Links Lost and Export-Report

[Stefan Vollmar]

[-- Attachment #1.1: Type: text/plain, Size: 2434 bytes --]

Dear Bernt,

On 18.08.2010, at 18:13, Bernt Hansen wrote:

> Stefan Vollmar <vollmar@nf.mpg.de> writes:
> 
>> Hello,
>> 
>> I currently have a "showstopper-class" problem with links. I am using Org-mode release_7.01h-129-g3363a with Emacs 23.2.1. I have 
>> 
>> (require 'org-install)
>> (require 'org-id)
>> 
>> in my .emacs file.
>> 
>> This works fine:
>> 
>> (1) org-link-to-org-use-id is t and when I do a C-c l (org-store-link), a "a globally unique ID property" is generated for the current section - so far so good.
>> (2) I then change to another org-file and insert the link with C-c C-l (org-store-link), looking at the org source shows me that, indeed, the newly generated ID ("id:3A50..." in that example) has been used.
>> (3) When I do a C-c C-o (org-open-at-point) on the new link, the target org-file is opened at exactly the right place - nice!
>> (4) When I export the file as HTML, the link works exactly as expected: "<a href="some-file.html#ID-3A50...">
>> 
>> Here is the problem: I have two computers with very similar setups (Emacs and Org-mode: same versions; laptop and desktop) and synchronize org-files via a subversion repository. If I do a subversion update (sync the org-files to that computer) and then export to HTML on that computer, the previously working links do not work any more: there is no "#ID-..." in the link, in fact, the link is simply to the file that contains the link (not even the file I want to link to). Maybe Org-mode was not able to find the "id"?

[...]

> I think you need to look in the ~/.emacs.d/.org-id-locations file.  This
> maps link ids to files.
> 
> -Bernt

I think it is very likely that this is part of problem - I synchronize the directories containing my org-files and have, so far, done nothing about "~/.emacs.d/.org-id-locations". 

What is the best way to deal with this?
- Synchronize?
- Delete?
- Force re-generation?

I have just taken a look at that file. It seems to map one id to one file. How does that help Org-mode if I rename an org-file that contains an id?

Many thanks for your help!
Warm regards,
 Stefan
-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Head of IT group
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de







[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Links Lost and Export-Report

[Bernt Hansen]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> Dear Bernt,
>
> On 18.08.2010, at 18:13, Bernt Hansen wrote:
>
>> Stefan Vollmar <vollmar@nf.mpg.de> writes:
>> 
>>> Hello,
>>> 
>>> I currently have a "showstopper-class" problem with links. I am using Org-mode release_7.01h-129-g3363a with Emacs 23.2.1. I have 
>>> 
>>> (require 'org-install)
>>> (require 'org-id)
>>> 
>>> in my .emacs file.
>>> 
>>> This works fine:
>>> 
>>> (1) org-link-to-org-use-id is t and when I do a C-c l (org-store-link), a "a globally unique ID property" is generated for the current section - so far so good.
>>> (2) I then change to another org-file and insert the link with C-c C-l (org-store-link), looking at the org source shows me that, indeed, the newly generated ID ("id:3A50..." in that example) has been used.
>>> (3) When I do a C-c C-o (org-open-at-point) on the new link, the target org-file is opened at exactly the right place - nice!
>>> (4) When I export the file as HTML, the link works exactly as expected: "<a href="some-file.html#ID-3A50...">
>>> 
>>> Here is the problem: I have two computers with very similar setups (Emacs and Org-mode: same versions; laptop and desktop) and synchronize org-files via a subversion repository. If I do a subversion update (sync the org-files to that computer) and then export to HTML on that computer, the previously working links do not work any more: there is no "#ID-..." in the link, in fact, the link is simply to the file that contains the link (not even the file I want to link to). Maybe Org-mode was not able to find the "id"?
>
> [...]
>
>> I think you need to look in the ~/.emacs.d/.org-id-locations file.  This
>> maps link ids to files.
>> 
>> -Bernt
>
> I think it is very likely that this is part of problem - I synchronize
> the directories containing my org-files and have, so far, done nothing
> about "~/.emacs.d/.org-id-locations".
>
> What is the best way to deal with this?
> - Synchronize?
> - Delete?
> - Force re-generation?

I think you can just regenerate the file with a call to

M-x org-id-update-id-locations

> I have just taken a look at that file. It seems to map one id to one
> file. How does that help Org-mode if I rename an org-file that
> contains an id?

This will probably make your ids and files out of sync so they don't
work for links in those files anymore.  Regenerating the list for all of
the files you have with id links should fix the problem.

I use the feature but don't rename my org files so I've never run into
this issue (if it is one).

Regards,
Bernt

Re: Links Lost and Export-Report

[Stefan Vollmar]

[-- Attachment #1: Type: text/plain, Size: 4323 bytes --]

Hello,

many thanks to Bernt Hansen for finding the root of my problem. 

Here is a recipe to unhappiness:

Start with two files a.org and b.org which are part of some files you have created with Org-mode in mind for efficently producing high quality HTML content. Set up Org-mode to use ids for links; do an org-store-link somewhere in file a.org, then an insert-store-link somewhere in file b.org. Enjoy short-lived happiness that everything works after exporting to HTML.

Then: delete ~/.emacs.d/.org-id-locations
This is a good way to simulate this common scenario: transfer your org files to another system, by preference with a version control system so you can be sure the files are identical on both systems and then continue working on that computer; keep in mind: this is the default state of affairs when working in a team! Also note that .org-id-locations uses absolute path names, so syncing that file across different user work stations will seldom work. 

Now observe that after exporting to HTML *none* of your ID based links work that refer to locations outside the originating org file. Moreover, observe that Org-mode certainly must have noticed something did not work and failed to mention it.

org-id-update-id-locations will not help you, unless (a) you remember to use it (preferably before you have already put HTML files with broken links online; this goes for your co-workers, too!), (b) files a.org and b.org happen to be agenda files or (c) happen to be mentioned in "org-id-locations" (this I learned from "describe function: org-id-update-id-locations"). None of the above is likely.

Org-mode for HTML publishing is an excellent tool (for lots and lots of other things, too, of course) and I think using globally unique ids is, in general, a great idea. This is why I get a bit emotional about this id link business (sorry).

Here are some suggestions:

- expand the use of ids, so that, in analogy to other links, the inserted link can also contain a file name if it refers to another org fileD, maybe like this: 
file-id:some-file.org#8329AEF0-4885-486A-A1B6-BB82B3E92184

- if an id is unknown, search all files in the current publishing project, using the same selection of org-files that will also be used for project-based publishing

- be verbose about links that could not be resolved, warn and potentially stop processing at that file (and while at it, also complain about unresolved macro definitions and finish with a summary line)

- mention in the org-manual that using ids is not recommended for using links between different files because they break so easily (this is hardly an option, is it?)

Let me emphasize that I am really generally a very happy org user and I am quite willing to contribute (as far as I can) to a solution to what I think is a problem with id-based links.

Warm regards,
 Stefan



On 18.08.2010, at 19:15, Bernt Hansen wrote:

> Stefan Vollmar <vollmar@nf.mpg.de> writes:
> 
>> [...]
>> 
>>> I think you need to look in the ~/.emacs.d/.org-id-locations file.  This
>>> maps link ids to files.
>>> 
>>> -Bernt
>> 
>> I think it is very likely that this is part of problem - I synchronize
>> the directories containing my org-files and have, so far, done nothing
>> about "~/.emacs.d/.org-id-locations".
>> 
>> What is the best way to deal with this?
>> - Synchronize?
>> - Delete?
>> - Force re-generation?
> 
> I think you can just regenerate the file with a call to
> 
> M-x org-id-update-id-locations
> 
>> I have just taken a look at that file. It seems to map one id to one
>> file. How does that help Org-mode if I rename an org-file that
>> contains an id?
> 
> This will probably make your ids and files out of sync so they don't
> work for links in those files anymore.  Regenerating the list for all of
> the files you have with id links should fix the problem.
> 
> I use the feature but don't rename my org files so I've never run into
> this issue (if it is one).




-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Head of IT group
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de







[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4409 bytes --]

Re: Links Lost and Export-Report

[Stefan Vollmar]

[-- Attachment #1.1: Type: text/plain, Size: 4312 bytes --]

Hello,

many thanks to Bernt Hansen for finding the root of my problem. 

Here is a recipe to unhappiness:

Start with two files a.org and b.org which are part of some files you have created with Org-mode in mind for efficently producing high quality HTML content. Set up Org-mode to use ids for links; do an org-store-link somewhere in file a.org, then an insert-store-link somewhere in file b.org. Enjoy short-lived happiness that everything works after exporting to HTML.

Then: delete ~/.emacs.d/.org-id-locations
This is a good way to simulate this common scenario: transfer your org files to another system, by preference with a version control system so you can be sure the files are identical on both systems and then continue working on that computer; keep in mind: this is the default state of affairs when working in a team! Also note that .org-id-locations uses absolute path names, so syncing that file across different user work stations will seldom work. 

Now observe that after exporting to HTML *none* of your ID based links work that refer to locations outside the originating org file. Moreover, observe that Org-mode certainly must have noticed something did not work and failed to mention it.

org-id-update-id-locations will not help you, unless (a) you remember to use it (preferably before you have already put HTML files with broken links online; this goes for your co-workers, too!), (b) files a.org and b.org happen to be agenda files or (c) happen to be mentioned in "org-id-locations" (this I learned from "describe function: org-id-update-id-locations"). None of the above is likely.

Org-mode for HTML publishing is an excellent tool (for lots and lots of other things, too, of course) and I think using globally unique ids is, in general, a great idea. This is why I get a bit emotional about this id link business (sorry).

Here are some suggestions:

- expand the use of ids, so that, in analogy to other links, the inserted link can also contain a file name if it refers to another org fileD, maybe like this: 
file-id:some-file.org#8329AEF0-4885-486A-A1B6-BB82B3E92184

- if an id is unknown, search all files in the current publishing project, using the same selection of org-files that will also be used for project-based publishing

- be verbose about links that could not be resolved, warn and potentially stop processing at that file (and while at it, also complain about unresolved macro definitions and finish with a summary line)

- mention in the org-manual that using ids is not recommended for using links between different files because they break so easily (this is hardly an option, is it?)

Let me emphasize that I am really generally a very happy org user and I am quite willing to contribute (as far as I can) to a solution to what I think is a problem with id-based links.

Warm regards,
Stefan

On 18.08.2010, at 19:15, Bernt Hansen wrote:

> Stefan Vollmar <vollmar@nf.mpg.de> writes:
> 
>> [...]
>> 
>>> I think you need to look in the ~/.emacs.d/.org-id-locations file.  This
>>> maps link ids to files.
>>> 
>>> -Bernt
>> 
>> I think it is very likely that this is part of problem - I synchronize
>> the directories containing my org-files and have, so far, done nothing
>> about "~/.emacs.d/.org-id-locations".
>> 
>> What is the best way to deal with this?
>> - Synchronize?
>> - Delete?
>> - Force re-generation?
> 
> I think you can just regenerate the file with a call to
> 
> M-x org-id-update-id-locations
> 
>> I have just taken a look at that file. It seems to map one id to one
>> file. How does that help Org-mode if I rename an org-file that
>> contains an id?
> 
> This will probably make your ids and files out of sync so they don't
> work for links in those files anymore.  Regenerating the list for all of
> the files you have with id links should fix the problem.
> 
> I use the feature but don't rename my org files so I've never run into
> this issue (if it is one).

-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Head of IT group
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de







[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Links Lost and Export-Report

[Sebastian Rose]

Bernt Hansen <bernt@norang.ca> writes:
> I think you can just regenerate the file with a call to
>
> M-x org-id-update-id-locations


This works for agenda files only, as well as files mentioned in
the variable `org-id-locations'.

Thus you need to rename the element in the list in your id-locations
file. E.G.:

Change ~/.emacs.d/.org-id-locations:
(("~/emacs/org/notes/index.org" "org:1234...") ...

sh$ mv ~/emacs/org/notes/index.org ~/emacs/org/notes/Linux/index.org


Change ~/.emacs.d/.org-id-locations accordingly:

(("~/emacs/org/notes/Linux/index.org" "org:1234...") ...



Shouldn't we have a command `org-move-file' or `org-rename-file' that takes
care of id-locations?



Best wishes

   Sebastian

Re: Re: Links Lost and Export-Report

[Stefan Vollmar]

[-- Attachment #1.1: Type: text/plain, Size: 2022 bytes --]

Dear Sebastian,

I think your mail and my last one were sent almost at the same time. 

You are right about the limitations of org-id-update-id-locations. My last point about fixing .org-id-locations is that this is not feasible if you are working in a team (syncing via a code repository) on several loosely coupled org files that constitute parts of a web site: keeping .org-id-locations up-to-date is just too complex in that case (and this is not such an excentric scenario).

To make org ids more robust, I suggested to: 

> - expand the use of ids, so that, in analogy to other links, the inserted link can also contain a file name if it refers to another org file, maybe like this: 
> file-id:some-file.org#8329AEF0-4885-486A-A1B6-BB82B3E9218


Most of the time, this would work (provided one uses relativ path names).

Warm regards,
 Stefan


On 18.08.2010, at 23:55, Sebastian Rose wrote:

> Bernt Hansen <bernt@norang.ca> writes:
>> I think you can just regenerate the file with a call to
>> 
>> M-x org-id-update-id-locations
> 
> 
> This works for agenda files only, as well as files mentioned in
> the variable `org-id-locations'.
> 
> Thus you need to rename the element in the list in your id-locations
> file. E.G.:
> 
> Change ~/.emacs.d/.org-id-locations:
> (("~/emacs/org/notes/index.org" "org:1234...") ...
> 
> sh$ mv ~/emacs/org/notes/index.org ~/emacs/org/notes/Linux/index.org
> 
> 
> Change ~/.emacs.d/.org-id-locations accordingly:
> 
> (("~/emacs/org/notes/Linux/index.org" "org:1234...") ...
> 
> 
> 
> Shouldn't we have a command `org-move-file' or `org-rename-file' that takes
> care of id-locations?
> 
> 
> 
> Best wishes
> 
>   Sebastian

-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Head of IT group
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de







[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4409 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode