configurable link export

configurable link export

[Austin Frank]

First off, thank you to Carsten and David for org and org-publish. 
These tools have made it easier to get myself organized locally and 
online than it ever has been before.

I'd like to suggest a configuration option that influences the way links 
are exported in the org-export-as-* functions.

When I create local org files, I link to whatever files on my disk are 
relevant and useful.  When I publish those org files, some of the org 
links (like to other published org files) still work and make sense as 
html links.  Others (like links to local documents or directories) don't 
make any sense when published-- the resources they pointed to on the 
local system aren't on the remote filesystem that hosts the html pages.

To be more specific, I maintain a directory ~/notes/ and a directory 
~/blog/, each with a bunch of org files underneath it.  I then use 
org-publish to generate the html files, which are then uploaded to my 
website.  Crucially, this same directory structure is mirrored at my 
website, so local org links between those files work perfectly as html 
links when they are published and uploaded.

I'd love to be able to specify that I want all org links of 
[file:///home/aufrank/notes/*] and [file:///home/aufrank/blog/*] to be 
exported as full html links, and exclude org links to any other files on 
my filesystem at export time.

I think that David has established a good system for including and 
excluding files during publishing.  Org-publish first generates a list 
of files to publish based on regex matching of the extensions of files 
in a directory.  These are filtered through a regex-based exclude list, 
and then individual files can be added back in with an include list.

Applying this strategy to link export, I might want to have something 
like the following in my org-config.el:

(setq org-export-links-extension "org\\|txt"
       org-export-links-exclude   "~/*"
       org-export-links-include   ("~/notes/", "~/blog/"))

This would result in exporting all of the .org and .txt files in 
~/notes/ and ~/blog/ as working links in html files, but would ignore 
links in org files to anything else in my home directory during export.

There's certainly a question about how org links that are not exported 
as html links should be handled.  I would actually be fine with just 
exporting the double-bracketed notation, but I bet there are better 
ideas out there.

Thanks again for the org suite,
/au

Re: configurable link export

[Carsten Dominik]

Hi Austin,

On May 19, 2006, at 21:33, Austin Frank wrote:
>
> I'd like to suggest a configuration option that influences the way 
> links are exported in the org-export-as-* functions.

[...]

I think you are raising a very valid point here.  This is something 
that needs to be handled in an appropriate way.  Anybody knows how 
Emacs Wiki and Muse deal with such things?

Anyway, I have been thinking about your argumentation and come to the 
conclusion that this is an issue I would like to push almost entirely 
onto Davids table.  David, are you listening? :-)

Validation of links is really related to publishing.  From the point of 
"exporting", somefile.html is simply a different representation of 
something.org, by default this exported file lives in the same 
directory, so all file links, be they absolute or relative should work 
just fine.

However, publishing is about moving files to a publishing directory, 
and here the issue of validating links becomes really important.  The 
way I would like to go about implementing this is the following:

The org-publish functions should pass a special property to the html 
exporter.  The value of this property should be a function that can be 
used to validate a file link.  I would like to be able to call this 
function with a file name as the argument and maybe the directory of 
the original org file as an additional argument (might be unnecessary). 
  This function should then return if it will be safe to link to this 
file, and maybe even return a modified path to this file.

Org-publish would then be responsible to create the right function, and 
the user could even write her own function.

Besides minimizing the work for me :-), this would have obvious 
advantages.  Org-publish is fully aware of the entire structure of a 
project, so it will know which files will also be exported, to what 
locations, and it can therefore make a very accurate determination on 
if and how a given file will be accessible from the publishing 
location.

There are many ways org-publish could go about doing this.  One would 
be the way you, Austin, proposed, through special configuration setting 
- not variables, but properties then.  Another, more elegant way,  
would be that the structure and content of org-publish-project-alist is 
used to fully automatically determine the validity of a link.  not 
trivial to implement, but possible and a nice challenge.

David, what do you think?  Austin, do you think this could be a viable 
way?

- Carsten

>
> When I create local org files, I link to whatever files on my disk are 
> relevant and useful.  When I publish those org files, some of the org 
> links (like to other published org files) still work and make sense as 
> html links.  Others (like links to local documents or directories) 
> don't make any sense when published-- the resources they pointed to on 
> the local system aren't on the remote filesystem that hosts the html 
> pages.
>
> To be more specific, I maintain a directory ~/notes/ and a directory 
> ~/blog/, each with a bunch of org files underneath it.  I then use 
> org-publish to generate the html files, which are then uploaded to my 
> website.  Crucially, this same directory structure is mirrored at my 
> website, so local org links between those files work perfectly as html 
> links when they are published and uploaded.
>
> I'd love to be able to specify that I want all org links of 
> [file:///home/aufrank/notes/*] and [file:///home/aufrank/blog/*] to be 
> exported as full html links, and exclude org links to any other files 
> on my filesystem at export time.
>
> I think that David has established a good system for including and 
> excluding files during publishing.  Org-publish first generates a list 
> of files to publish based on regex matching of the extensions of files 
> in a directory.  These are filtered through a regex-based exclude 
> list, and then individual files can be added back in with an include 
> list.
>
> Applying this strategy to link export, I might want to have something 
> like the following in my org-config.el:
>
> (setq org-export-links-extension "org\\|txt"
>       org-export-links-exclude   "~/*"
>       org-export-links-include   ("~/notes/", "~/blog/"))
>
> This would result in exporting all of the .org and .txt files in 
> ~/notes/ and ~/blog/ as working links in html files, but would ignore 
> links in org files to anything else in my home directory during 
> export.
>
> There's certainly a question about how org links that are not exported 
> as html links should be handled.  I would actually be fine with just 
> exporting the double-bracketed notation, but I bet there are better 
> ideas out there.
>
> Thanks again for the org suite,
> /au
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
>

--
Carsten Dominik
Sterrenkundig Instituut "Anton Pannekoek"
Universiteit van Amsterdam
Kruislaan 403
NL-1098SJ Amsterdam
phone: +31 20 525 7477

Re: configurable link export

[David O'Toole]

Carsten Dominik <dominik@science.uva.nl> writes:

> Anyway, I have been thinking about your argumentation and come to the
> conclusion that this is an issue I would like to push almost entirely
> onto Davids table.  David, are you listening? :-)

Yes :-)

> way,  would be that the structure and content of
> org-publish-project-alist is used to fully automatically determine the
> validity of a link.  not trivial to implement, but possible and a nice
> challenge.

I have been thinking about inter-file links in preparation for
implementing some of the more advanced features for org-publish (page
rewriting, sitemap generation, etc) and I think this will be very
simple to straighten out.

All we need is for people to specify a property called :link-base that
is the prefix for a URL. For example, if :publishing-directory is
"/ssh:user@host:~/public_html/images", the user should probably set
the :link-base to "/images" or "http://foosite.org/images"

Given this one extra piece of information, then it is a simple matter
for the :link-transform function to actually generate proper URLs
(instead of simply validating them.) This would free projects from the
requirement that the web server must have the exact same directory
layout as your local project files.

This would allow, as you say, for link targets to be validated and
transformed automatically, given just the structure and contents of
org-publish-project-alist.

I will work on this and let you know how it goes. 

Austin--- how does this sound? 


>> When I create local org files, I link to whatever files on my disk
>> are relevant and useful.  When I publish those org files, some of
>> the org links (like to other published org files) still work and
>> make sense as html links.  Others (like links to local documents or
>> directories) don't make any sense when published-- the resources
>> they pointed to on the local system aren't on the remote filesystem
>> that hosts the html pages.
>>
>> To be more specific, I maintain a directory ~/notes/ and a directory
>> ~/blog/, each with a bunch of org files underneath it.  I then use
>> org-publish to generate the html files, which are then uploaded to
>> my website.  Crucially, this same directory structure is mirrored at
>> my website, so local org links between those files work perfectly as
>> html links when they are published and uploaded.
>>
>> I'd love to be able to specify that I want all org links of
>> [file:///home/aufrank/notes/*] and [file:///home/aufrank/blog/*] to
>> be exported as full html links, and exclude org links to any other
>> files on my filesystem at export time.
>>
>> I think that David has established a good system for including and
>> excluding files during publishing.  Org-publish first generates a
>> list of files to publish based on regex matching of the extensions
>> of files in a directory.  These are filtered through a regex-based
>> exclude list, and then individual files can be added back in with an
>> include list.
>>
>> Applying this strategy to link export, I might want to have
>> something like the following in my org-config.el:
>>
>> (setq org-export-links-extension "org\\|txt"
>>       org-export-links-exclude   "~/*"
>>       org-export-links-include   ("~/notes/", "~/blog/"))
>>
>> This would result in exporting all of the .org and .txt files in
>> ~/notes/ and ~/blog/ as working links in html files, but would
>> ignore links in org files to anything else in my home directory
>> during export.
>>
>> There's certainly a question about how org links that are not
>> exported as html links should be handled.  I would actually be fine
>> with just exporting the double-bracketed notation, but I bet there
>> are better ideas out there.
>>
>> Thanks again for the org suite,
>> /au
>>
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
>>
>
> --
> Carsten Dominik
> Sterrenkundig Instituut "Anton Pannekoek"
> Universiteit van Amsterdam
> Kruislaan 403
> NL-1098SJ Amsterdam
> phone: +31 20 525 7477
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>

-- 
Dave O'Toole
dto@gnu.org

Re: configurable link export

[Austin Frank]

David O'Toole wrote:
> All we need is for people to specify a property called :link-base that
> is the prefix for a URL. For example, if :publishing-directory is
> "/ssh:user@host:~/public_html/images", the user should probably set
> the :link-base to "/images" or "http://foosite.org/images"

This will definitely work for linking between projects.  But not all of
my links in a given org file are to files in projects.  If I have a link
[[file:///home/aufrank/my_movie.mpg][movie I made]], and that movie is
not part of a project, and there is no chance I am ever going to upload
that movie to my webserver, then I don't really want the link to be
published as <link-base>/my_movie.mpg.  I want to have a way to say "you
know what, don't bother trying to make this into a hyperlink.  I already
know it won't work as one."

> Given this one extra piece of information, then it is a simple matter
> for the :link-transform function to actually generate proper URLs
> (instead of simply validating them.) This would free projects from the
> requirement that the web server must have the exact same directory
> layout as your local project files.

This makes sense, and I would love to see it implemented.  But I'm still
not sure it addressed the problem described above.  If it does and I'm
just not getting it, please explain :)

> This would allow, as you say, for link targets to be validated and
> transformed automatically, given just the structure and contents of
> org-publish-project-alist.

Again, ideal for publishing projects and the links between them, but
does not address how to handle links to non-project local files.  My
vote is to export them as bracketed text, just as they're written in the
raw org file, but the really important thing is that they _not_ be
exported as hyperlinks if I tell org ahead of time that they're
guaranteed to be dead-on-upload.

Thanks,
/au

Re: configurable link export

[Austin Frank]

Carsten Dominik wrote:

> Anyway, I have been thinking about your argumentation and come to the
> conclusion that this is an issue I would like to push almost entirely
> onto Davids table.  David, are you listening? :-)

I seem to have a knack for setting David's table.  Hi dto!

> [...]
>
> David, what do you think?  Austin, do you think this could be a viable way?

The solution you propose looks workable to me, and I fully agree with
the distinction you make between the role of export and publish in this
case.  Thanks for taking the time to think it through.

/au

Re: configurable link export

[Carsten Dominik]

On May 20, 2006, at 15:01, David O'Toole wrote:
> I have been thinking about inter-file links in preparation for
> implementing some of the more advanced features for org-publish (page
> rewriting, sitemap generation, etc) and I think this will be very
> simple to straighten out.
>
> All we need is for people to specify a property called :link-base that
> is the prefix for a URL. For example, if :publishing-directory is
> "/ssh:user@host:~/public_html/images", the user should probably set
> the :link-base to "/images" or "http://foosite.org/images"
>
> Given this one extra piece of information, then it is a simple matter
> for the :link-transform function to actually generate proper URLs
> (instead of simply validating them.) This would free projects from the
> requirement that the web server must have the exact same directory
> layout as your local project files.

I don't quite understand yet how this will allow you to generate a 
different directory structure on the server.  Yes, within one project 
component this would allow you to use absolute path names for links.  
But if I am in an org-mode file and I am linking to 
"../../images/David.jpg", at least the current component does not know 
where the link files will end up.

My approach would have been the following:
1. Yes, enforce the same directory structure on the server.  I think 
this makes a lot of sense anyway.

2. When publishing a project, make a list of all files published by 
this project, including source files, images, you name it.  Cache that 
list somewhere during the publishing process.

3. Check any queries for safe links against this list.

As far as I can see this would remove the need to do *any* further 
customization.  Or am I overlooking something?

But it is also quite possible that I simply have not fully understood 
your idea about handling this.

- Carsten