Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir

Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir

[gnuric]

Hello,

I have a file s.org in ./ (current directory), which includes
another file s0.org living under ./media/s/:

./s.org:

#+TITLE: s
#+INCLUDE: "./media/s/s0.org"

./media/s/s0.org:

* s0
** s01
   file:media/s01_image.png

I want the org export to html to treat the file: paths in
./media/s/s0.org with respect to the directory of the ./s.org
file. In other words, I want export NOT TO TOUCH the file: paths
as written included files and effectively just paste the text of
s0.org verbatim in s.org (basically similar to what #include
directive in C preprocessor would do, e.g.). As of now, the above
s.org exports to s.html which has the file: link converted to

file:///home/user/media/s/media/s01_image.png

I want that link to instead become

file:///home/user/media/s01_image.png

which used to be the default behavior in a previous version of Org
mode (I don't recall which one). Is there a flag that could be
passed to #+INCLUDE: to change the 'relativeness' of the link
treatment when it comes to #+INCLUDE: directives?

I see in ox.el,

(defun org-export-expand-include-keyword (&optional included dir
footnotes)
  "Expand every include keyword in buffer.
Optional argument INCLUDED is a list of included file names along
with their line restriction, when appropriate.  It is used to
avoid infinite recursion.  Optional argument DIR is the current
working directory.  It is used to properly resolve relative
paths.  Optional argument FOOTNOTES is a hash-table used for
storing and resolving footnotes.  It is created automatically."

Is this DIR what's needed (to be given to #+INCLUDE: with a flag)
to change the link translation behavior?

Org mode version 9.4.4 (release_9.4.4)
GNU Emacs 27.2

Thanks,
Omid

[FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

gnuric@pm.me writes:

> I have a file s.org in ./ (current directory), which includes
> another file s0.org living under ./media/s/:
>
> ./s.org:
>
> #+TITLE: s
> #+INCLUDE: "./media/s/s0.org"
>
> ./media/s/s0.org:
>
> * s0
> ** s01
>    file:media/s01_image.png
>
> I want the org export to html to treat the file: paths in
> ./media/s/s0.org with respect to the directory of the ./s.org
> file. In other words, I want export NOT TO TOUCH the file: paths
> as written included files and effectively just paste the text of
> s0.org verbatim in s.org (basically similar to what #include
> directive in C preprocessor would do, e.g.). As of now, the above
> s.org exports to s.html which has the file: link converted to

This sounds like a reasonable request.
What we may do here is allowing a new parameter :verbatim

#+include: "/path/to/file" :verbatim t

This will disable footnote and link processing in
`org-export--prepare-file-contents'.

Though I am not sure if it is good enough to provide disable all/enable
all switch. Maybe even more fine-grained? - Just footnotes or just
links.

I am also not sure about other things
`org-export--prepare-file-contents' does, like indentation handling.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Hi Ihor,

> This sounds like a reasonable request.
> What we may do here is allowing a new parameter :verbatim

From a read of the original email, it sounds like a `:dir' parameter could also
solve this use case, and allow for a bit more flexibility.

All the best,
Timothy

-- 
Timothy (‘tecosaur’/‘TEC’), Org mode contributor.
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/tec>.

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[gnuric]

Hi Timothy,

Thanks for your reply.

"Timothy" <orgmode@tec.tecosaur.net> writes:

> Hi Ihor,
>
>> This sounds like a reasonable request.
>> What we may do here is allowing a new parameter :verbatim
>
> From a read of the original email, it sounds like a `:dir'
> parameter could also
> solve this use case, and allow for a bit more flexibility.
>
Is `:dir' a parameter to pass to `#+INCLUDE:'? I don't see it in
the manual:
https://orgmode.org/manual/Include-Files.html
> All the best,
> Timothy

Regards,
Omid

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[gnuric]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Hi,

I was wondering if anyone had any new ideas on this question (or an explanation on what Timothy meant by a :dir parameter).

Right now I'm having to use a search and replace back on the whole generated html buffer, which is fragile and inefficient.

Thanks,
Omid

-------- Original Message --------
On Feb 28, 2023, 1:34 PM, wrote:

> Hi Timothy, Thanks for your reply. "Timothy"  writes: > Hi Ihor, > >> This sounds like a reasonable request. >> What we may do here is allowing a new parameter :verbatim > > From a read of the original email, it sounds like a `:dir' > parameter could also > solve this use case, and allow for a bit more flexibility. > Is `:dir' a parameter to pass to `#+INCLUDE:'? I don't see it in the manual: https://orgmode.org/manual/Include-Files.html > All the best, > Timothy Regards, Omid

[-- Attachment #2: Type: text/html, Size: 985 bytes --]

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

gnuric@pm.me writes:

> I was wondering if anyone had any new ideas on this question (or an explanation on what Timothy meant by a :dir parameter).

Timothy meant that we can introduce a new #+INCLUDE option :dir that
will define custom path used to resolve relative paths inside links in
the included file.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[gnuric]

Hi Ihor,
Thanks for your reply and clarification on what Timothy meant.

1. Is there a general workaround that could be used as of now?
2. Is there something I can do to help with a :dir or similar option's development? I have some (very) basic knowledge of lisp.
3. If it helps, this behavior has changed since (at least) Org 9.1.9-65-g5e4542, which is the default distributed with GNU Emacs 26.3. With that Org, export to html took the links in the #+INCLUDE'd .org files as being relative to the includer's dir (i.e., the PARENT .org file's dir), which I consider 'verbatim' inclusion.

-Omid

"Ihor Radchenko" <yantar92@posteo.net> writes:

> gnuric@pm.me writes:
>
>> I was wondering if anyone had any new ideas on this question (or an explanation on what Timothy meant by a :dir parameter).
>
> Timothy meant that we can introduce a new #+INCLUDE option :dir that
> will define custom path used to resolve relative paths inside links in
> the included file.

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

gnuric@pm.me writes:

> Hi Ihor,
> Thanks for your reply and clarification on what Timothy meant.
>
> 1. Is there a general workaround that could be used as of now?

Nothing great. You may have to use a custom macro, but you will then
miss auto adjustment of heading levels and other #+INCLUDE features.

A bit more dramatic would be advising `org-export--update-included-link'
with :around advice #'ignore. That will skip link updates completely.

> 2. Is there something I can do to help with a :dir or similar option's development? I have some (very) basic knowledge of lisp.

You can look into `org-export--prepare-file-contents' that is doing all
the processing of the included files. It will have to be modified to
take the new options. The options are parsed by
`org-export-expand-include-keyword'.

If you have experience with other programming languages, things should
not be too hard - just follow the existing code.

> 3. If it helps, this behavior has changed since (at least) Org 9.1.9-65-g5e4542, which is the default distributed with GNU Emacs 26.3. With that Org, export to html took the links in the #+INCLUDE'd .org files as being relative to the includer's dir (i.e., the PARENT .org file's dir), which I consider 'verbatim' inclusion.

This behaviour is not documented and not defined.
The relative file link resolution was requested in
https://lists.gnu.org/r/emacs-orgmode/2019-02/msg00103.html and then
implemented in 931b7b8faf9.

Note that #+INCLUDE generally assumes regular Org files and the new
behaviour you stumbled upon makes more sense as the default.

Unfortunately, this change has not been announced in ORG-NEWS.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Samuel Wales]

off topic, but is it possible to leverage such a default into being
able to include a paragraph [or a section] from the current file?
istr this required specifying both dir [fixed with this default] and
the file basename.

here are some of the things i tried.

bad syntax no doubt.

# #+include: [[id:d3df9f9f-946d-44b6-9b58-b3a69e069458][About]] :only-contents t
# #+include: "#about" :only-contents t
# #+include: "*about" :only-contents t
# #+include: [*about] :only-contents t


On 4/3/23, Ihor Radchenko <yantar92@posteo.net> wrote:
> gnuric@pm.me writes:
>
>> Hi Ihor,
>> Thanks for your reply and clarification on what Timothy meant.
>>
>> 1. Is there a general workaround that could be used as of now?
>
> Nothing great. You may have to use a custom macro, but you will then
> miss auto adjustment of heading levels and other #+INCLUDE features.
>
> A bit more dramatic would be advising `org-export--update-included-link'
> with :around advice #'ignore. That will skip link updates completely.
>
>> 2. Is there something I can do to help with a :dir or similar option's
>> development? I have some (very) basic knowledge of lisp.
>
> You can look into `org-export--prepare-file-contents' that is doing all
> the processing of the included files. It will have to be modified to
> take the new options. The options are parsed by
> `org-export-expand-include-keyword'.
>
> If you have experience with other programming languages, things should
> not be too hard - just follow the existing code.
>
>> 3. If it helps, this behavior has changed since (at least) Org
>> 9.1.9-65-g5e4542, which is the default distributed with GNU Emacs 26.3.
>> With that Org, export to html took the links in the #+INCLUDE'd .org files
>> as being relative to the includer's dir (i.e., the PARENT .org file's
>> dir), which I consider 'verbatim' inclusion.
>
> This behaviour is not documented and not defined.
> The relative file link resolution was requested in
> https://lists.gnu.org/r/emacs-orgmode/2019-02/msg00103.html and then
> implemented in 931b7b8faf9.
>
> Note that #+INCLUDE generally assumes regular Org files and the new
> behaviour you stumbled upon makes more sense as the default.
>
> Unfortunately, this change has not been announced in ORG-NEWS.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

Samuel Wales <samologist@gmail.com> writes:

> off topic, but is it possible to leverage such a default into being
> able to include a paragraph [or a section] from the current file?
> istr this required specifying both dir [fixed with this default] and
> the file basename.

You can just include the same file.

For example, try the following "1.org" file:
----
#+include: "./1.org::*Heading" :only-contents t
* Heading
Text.
----

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Samuel Wales]

ah thanks.  i didn't mean to require specifying the current file.
that would be brittle to renaming hte file.  i was thinking maybe i
could use nil or some macro trick or a new feature of include to just
say current file.  not a pressing need though.

On 4/4/23, Ihor Radchenko <yantar92@posteo.net> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> off topic, but is it possible to leverage such a default into being
>> able to include a paragraph [or a section] from the current file?
>> istr this required specifying both dir [fixed with this default] and
>> the file basename.
>
> You can just include the same file.
>
> For example, try the following "1.org" file:
> ----
> #+include: "./1.org::*Heading" :only-contents t
> * Heading
> Text.
> ----
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[gnuric]

Thank you for your detailed reply, Ihor.

"Ihor Radchenko" <yantar92@posteo.net> writes:

> gnuric@pm.me writes:
>
>> Hi Ihor,
>> Thanks for your reply and clarification on what Timothy meant.
>>
>> 1. Is there a general workaround that could be used as of now?
>
> Nothing great. You may have to use a custom macro, but you will then
> miss auto adjustment of heading levels and other #+INCLUDE features.
>
> A bit more dramatic would be advising `org-export--update-included-link'
> with :around advice #'ignore. That will skip link updates completely.
>
>> 2. Is there something I can do to help with a :dir or similar
>> option's development? I have some (very) basic knowledge of lisp.
>
> You can look into `org-export--prepare-file-contents' that is doing all
> the processing of the included files. It will have to be modified to
> take the new options. The options are parsed by
> `org-export-expand-include-keyword'.
>
> If you have experience with other programming languages, things should
> not be too hard - just follow the existing code.
>
So, looking in ox.el (from Org 9.4.4 coming in GNU Emacs 27.2), it
appears that there is an optional INCLUDER arg that is supposed to do
what I'm looking for, except that there seems to be a bug somewhere:

Try the following:

-------------------- CORRECT --------------------
includer.org:
#+INCLUDE: "./a/included.org" :includer .

a/included.org:
[[./a/img.png]]

org-html-export-to-html results in the expected output in includer.html:
  <img src="./a/img.png" alt="img.png" />

Notice there is only ONE a/ in the img src path. Without :includer arg
we would get
  <img src="./a/a/img.png" alt="img.png" />

Notice there are TWO a/ (a/a/) in the img src path.
--------------------------------------------------

-------------------- WRONG --------------------
But if we just add a * section/heading to a/includer.org, e.g.,

a/includer.org:
* some
  [[./a/img.png]]

Now, org-html-export-to-html spits

#+BEGIN<sub>includer</sub>
...
  <img src="./a/img.png" alt="img.png" />
#+END<sub>includer</sub>


Why are these #+BEGIN_includer #+END_includer delimiters
are being produced?
-----------------------------------------------
>> 3. If it helps, this behavior has changed since (at least) Org
>> 9.1.9-65-g5e4542, which is the default distributed with GNU Emacs
>> 26.3. With that Org, export to html took the links in the
>> #+INCLUDE'd .org files as being relative to the includer's dir
>> (i.e., the PARENT .org file's dir), which I consider 'verbatim'
>> inclusion.
>
> This behaviour is not documented and not defined.
> The relative file link resolution was requested in
> https://lists.gnu.org/r/emacs-orgmode/2019-02/msg00103.html and then
> implemented in 931b7b8faf9.
>
> Note that #+INCLUDE generally assumes regular Org files and the new
> behaviour you stumbled upon makes more sense as the default.
>
> Unfortunately, this change has not been announced in ORG-NEWS.

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

gnuric@pm.me writes:

> So, looking in ox.el (from Org 9.4.4 coming in GNU Emacs 27.2), it
> appears that there is an optional INCLUDER arg that is supposed to do
> what I'm looking for, except that there seems to be a bug somewhere:

If you want to play around with Org code, please use the latest main.
See link to savannah at https://orgmode.org/ and
https://orgmode.org/worg/org-contribute.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow to #+include files verbatim without any processing (was: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir)

[Ihor Radchenko]

Timothy <orgmode@tec.tecosaur.net> writes:

> Hi Ihor,
>
>> This sounds like a reasonable request.
>> What we may do here is allowing a new parameter :verbatim
>
> From a read of the original email, it sounds like a `:dir' parameter could also
> solve this use case, and allow for a bit more flexibility.

Sure, but is this particular use-case common? Do people generally need
to leave the links as is, but adjust everything else?

`org-export--prepare-file-contents' does the following:
1. Indents everything from beginning of requested part of the file down
   to the first heading, if #+INCLUDE keyword itself is indented.
2. Updates the link paths
3. Removes the blank lines at the beginning/end of the included region
4. Adjusts heading levels, promoting when INCLUDE is inside another heading
5. Changes footnote references and definitions to make them unique and
   not clash with other files included

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir

[Max Nikulin]

On 26/02/2023 08:52, gnuric@pm.me wrote:
> (basically similar to what #include
> directive in C preprocessor would do, e.g.). As of now, the above
> s.org exports to s.html which has the file: link converted to
> 
> file:///home/user/media/s/media/s01_image.png

Doesn't cpp behave in a similar way?

grep --include=*.h -r '' h-0.h include-1/
h-0.h:
h-0.h:#include "include-1/h-1.h"
h-0.h:
h-0.h:extern int h0;
include-1/h-1.h:
include-1/h-1.h:#include "include-2/h-2.h"
include-1/h-1.h:
include-1/h-1.h:extern int h1;
include-1/include-2/h-2.h:
include-1/include-2/h-2.h:extern int h2;

cpp -nostdinc h-0.h
# 1 "h-0.h"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "h-0.h"

# 1 "include-1/h-1.h" 1

# 1 "include-1/include-2/h-2.h" 1

extern int h2;
# 3 "include-1/h-1.h" 2

extern int h1;
# 3 "h-0.h" 2

extern int h0;


So in a file residing in ./include-1/ reference to "include-2/h-2.h" 
means ./include-1/include-2/h-2.h, not ./include-2/h-2.h

Re: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir

[gnuric]

Hi Max,

Thanks for your reply.

"Max Nikulin" <manikulin@gmail.com> writes:

> On 26/02/2023 08:52, gnuric@pm.me wrote:
>> (basically similar to what #include
>> directive in C preprocessor would do, e.g.). As of now, the
>> above
>> s.org exports to s.html which has the file: link converted to
>>
>> file:///home/user/media/s/media/s01_image.png
>
> Doesn't cpp behave in a similar way?
>
> grep --include=*.h -r '' h-0.h include-1/
> h-0.h:
> h-0.h:#include "include-1/h-1.h"
> h-0.h:
> h-0.h:extern int h0;
> include-1/h-1.h:
> include-1/h-1.h:#include "include-2/h-2.h"
> include-1/h-1.h:
> include-1/h-1.h:extern int h1;
> include-1/include-2/h-2.h:
> include-1/include-2/h-2.h:extern int h2;
>
You have two #include directives in your example, which are
processed recursively according to the preprocessor rules. What I
had in mind was one #include preprocessor directive (similar to
one #+INCLUDE: 'directive' in Org) and a non-preprocessor
directive (`file:' in Org).
> cpp -nostdinc h-0.h
> # 1 "h-0.h"
> # 1 "<built-in>"
> # 1 "<command-line>"
> # 1 "h-0.h"
>
> # 1 "include-1/h-1.h" 1
>
> # 1 "include-1/include-2/h-2.h" 1
>
> extern int h2;
> # 3 "include-1/h-1.h" 2
>
> extern int h1;
> # 3 "h-0.h" 2
>
> extern int h0;
>
>
> So in a file residing in ./include-1/ reference to
> "include-2/h-2.h"
> means ./include-1/include-2/h-2.h, not ./include-2/h-2.h
Regards,
Omid

Re: Have export treat file: paths in INCLUDED file relative to the INCLUDING file's dir

[Max Nikulin]

On 01/03/2023 04:42, gnuric wrote:
>> h-0.h:#include "include-1/h-1.h"
>> h-0.h:
>> h-0.h:extern int h0;
>> include-1/h-1.h:
>> include-1/h-1.h:#include "include-2/h-2.h"
...
> You have two #include directives in your example, which are
> processed recursively according to the preprocessor rules. What I
> had in mind was one #include preprocessor directive (similar to
> one #+INCLUDE: 'directive' in Org) and a non-preprocessor
> directive (`file:' in Org).

My idea was that second include should simulate a link relative to the 
include-1/h-1.h file. I have no idea which feature of cpp may be used to 
distinguish behavior of include and link in respect to relative paths.

My expectation is that relative paths are resolved in respect to the 
file where they reside. I admit that in some cases included file may be 
a kind of template, so it may add e.g. a logo specific to each project. 
However, perhaps, I would prefer to define a macro.

Notice that CPP has another feature: include path, so while searching 
for a file directories from a list tried till file is found:

#include <file.h>

and

cpp -I /usr/include/lib1 -I /usr/include/lib2

Org does not have such concept.