Some links in online manual do not work

Some links in online manual do not work

[Tim Landscheidt]

Hi,

at https://orgmode.org/manual/HTML-Export.html, the links
for the first five (5) and the last two (2) subsections
work, the links for:

- "Headlines in HTML export"
  (https://orgmode.org/manual/Headlines-in-HTML-export.html)
- "Links in HTML export"
  (https://orgmode.org/manual/Links-in-HTML-export.html)
- "Tables in HTML export"
  (https://orgmode.org/manual/Tables-in-HTML-export.html)
- "Images in HTML export"
  (https://orgmode.org/manual/Images-in-HTML-export.html)
- "Math formatting in HTML export"
  (https://orgmode.org/manual/Math-formatting-in-HTML-export.html)
- "Text areas in HTML export"
  (https://orgmode.org/manual/Text-areas-in-HTML-export.html)

however all return "301 Moved Permanently", pointing back to
"HTML-Export.html".

The fact that all failing links are named
"Something-in-HTML-export.html" might suggest an issue with
the webserver configuration.

Tim

Re: Some links in online manual do not work

[Tim Cross]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> Hi,
>
> at https://orgmode.org/manual/HTML-Export.html, the links
> for the first five (5) and the last two (2) subsections
> work, the links for:
>
> - "Headlines in HTML export"
>   (https://orgmode.org/manual/Headlines-in-HTML-export.html)
> - "Links in HTML export"
>   (https://orgmode.org/manual/Links-in-HTML-export.html)
> - "Tables in HTML export"
>   (https://orgmode.org/manual/Tables-in-HTML-export.html)
> - "Images in HTML export"
>   (https://orgmode.org/manual/Images-in-HTML-export.html)
> - "Math formatting in HTML export"
>   (https://orgmode.org/manual/Math-formatting-in-HTML-export.html)
> - "Text areas in HTML export"
>   (https://orgmode.org/manual/Text-areas-in-HTML-export.html)
>
> however all return "301 Moved Permanently", pointing back to
> "HTML-Export.html".
>
> The fact that all failing links are named
> "Something-in-HTML-export.html" might suggest an issue with
> the webserver configuration.
>

This looks like the nginx case issue again.

I've looked at this and there does not seem to be any 'clean' way to fix
this which also doesn't have significant processing overhead or a
maintenance burden.

I wonder if it would be worthwhile adding an option to HTML export which
would force all link targets and exported filenames to lower case,
thereby avoiding issues on platforms and with web servers which have
different positions wrt case sensitivity?

Re: Some links in online manual do not work

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I wonder if it would be worthwhile adding an option to HTML export which
> would force all link targets and exported filenames to lower case,
> thereby avoiding issues on platforms and with web servers which have
> different positions wrt case sensitivity?

Could you please elaborate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some links in online manual do not work

[Tim Landscheidt]

Tim Cross <theophilusx@gmail.com> wrote:

>> at https://orgmode.org/manual/HTML-Export.html, the links
>> for the first five (5) and the last two (2) subsections
>> work, the links for:

>> - "Headlines in HTML export"
>>   (https://orgmode.org/manual/Headlines-in-HTML-export.html)
>> - "Links in HTML export"
>>   (https://orgmode.org/manual/Links-in-HTML-export.html)
>> - "Tables in HTML export"
>>   (https://orgmode.org/manual/Tables-in-HTML-export.html)
>> - "Images in HTML export"
>>   (https://orgmode.org/manual/Images-in-HTML-export.html)
>> - "Math formatting in HTML export"
>>   (https://orgmode.org/manual/Math-formatting-in-HTML-export.html)
>> - "Text areas in HTML export"
>>   (https://orgmode.org/manual/Text-areas-in-HTML-export.html)

>> however all return "301 Moved Permanently", pointing back to
>> "HTML-Export.html".

>> The fact that all failing links are named
>> "Something-in-HTML-export.html" might suggest an issue with
>> the webserver configuration.

> This looks like the nginx case issue again.

> I've looked at this and there does not seem to be any 'clean' way to fix
> this which also doesn't have significant processing overhead or a
> maintenance burden.

> I wonder if it would be worthwhile adding an option to HTML export which
> would force all link targets and exported filenames to lower case,
> thereby avoiding issues on platforms and with web servers which have
> different positions wrt case sensitivity?

Is the nginx configuration/webserver setup documented some-
where?

Tim

Re: Some links in online manual do not work

[Ihor Radchenko]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

>> This looks like the nginx case issue again.
>
>> I've looked at this and there does not seem to be any 'clean' way to fix
>> this which also doesn't have significant processing overhead or a
>> maintenance burden.
>
>> I wonder if it would be worthwhile adding an option to HTML export which
>> would force all link targets and exported filenames to lower case,
>> thereby avoiding issues on platforms and with web servers which have
>> different positions wrt case sensitivity?
>
> Is the nginx configuration/webserver setup documented some-
> where?

AFAIK, our nginx configs are not public, but Bastien may privately share
them if you are willing to help.

CCing Bastien.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some links in online manual do not work

[Tim Landscheidt]

Ihor Radchenko <yantar92@gmail.com> wrote:

>>> This looks like the nginx case issue again.

>>> I've looked at this and there does not seem to be any 'clean' way to fix
>>> this which also doesn't have significant processing overhead or a
>>> maintenance burden.

>>> I wonder if it would be worthwhile adding an option to HTML export which
>>> would force all link targets and exported filenames to lower case,
>>> thereby avoiding issues on platforms and with web servers which have
>>> different positions wrt case sensitivity?

>> Is the nginx configuration/webserver setup documented some-
>> where?

> AFAIK, our nginx configs are not public, but Bastien may privately share
> them if you are willing to help.

Willingness implies commitment, and I don't have that (nor
deeper nginx experience). :-) (That's one of the advantages
of free software for me: I can look at a bunch of problems,
pick the one I know the solution for and get instant grati-
fication :-).)

Tim

Re: Some links in online manual do not work

[Bastien Guerry]

Ihor Radchenko <yantar92@gmail.com> writes:

> AFAIK, our nginx configs are not public, but Bastien may privately share
> them if you are willing to help.

FWIW, I've shared the nginx.config here:
https://git.sr.ht/~bzg/worg/tree/master/item/nginx.conf

-- 
 Bastien

Re: Some links in online manual do not work

[Tim Landscheidt]

Bastien Guerry <bzg@gnu.org> wrote:

>> AFAIK, our nginx configs are not public, but Bastien may privately share
>> them if you are willing to help.

> FWIW, I've shared the nginx.config here:
> https://git.sr.ht/~bzg/worg/tree/master/item/nginx.conf

Thanks.  Assuming the relevant line is:

|     rewrite HTML-export.html HTML-Export.html permanent;

looking at
http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite
suggests that this will match any URL that contains the
string "HTML-export.html", i. e. the regular expression is
not anchored anywhere.

So if that regular expression is intended to match on URLs
that end in "/HTML-export.html", an (untested) fix would be
something along the lines of:

|     rewrite /HTML-export\.html$ HTML-Export.html permanent;

However I do not understand the need for these rewrites and
fixing the underlying issue is probably worthwhile.

Tim

Re: Some links in online manual do not work

[Ihor Radchenko]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> However I do not understand the need for these rewrites and
> fixing the underlying issue is probably worthwhile.

See the discussion in https://list.orgmode.org/orgmode/m2sfni5xcb.fsf@me.com/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some links in online manual do not work

[Max Nikulin]

On 08/10/2022 02:50, Tim Landscheidt wrote:
> Bastien Guerry <bzg@gnu.org> wrote:
> 
>> https://git.sr.ht/~bzg/worg/tree/master/item/nginx.conf
> 
> Thanks.  Assuming the relevant line is:
> 
> |     rewrite HTML-export.html HTML-Export.html permanent;
> 
> looking at
> http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite
> suggests that this will match any URL that contains the
> string "HTML-export.html", i. e. the regular expression is
> not anchored anywhere.
> 
> So if that regular expression is intended to match on URLs
> that end in "/HTML-export.html", an (untested) fix would be
> something along the lines of:
> 
> |     rewrite /HTML-export\.html$ HTML-Export.html permanent;

Bastien, I have not tried full configuration, but after a quick check I 
believe that it is a reasonable suggestion. It prevents 301 redirection 
from valid URLs like
https://orgmode.org/manual/Links-in-HTML-export.html
to
https://orgmode.org/manual/HTML-Export.html

In addition I would consider

     location /manual/ {
     }

around the rewrite directives to prevent unintentional false positives 
in e.g. worg.

P.S. Workarounds:
M-: (info "(org) Links in HTML Export")
https://orgmode.org/org.html#HTML-Export

Re: Some links in online manual do not work

[Bastien]

Hi Max,

Max Nikulin <manikulin@gmail.com> writes:

> Bastien, I have not tried full configuration, but after a quick check
> I believe that it is a reasonable suggestion. It prevents 301
> redirection from valid URLs like
> https://orgmode.org/manual/Links-in-HTML-export.html
> to
> https://orgmode.org/manual/HTML-Export.html

Indeed, thanks for the heads up.

> In addition I would consider
>
>     location /manual/ {
>     }
>
> around the rewrite directives to prevent unintentional false positives
> in e.g. worg.

Yes -- there is also location /guide/.

Not that if we use "/manual/" with the ending slash in the location
directive, I don't think the "/" is needed at the beginning of, say,
"HTML-Export.html" to disambiguate.

I completely reviewed the nginx configuration again and updated it
on Worg:  https://git.sr.ht/~bzg/worg/commit/601ebf48

Thanks,

-- 
 Bastien

Re: Some links in online manual do not work

[Max Nikulin]

On 11/10/2022 12:37, Bastien wrote:
> Max Nikulin writes:
> 
>> Bastien, I have not tried full configuration, but after a quick check
>> I believe that it is a reasonable suggestion. It prevents 301
>> redirection from valid URLs like
>> https://orgmode.org/manual/Links-in-HTML-export.html
>> to
>> https://orgmode.org/manual/HTML-Export.html
> 
> I don't think the "/" is needed at the beginning of, say,
> "HTML-Export.html" to disambiguate.
> 
> I completely reviewed the nginx configuration again and updated it
> on Worg:  https://git.sr.ht/~bzg/worg/commit/601ebf48

Bastien, unfortunately you have not fix the problem. Slash before 
HTML-export\.html is necessary. The page
https://orgmode.org/manual/HTML-Export.html
has a valid link to
https://orgmode.org/manual/Links-in-HTML-export.html
that is still inaccessible due to the active the rewrite rule

rewrite LaTeX-export\.html LaTeX-Export.html permanent;

curl -I https://orgmode.org/manual/Links-in-HTML-export.html
HTTP/1.1 301 Moved Permanently
Server: nginx/1.14.2
Date: Tue, 11 Oct 2022 06:55:28 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: HTML-Export.html

There should not be such issue for

rewrite /LaTeX-export\.html LaTeX-Export.html permanent;

Re: Some links in online manual do not work

[Bastien]

Max Nikulin <manikulin@gmail.com> writes:

> Bastien, unfortunately you have not fix the problem.

You're right, I updated the nginx conf again.

Let me know if you find any remaining problem.

Thanks!

-- 
 Bastien

Re: Some links in online manual do not work

[Max Nikulin]

On 11/10/2022 14:57, Bastien wrote:
> Max Nikulin writes:
> 
>> Bastien, unfortunately you have not fix the problem.
> 
> You're right, I updated the nginx conf again.
> 
> Let me know if you find any remaining problem.

         rewrite /HTML-export-commands\.html HTML-export-commands.html 
permanent;

Redirection loop

curl -I https://orgmode.org/manual/HTML-export-commands.html
HTTP/1.1 301 Moved Permanently
Server: nginx/1.14.2
Date: Tue, 11 Oct 2022 08:30:08 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: HTML-export-commands.html

Re: Some links in online manual do not work

[Bastien]

Max Nikulin <manikulin@gmail.com> writes:

>         rewrite /HTML-export-commands\.html HTML-export-commands.html
>         permanent;
>
> Redirection loop

Fixed, thanks!

-- 
 Bastien

Re: Some links in online manual do not work

[Max Nikulin]

On 11/10/2022 16:09, Bastien wrote:
> Max Nikulin writes:
> 
>>          rewrite /HTML-export-commands\.html HTML-export-commands.html
>>          permanent;
>>
>> Redirection loop
> 
> Fixed, thanks!

More:

rewrite /LaTeX-math-snippets\.html LaTeX-math-snippets.html permanent;
rewrite /Markup-for-Rich-Contents\.html Markup-for-Rich-Contents.html 
permanent;
rewrite /Presentation-and-Sorting\.html Presentation-and-Sorting.html 
permanent;
rewrite /Refile-and-Copy\.html Refile-and-Copy.html permanent;
rewrite /Refiling-and-Archiving\.html Refiling-and-Archiving.html permanent;
rewrite /Results-of-Evaluation\.html Results-of-Evaluation.html permanent;
rewrite /Setting-up-capture\.html Setting-up-capture.html permanent;
rewrite /Site-map\.html Site-map.html permanent;
rewrite /Sorting-of-agenda-items\.html Sorting-of-agenda-items.html 
permanent;
rewrite /Structure-of-Code-Blocks\.html Structure-of-Code-Blocks.html 
permanent;
rewrite /Table-of-Contents\.html Table-of-Contents.html permanent;
rewrite /The-capture-protocol\.html The-capture-protocol.html permanent;
rewrite /Using-the-Mapping-API\.html Using-the-Mapping-API.html permanent;
rewrite /Using-the-Property-API\.html Using-the-Property-API.html permanent;
rewrite /Working-with-Source-Code\.html Working-with-Source-Code.html 
permanent;

The guide:

         rewrite /Dates-and-Times\.html Dates-and-Times.html permanent;
         rewrite /Deadlines-and-Scheduling\.html 
Deadlines-and-Scheduling.html permanent;
         rewrite /Emphasis-and-Monospace\.html 
Emphasis-and-Monospace.html permanent;
         rewrite /Refile-and-Copy\.html Refile-and-Copy.html permanent;
         rewrite /Setting-up-capture\.html Setting-up-capture.html 
permanent;
         rewrite /Table-of-Contents\.html Table-of-Contents.html permanent;
         rewrite /Working-with-Source-Code\.html 
Working-with-Source-Code.html permanent;

Re: Some links in online manual do not work

[Bastien Guerry]

Max Nikulin <manikulin@gmail.com> writes:

> More:

Fixed, thanks!  

https://git.sr.ht/~bzg/worg/commit/408f05a0

-- 
 Bastien

Re: Some links in online manual do not work

[Max Nikulin]

On 12/10/2022 00:06, Bastien Guerry wrote:
> Max Nikulin writes:
>> More:
> 
> Fixed, thanks!

Unfortunately some redirection targets still respond with 301

301 https://orgmode.org/manual/Dates-and-Times.html
301 https://orgmode.org/manual/Deadlines-and-Scheduling.html
301 https://orgmode.org/manual/Emphasis-and-Monospace.html
301 https://orgmode.org/manual/Export-in-Foreign-Buffers.html
301 https://orgmode.org/manual/HTML-export-commands.html
301 https://orgmode.org/manual/Properties-and-Columns.html

If redirection directives were included as separate files then it would 
be possible to just check them by a command like

     awk '{ if ($NF >= 3) print $3; }' /tmp/manual.txt  |
         xargs --replace -- \
         curl --head --write-out '%{http_code} %{url_effective}\n' \
             --silent --show-error --output /dev/null \
             'https://orgmode.org/manual/{}'

and the same for the guide.

Original proposal to add redirections contained an s-expression with 
mappings. I would consider tracking it in the main Org repository. I 
believe, list of info nodes in the released manual should be added to it 
as known names. The idea is to make it easier to add redirections before 
new release. With such list as an input, a simple script could detect 
nodes absent in new release but existed in the earlier one. Another case 
is names appeared again making redirection rules obsolete.

Re: Some links in online manual do not work

[Bastien Guerry]

Max Nikulin <manikulin@gmail.com> writes:

> Unfortunately some redirection targets still respond with 301
>
> 301 https://orgmode.org/manual/Dates-and-Times.html
> 301 https://orgmode.org/manual/Deadlines-and-Scheduling.html
> 301 https://orgmode.org/manual/Emphasis-and-Monospace.html
> 301 https://orgmode.org/manual/Export-in-Foreign-Buffers.html
> 301 https://orgmode.org/manual/HTML-export-commands.html
> 301 https://orgmode.org/manual/Properties-and-Columns.html

Fixed, thanks.

> If redirection directives were included as separate files then it
> would be possible to just check them by a command like
>
>     awk '{ if ($NF >= 3) print $3; }' /tmp/manual.txt  |
>         xargs --replace -- \
>         curl --head --write-out '%{http_code} %{url_effective}\n' \
>             --silent --show-error --output /dev/null \
>             'https://orgmode.org/manual/{}'

https://git.sr.ht/~bzg/worg/tree/master/item/nginx.conf contains the
list of redirections -- the checks could be done from here, right?

> Original proposal to add redirections contained an s-expression with
> mappings. I would consider tracking it in the main Org repository. I
> believe, list of info nodes in the released manual should be added to
> it as known names. 

I'm not sure I understand.  Nothing should be added to the main Org
repository to fix a problem with the orgmode.org website, even if it
is a problem with the HTML manual as produced from org-mode.git.

> The idea is to make it easier to add redirections
> before new release. With such list as an input, a simple script could
> detect nodes absent in new release but existed in the earlier
> one. Another case is names appeared again making redirection rules
> obsolete.

I'm not in favor of going into this direction.

So far we provide an easy fix via rewrite rules to the problem created
with the change in the way Texinfo produces URLs.

Such rewrite rules are fine when they are like "rewrite THIS-URL.html
this-url.html" because stumbling on a dead link just because of lower
vs upper case letters is way too frustrating.

But more complex rewrite rules (from old manual nodes to new ones) is
IMHO calling for trouble. What if we split the "Properties and Column"
manual page into "Properties" and "Columns"?  Where to redirect?

-- 
 Bastien

Re: Some links in online manual do not work

[Max Nikulin]

On 12/10/2022 11:11, Bastien Guerry wrote:
> Max Nikulin writes:
> 
>> If redirection directives were included as separate files then it
>> would be possible to just check them by a command like
>>
>>      awk '{ if ($NF >= 3) print $3; }' /tmp/manual.txt  |
>>          xargs --replace -- \
>>          curl --head --write-out '%{http_code} %{url_effective}\n' \
>>              --silent --show-error --output /dev/null \
>>              'https://orgmode.org/manual/{}'
> 
> https://git.sr.ht/~bzg/worg/tree/master/item/nginx.conf contains the
> list of redirections -- the checks could be done from here, right?

It is not hard to copy text from nginx.conf to separate text files. 
Implementing nginx parser to filter only manual or just guide rules is 
more tricky. My opinion, the following structure is even more convenient 
to maintain:

nginx.conf:
# ...
     rewrite ^/list(.*) https://list.orgmode.org$1 permanent;

     location /manual {
         include manual-rewrite.inc;
     }
     location /guide {
         include guide-rewrite.inc;
     }
}
# ...

manual-rewrite.inc:

rewrite /Add_002don-packages\.html Add_002don-Packages.html permanent;
rewrite /Adding-export-back_002dends\.html 
Adding-Export-Back_002dends.html permanent;
rewrite /Adding-hyperlink-Types\.html Adding-Hyperlink-Types.html permanent;
# ...

I suggested it assuming tracking of changes in the manual. If you are 
against it then this list is almost static and regular check is less 
important. However it might catch removing of a section that is a 
redirection target.

>> Original proposal to add redirections contained an s-expression with
>> mappings. I would consider tracking it in the main Org repository. I
>> believe, list of info nodes in the released manual should be added to
>> it as known names.
> 
> I'm not sure I understand.  Nothing should be added to the main Org
> repository to fix a problem with the orgmode.org website, even if it
> is a problem with the HTML manual as produced from org-mode.git.

I do not insist. The idea was to put a file that is tightly bound to 
contents and edit history of manual close to the source of the docs 
since a script that can detect necessary changes requires list of HTML 
files generated from org-manual.org.

> But more complex rewrite rules (from old manual nodes to new ones) is
> IMHO calling for trouble. What if we split the "Properties and Column"
> manual page into "Properties" and "Columns"?  Where to redirect?

 From my point of view, any variant is better than 404. Alternatively 
redirection target may be an anchor to any of the new items in the table 
of contents.

Thank you for restoring rewrite rules. After earlier Tim's messages I 
was quite pessimistic believing that such attempt of improvement was buried.

Re: Some links in online manual do not work

[Tim Landscheidt]

Bastien Guerry <bzg@gnu.org> wrote:

>> More:

> Fixed, thanks!

> https://git.sr.ht/~bzg/worg/commit/408f05a0

Thanks to everyone involved, works for me now!

Tim