Library of babel help

Library of babel help

[Tim Cross]

Hi All,

in my attempt to fix up some issues on the Worg site, I'm finding there
is considerably more things broken than I initially realised. One of
these things is 'the library of babel". 

There is a link to the library-of-babel.org file on worg from within the
Emacs manual. This link is currently failing with a 404 error. However,
if you use the link embedded in the actual worg pages, you will get a
link to the library-of-babel.org file hosted in the git repository on
sourcehut, not worg. (it would appear that either the current build
process is not copying the *.org files to the web server or the web
server has not been configured to allow access to *.org files and always
redirects you to the *.html version).  

I believe the reason the link to the HTML file is failing is because the
conversion from *.org to *.html is failing because of invalid emacs lisp
in the org file source blocks. The problem is there are 5 references to
flet, which was deprecated in Emacs 24.3. 

To my questions -

1. Has anyone got a more recent version of the library-of-babel.org file
which has the fet references replaced with something more appropriate
(perhaps cl-flet, cl-letf or noflet)?

If nobody has, then I can have a go at fixing it up, but as I don't use
it, I will have trouble making sure it works correctly. 

2. I seem to recall that at one point, you could view the *.org sources
of pages on worg. I think this was a useful feature and we should
re-enable it. However, I suspect the nginx server will need some
tweaking. Is updating the server config to allow *.org access a
reasonable thing to do?

3. What should we do about the manual? We could (temporarily at least)
change the reference to point to the current (broken) version in the
sourcehut git repository or we can leave it until the file is again
being served by the worg server or .....

Re: Library of babel help

[Tim Cross]

Tim Cross <theophilusx@gmail.com> writes:

> Hi All,
>
> in my attempt to fix up some issues on the Worg site, I'm finding there
> is considerably more things broken than I initially realised. One of
> these things is 'the library of babel". 
>
> There is a link to the library-of-babel.org file on worg from within the
> Emacs manual. This link is currently failing with a 404 error. However,
> if you use the link embedded in the actual worg pages, you will get a
> link to the library-of-babel.org file hosted in the git repository on
> sourcehut, not worg. (it would appear that either the current build
> process is not copying the *.org files to the web server or the web
> server has not been configured to allow access to *.org files and always
> redirects you to the *.html version).  
>
> I believe the reason the link to the HTML file is failing is because the
> conversion from *.org to *.html is failing because of invalid emacs lisp
> in the org file source blocks. The problem is there are 5 references to
> flet, which was deprecated in Emacs 24.3. 
>
> To my questions -
>
> 1. Has anyone got a more recent version of the library-of-babel.org file
> which has the fet references replaced with something more appropriate
> (perhaps cl-flet, cl-letf or noflet)?
>
> If nobody has, then I can have a go at fixing it up, but as I don't use
> it, I will have trouble making sure it works correctly. 
>
> 2. I seem to recall that at one point, you could view the *.org sources
> of pages on worg. I think this was a useful feature and we should
> re-enable it. However, I suspect the nginx server will need some
> tweaking. Is updating the server config to allow *.org access a
> reasonable thing to do?
>
> 3. What should we do about the manual? We could (temporarily at least)
> change the reference to point to the current (broken) version in the
> sourcehut git repository or we can leave it until the file is again
> being served by the worg server or .....

Something else I forgot to mention just in case someone is looking at
the git repository. 

There are actually two library-of-babel.org files. One is in the root
folder and the other is in org-contrib/babel. The file
org-contrib/index.org actually links to the version in the root folder
and not the one in the same folder as it is located. Looking at them
both, I think the one in org-contrib/babel is older (certainly smaller
wiht fewer code blocks). Ironically, it doesn't have the flet issue. 

I plan to move the org-contrib/babel/library-of-babel.org file to the
archive directory to avoid confusion in future. 

Re: Library of babel help

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> in my attempt to fix up some issues on the Worg site, I'm finding there
> is considerably more things broken than I initially realised. One of
> these things is 'the library of babel". 
>
> There is a link to the library-of-babel.org file on worg from within the
> Emacs manual. This link is currently failing with a 404 error. However,
> if you use the link embedded in the actual worg pages, you will get a
> link to the library-of-babel.org file hosted in the git repository on
> sourcehut, not worg. (it would appear that either the current build
> process is not copying the *.org files to the web server or the web
> server has not been configured to allow access to *.org files and always
> redirects you to the *.html version).  
>
> I believe the reason the link to the HTML file is failing is because the
> conversion from *.org to *.html is failing because of invalid emacs lisp
> in the org file source blocks. The problem is there are 5 references to
> flet, which was deprecated in Emacs 24.3. 

The latest manual has the correct link. I assume that you have fixed it.
I guess that some of your questions do no apply anymore.

> 2. I seem to recall that at one point, you could view the *.org sources
> of pages on worg. I think this was a useful feature and we should
> re-enable it. However, I suspect the nginx server will need some
> tweaking. Is updating the server config to allow *.org access a
> reasonable thing to do?

Aren't they currently available? I can and I was always able to get the
org sources by changing the link from .html to .org.

Best,
Ihor

Re: Library of babel help

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> in my attempt to fix up some issues on the Worg site, I'm finding there
>> is considerably more things broken than I initially realised. One of
>> these things is 'the library of babel". 
>>
>> There is a link to the library-of-babel.org file on worg from within the
>> Emacs manual. This link is currently failing with a 404 error. However,
>> if you use the link embedded in the actual worg pages, you will get a
>> link to the library-of-babel.org file hosted in the git repository on
>> sourcehut, not worg. (it would appear that either the current build
>> process is not copying the *.org files to the web server or the web
>> server has not been configured to allow access to *.org files and always
>> redirects you to the *.html version).  
>>
>> I believe the reason the link to the HTML file is failing is because the
>> conversion from *.org to *.html is failing because of invalid emacs lisp
>> in the org file source blocks. The problem is there are 5 references to
>> flet, which was deprecated in Emacs 24.3. 
>
> The latest manual has the correct link. I assume that you have fixed it.
> I guess that some of your questions do no apply anymore.
>

Correct. I made some minimal changes to address the worst failing
aspects. However, a lot more work to do to really clean things up.
Bastein applied the patches I sent in and has given me write access.
However, I think I will probably create a new branch to do the
updates/changes and make them available to view (somewhere) as they will
be significant and I want a clean rollback strategy in case things go
wrong (especially as I will be re-defining the whole build process for
worg and probably making significant changes to the styling). . 

>> 2. I seem to recall that at one point, you could view the *.org sources
>> of pages on worg. I think this was a useful feature and we should
>> re-enable it. However, I suspect the nginx server will need some
>> tweaking. Is updating the server config to allow *.org access a
>> reasonable thing to do?
>
> Aren't they currently available? I can and I was always able to get the
> org sources by changing the link from .html to .org.
>

No, that doesn't work for me using either chrome or firefox. Chrome just
keeps switching back to the HTML url and firefox just hangs, returning
nothing. Bastien has sent me the nginx config being used, but I've not
yet had a look at it. My recollection is that there use to be a
link in the footer which you could use to get the source (org file). I
think an explicit link is better than relying on 'knowing' you can
change the extension to .org (even if it did work) as that is just a
little to close to 'secret guild knowledge' for my tastes.

Some of the documentation for worg implies the sources are supposed to
be in worg/source - allowing you to put a link to the sources (or any
source) inside the org file. This would likely also make the nginx
config cleaner as you can specify more accurately what is to be served
from where. 

Re: Library of babel help

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

>> The latest manual has the correct link. I assume that you have fixed it.
>> I guess that some of your questions do no apply anymore.
>
> Correct. I made some minimal changes to address the worst failing
> aspects. However, a lot more work to do to really clean things up.
> Bastein applied the patches I sent in and has given me write access.
> However, I think I will probably create a new branch to do the
> updates/changes and make them available to view (somewhere) as they will
> be significant and I want a clean rollback strategy in case things go
> wrong (especially as I will be re-defining the whole build process for
> worg and probably making significant changes to the styling). . 

Thanks again for working on this! WORG really needs more care.

>>> 2. I seem to recall that at one point, you could view the *.org sources
>>> of pages on worg. I think this was a useful feature and we should
>>> re-enable it. However, I suspect the nginx server will need some
>>> tweaking. Is updating the server config to allow *.org access a
>>> reasonable thing to do?
>>
>> Aren't they currently available? I can and I was always able to get the
>> org sources by changing the link from .html to .org.
>
> No, that doesn't work for me using either chrome or firefox. Chrome just
> keeps switching back to the HTML url and firefox just hangs, returning
> nothing. Bastien has sent me the nginx config being used, but I've not
> yet had a look at it. My recollection is that there use to be a
> link in the footer which you could use to get the source (org file). I
> think an explicit link is better than relying on 'knowing' you can
> change the extension to .org (even if it did work) as that is just a
> little to close to 'secret guild knowledge' for my tastes.

FYI, I am using Qutebrowser, which uses qtwebengine (5.15.5) under the
hood. https://orgmode.org/worg/org-contribute.org makes my Qutebrowser
displays download save dialogue.

The .org file is also downloaded using Firefox-100.0. No hanging.

Having a link to .org source in the footer is a good idea. Also, I
recall someone complaining that there is not .pdf version of Org manual.
I guess that similar feature could be useful in WORG.

Best,
Ihor

Serving .org files for worg (was: Re: Library of babel help)

[Max Nikulin]

On 30/06/2022 16:06, Tim Cross wrote:
> 
>> Aren't they currently available? I can and I was always able to get the
>> org sources by changing the link from .html to .org.
> 
> No, that doesn't work for me using either chrome or firefox. Chrome just
> keeps switching back to the HTML url and firefox just hangs, returning
> nothing. Bastien has sent me the nginx config being used, but I've not
> yet had a look at it.

It is rather strange

curl -I 'https://orgmode.org/worg/doc.org'
...
Content-Type: application/octet-stream

browsers should offer "save as" dialog in such case. It is possible to 
add Content-Disposition header to force "save" prompt, but I am unsure 
if it the best option. I would prefer some desktop-wide MIME handler, 
but there is not specific type for org. text/plain will be likely 
handled by browsers internally. Unsure if something like "text/x-org" is 
better since anyway custom configuration will be required.

Tim, did you face the problem with some particular file? Browsers might 
try to guess real MIME type from file content. If the problem is 
reproducible, you may experiment with "no sniff" header.

Re: Serving .org files for worg (was: Re: Library of babel help)

[Max Nikulin]

On 30/06/2022 22:45, Max Nikulin wrote:
> On 30/06/2022 16:06, Tim Cross wrote:
>>
>> No, that doesn't work for me using either chrome or firefox. Chrome just
>> keeps switching back to the HTML url and firefox just hangs, returning
>> nothing.

Tim, have you checked your Downloads folder? It may be full of .org 
files you tried to download from worg. Chromium just saves files unless 
it is explicitly configured to display save dialog. Behavior of Firefox 
changed around version 100 toward this direction (and I have to tweak 
setting to restore older behavior with /tmp for files intended to be 
opened in external applications).

Re: Serving .org files for worg (was: Re: Library of babel help)

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 30/06/2022 16:06, Tim Cross wrote:
>> 
>>> Aren't they currently available? I can and I was always able to get the
>>> org sources by changing the link from .html to .org.
>> No, that doesn't work for me using either chrome or firefox. Chrome just
>> keeps switching back to the HTML url and firefox just hangs, returning
>> nothing. Bastien has sent me the nginx config being used, but I've not
>> yet had a look at it.
>
> It is rather strange
>
> curl -I 'https://orgmode.org/worg/doc.org'
> ...
> Content-Type: application/octet-stream
>
> browsers should offer "save as" dialog in such case. It is possible to add
> Content-Disposition header to force "save" prompt, but I am unsure if it the
> best option. I would prefer some desktop-wide MIME handler, but there is not
> specific type for org. text/plain will be likely handled by browsers internally.
> Unsure if something like "text/x-org" is better since anyway custom
> configuration will be required.
>
> Tim, did you face the problem with some particular file? Browsers might try to
> guess real MIME type from file content. If the problem is reproducible, you may
> experiment with "no sniff" header.

There are 2 issues as I see it.

1. Just using the .org as the suffix of the url instead of the .html did
not work for me using two different browser. However, it did work for
ihor, so either I did something wrong or there is something in my setup
which is preventing that from working. Need to investigate further.
However, that is not my main issue.

My main issue is that having to know you can change the suffix from
.html to .org in order to get the source is insufficient. It won't be
obvious to many users and will only make some sense to experienced org
users. I think the link should be obvious and I think the server
configuration should be set so that accessing the .org url gives a
sensible result (i.e. prefer opening it as plain text to offering to
download).

My aim is to fix this, but I've not yet had time to even look at it. I
do have the server config file, so am able to review that and will sort
this out once time permits.

One of the changes I have started to implement is to standardise the
<head></head> and the page header/footer sections to make them
consistent across the worg site and get consistency with internal URLs
(some a relative, some are absolute), some have hard coded values, other
use site wide variables etc.

The initial aim is to make the site consistent and the build process
server independent. Ideally, anyone will be able to clone the repo, set
a document root and run the build process to create a fully working
local worg site. Little in this first stage is terribly contentious. 

Once this is done, the second stage is to work on improving the default
style and make the worg content easy to discover/browse. There is likely
to be varied opinions here and my intention will be to create a dev or
uat site where interested uses can have a look and provide
feedback/suggestions (as well as test). 

While I have made some progress, time is a little scarce at present and
I'll only have a few hours each weekend for the next few weeks to work
on this. However, I'm hoping some other commitments will scale down soon
and I'll have more time to devote to this task. It is quite rewarding,
just time consuming at present as I nail down all the moving parts!

Re: Serving .org files for worg (was: Re: Library of babel help)

[Max Nikulin]

On 01/07/2022 04:48, Tim Cross wrote:
> 
> 1. Just using the .org as the suffix of the url instead of the .html did
> not work for me using two different browser. However, it did work for
> ihor, so either I did something wrong or there is something in my setup
> which is preventing that from working. Need to investigate further.
> However, that is not my main issue.

I do not think that a chance of such event is noticeable, but you may 
observe different behavior because you tried to access different files. 
More likely you get files saved to Downloads.

Tim, likely you can not work with browsers unless significant 
customization is applied. Generally it is better to have virtual 
machines with popular environments and as little modifications as 
possible. Unfortunately just now I am preparing to move my VMs to 
another disk, so I would prefer to postpone setting up another instance 
suitable for such purpose.

> My main issue is that having to know you can change the suffix from
> .html to .org in order to get the source is insufficient. It won't be
> obvious to many users and will only make some sense to experienced org
> users. I think the link should be obvious and I think the server
> configuration should be set so that accessing the .org url gives a
> sensible result (i.e. prefer opening it as plain text to offering to
> download).

I do not mind to have explicit link to .org file. It may be placed to 
the header section with table of contents and the donation link. Such 
link may be like 
https://git.sr.ht/~bzg/worg/tree/master/item/org-artwork.org (I have no 
idea if it may add noticeable load to Drew DeVault's servers). However 
SourceHut does not highlight org sources currently. On the other hand it 
does not try to render them formatted, so sources are visible. Another 
problem is long lines. Unfortunately they are not wrapped and the 
following causes discrepancy with line numbers, so I can not suggest a fix:

/* main.min.112ca73b.css | https://git.sr.ht/static/main.min.112ca73b.css */

.code-view .highlight pre {
   white-space: pre-wrap;
}

> The initial aim is to make the site consistent and the build process
> server independent. Ideally, anyone will be able to clone the repo, set
> a document root and run the build process to create a fully working
> local worg site. Little in this first stage is terribly contentious.

I am afraid, to run local instance of worg, it would be necessary to 
explicitly specify MIME type for .org files otherwise they may be 
attributed to Lotus.

Re: Serving .org files for worg (was: Re: Library of babel help)

[Max Nikulin]

On 01/07/2022 04:48, Tim Cross wrote:
> 
> 1. Just using the .org as the suffix of the url instead of the .html did
> not work for me using two different browser. However, it did work for
> ihor, so either I did something wrong or there is something in my setup
> which is preventing that from working. Need to investigate further.
> However, that is not my main issue.

Tim, have you managed to achieve any progress with either nginx or with 
settings of your browsers?

I have tried some variants of desktop environments with almost default 
setting without issues similar to yours.

Org files are fetched to the Downloads folder without additional user 
actions (besides Firefox-91). It is not always obvious at first glance. 
With default settings:
- Chromium displays a bar at the bottom of the window.
- Firefox-102 (Ubuntu and Gnome, Arch and KDE) opens transient popup 
window. In Ububntu-22.04 firefox and chromium are installed as snap 
packages.
- Firefox-91 (Debian, Xfce) displays more traditional popup window to 
confirm download (somehow it guesses that it is an org-mode file even 
for application/octet-stream MIME-type).

If I change MIME type then Chromium displays internally both text/plain 
and text/x-org, Firefox asks for an external application for text/x-org 
while text/plain is displayed in the browser tab.

The following snippet should be added to nginx configuration to assign 
MIME type for .org files:

types {
         # Chromium opens text/x-org in the browser tab,
         # Firefox downloads files and offers to open in some other 
application.
         # text/x-org      org;
         text/plain      org;
}

See http://nginx.org/en/docs/http/ngx_http_core_module.html#types
It is assumed that default mime.types is included somewhere.

Re: Serving .org files for worg (was: Re: Library of babel help)

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 01/07/2022 04:48, Tim Cross wrote:
>> 1. Just using the .org as the suffix of the url instead of the .html did
>> not work for me using two different browser. However, it did work for
>> ihor, so either I did something wrong or there is something in my setup
>> which is preventing that from working. Need to investigate further.
>> However, that is not my main issue.
>
> Tim, have you managed to achieve any progress with either nginx or with settings of your
> browsers?
>
> I have tried some variants of desktop environments with almost default setting without
> issues similar to yours.
>
> Org files are fetched to the Downloads folder without additional user actions (besides
> Firefox-91). It is not always obvious at first glance. With default settings:
> - Chromium displays a bar at the bottom of the window.
> - Firefox-102 (Ubuntu and Gnome, Arch and KDE) opens transient popup window. In
>   Ububntu-22.04 firefox and chromium are installed as snap packages.
> - Firefox-91 (Debian, Xfce) displays more traditional popup window to confirm download
>   (somehow it guesses that it is an org-mode file even for application/octet-stream
>  MIME-type).
>
> If I change MIME type then Chromium displays internally both text/plain and text/x-org,
> Firefox asks for an external application for text/x-org while text/plain is displayed in
> the browser tab.
>
> The following snippet should be added to nginx configuration to assign MIME type for .org
> files:
>
> types {
>         # Chromium opens text/x-org in the browser tab,
>         # Firefox downloads files and offers to open in some other application.
>         # text/x-org      org;
>         text/plain      org;
> }
>
> See http://nginx.org/en/docs/http/ngx_http_core_module.html#types
> It is assumed that default mime.types is included somewhere.

Yes, I do have it working in my local dev environment. I have also made
some progress with improvements with both the nginx and worg layout, but
it isn't ready for 'prime time' yet. My testing with different browsers
did show variation across browsers and this is something I would like to
avoid. I also have come to a similar conclusion to you and think it is
about setting a mime handler for org files in nginx.  

Unfortunately, during this whole process, I had a major hardware failure
and have had to re-build my whole system. One of the things I did have
on my TODO list was to migrate from using Ubuntu to Fedora and while I
was going to do that later, given that I had to restore from backups
onto a new system, I decided to make the switch now.

While immensely happy with Fedora and pleased to be free of some of the
stuff I didn't like with current Ubuntu, it has been 20 years since I've
run a RH based distro, so there has been some learning curve to deal
with which has slowed down my progress with worg (especially getting up
to speed with SELinux, dnf and flatpak). However, it is progressing and
I hope to have something for everyone to look at sooner rather than
later.

I have now become a member of the SourceHut community and I think it
should be possible to deploy a dev/uat version of the new worg site to
their pages site. This should provide a relatively easy way for everyone
who is interested to have a look at the changes I'm making. I'm also
hoping this will also provide a workable 'dev' site for the main worg
repository so that anyone with the correct permissions can make changes,
deploy to the dev site to verify nothing is broken and then push it to
the production site. My hope is this will make people more relaxed and
confident in contributing while also ensuring a good user experience for
worg users. I still have a bit of work to do getting to grips wiht the
sh tools and understanding how they can help. There is definitely some
good stuff here, but there are still some rough edges you have to work
with and some different concepts/approaches I'm not yet familiar with. 

The good news is that I think all the necessary prerequisite stuff is
now working well and I can focus on the specifics of worg and the
various moving parts which make up orgmode.org!

Re: Serving .org files for worg

[Bastien]

Hi Max,

Max Nikulin <manikulin@gmail.com> writes:

> The following snippet should be added to nginx configuration to assign
> MIME type for .org files:
>
> types {
>         # Chromium opens text/x-org in the browser tab,
>         # Firefox downloads files and offers to open in some other
>           application.
>         # text/x-org      org;
>         text/plain      org;
> }

this is now done -- thanks for the reminder.

-- 
 Bastien