RFC: Proposal for an Org Special Block for ox-html

RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 3188 bytes --]

Hello,

Yesterday, I came up with a way to generate the HTML details disclosure
fairly easily using Org Special Blocks. I implemented that in ox-hugo. But
I believe the same would be useful for ox-html too.

Here's the relevant code snippet:

    ((string= block-type "details")
      ;; Recognize Org Special blocks like:
      ;;   #+begin_details
      ;;   summary. This will be wrapped in <summary> and </summary>
      ;;   ---
      ;;   details
      ;;   #+end_details
      (let* ((is-open (member "open" (org-element-property :attr_html
special-block)))
             (str1 (org-blackfriday-special-block special-block contents
nil))
             ;; Insert a new-line before the closing </details> tag
             ;; for correct Markdown parsing for cases when the
             ;; Special Block content ends a code block. Without this
             ;; inserted newline, the Markdown converted content will
             ;; look like below, and Blackfriday won't parse it
             ;; correctly.
             ;;   ```emacs-lisp
             ;;   (message "a code block")
             ;;   ```</details>
             ;; A closing </p> tag is also added.. the opening <p>
             ;; tag is later added in the `str2' var if summary is
             ;; present, else in `str3' var.
             (str1 (replace-regexp-in-string "</details>\\'" "\n</p>\\&"
str1))
             ;; Detect the summary divider special string "---".  It
             ;; must begin at the beginning of a line.  Also ensure to
             ;; replace only the first match, if any.
             ;; Also add the opening <p> tag with "details" class
             ;; so that just as CSS rules can be set for summary
             ;; ("details summary"), they can be set for the details
             ;; portion following the <summary> too, using "details
             ;; .details".
             (str2 (replace-regexp-in-string
                    "^\\(---\\)\n\\(.\\|\n\\)*\\'"
                    "</summary><p class=\"details\">"
                    str1 nil nil 1))
             (has-summary (not (string= str1 str2)))
             str3)
        ;; (message "[DBG details/summary]: is-open:%S `%s' `%s'" is-open
str1 str2)
        (setq str3 (if has-summary
                       (replace-regexp-in-string "\\`<details>"
"\\&<summary>" str2)
                     (replace-regexp-in-string "\\`<details>" "\\&<p
class=\"details\">" str2)))
        (if is-open
            (replace-regexp-in-string "\\`\\(<details\\)>" "\\1 open>" str3)
          str3)))

I used "---" as the summary/details divider so that even if someone exports
that Org document to HTML without that support in ox-html, it exports as
Summary &mdash; Details.

See this for examples: https://ox-hugo.scripter.co/doc/details-and-summary/

Here is the full org-hugo-special-block:
https://github.com/kaushalmodi/ox-hugo/blob/14533c96195c90e417fcbde9d938650e83a39125/ox-hugo.el#L2239-L2363

If this feature is approved for addition to ox-html, I can make a formal
patch where instead of looking for "---", I look for the appropriate string
based on the value of (plist-get info :with-special-strings).

Thanks!

Kaushal
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 4723 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Aaron Ecay]

Hi Kaushal,

It seems like a good idea.  My two comments are:

- Remember that ox-html can export to HTML4, so the code would need to
  detect that case and have a sensible fallback
- The approach of looking for “magic” strings in the contents seems
  hackish.  What if the summary was treated as a caption?

#+caption: Open for details
#+begin_details
Many details here.
#+end_details

(Admittedly, the mismatch between “caption” and “summary” is not ideal,
but caption is the only suitable keyword that the syntax gives us...).

Another idea would be:

#+begin_details
#+summary: Open for details.
Many details here.
#+end_details

This approach would require a supporing change to be made to the
‘org-html-keyword’ function.

Just food for thought...

-- 
Aaron Ecay

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

Hi Aaron,

Thanks for the feedback.

On Thu, May 24, 2018 at 1:42 PM Aaron Ecay <aaronecay@gmail.com> wrote:

> Hi Kaushal,
>
> It seems like a good idea.  My two comments are:
>
> - Remember that ox-html can export to HTML4, so the code would need to
>   detect that case and have a sensible fallback
>

Yes, I think that would need to be handle just as ox-html does now.. use
div tags instead of details and summary tags.


> - The approach of looking for “magic” strings in the contents seems
>   hackish.  What if the summary was treated as a caption?
>

I also feel like that's a hack, but seems quite robust and functional at
least in my tests. The main reason to make summary part of details was to
no impose a one-paragraph limit on summaries.

Here's my pathological test for this feature:
https://ox-hugo.scripter.co/test/posts/details-and-summary/

#+caption: Open for details
> #+begin_details
> Many details here.
> #+end_details
>
> (Admittedly, the mismatch between “caption” and “summary” is not ideal,
> but caption is the only suitable keyword that the syntax gives us...).
>

I agree. caption doesn't go well with details.. looks like making a stew
out of whatever's in the fridge. Hope that bad metaphor makes sense :)

Another idea would be:
>
> #+begin_details
> #+summary: Open for details.
> Many details here.
> #+end_details
>

HTML allows multi-paragraph summaries.. (see my test link above). Expanding
on your idea.. what if we had another Special block with name summary?

#+begin_details
#+begin_summary
Open for details

More summary.
#+end_summary
Many details here.
#+end_details

It doesn't look as pretty as using the magic string "---", but it should
work.  But still not sure..




> Just food for thought...
>

I appreciate that.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 3174 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Hello Aaron,

On Thu, May 24, 2018 at 2:08 PM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> Expanding on your idea.. what if we had another Special block with name
> summary?
>
> #+begin_details
> #+begin_summary
> Open for details
>
> More summary.
> #+end_summary
> Many details here.
> #+end_details
>

From quick testing, this works (doesn't support the div tags for HTML4).
The good part is that the "summary" Special Block is automatically handled
by ox-html.

     ((string= block-type "details")
      ;; Recognize Org Special blocks like:
      ;;   #+begin_details
      ;;   #+begin_summary
      ;;   This is summary.
      ;;   #+end_summary
      ;;   Here are the details.
      ;;   #+end_details
      (let* ((is-open (member "open" (org-element-property :attr_html
special-block)))
             (str1 (org-blackfriday-special-block special-block contents
nil))
             ;; Insert a new-line before the closing </details> tag
             ;; for correct Markdown parsing for cases when the
             ;; Special Block content ends a code block. Without this
             ;; inserted newline, the Markdown converted content will
             ;; look like below, and Blackfriday won't parse it
             ;; correctly.
             ;;   ```emacs-lisp
             ;;   (message "a code block")
             ;;   ```</details>
             ;; A closing </p> tag is also added.. the opening <p>
             ;; tag is later added in the `str2' var if summary is
             ;; present, else in `str3' var.
             (str1 (replace-regexp-in-string "</details>\\'" "\n</p>\\&"
str1))
             ;; Detect the summary closing tag "</summary>".
             ;; Also add the opening <p> tag with "details" class
             ;; so that just as CSS rules can be set for summary
             ;; ("details summary"), they can be set for the details
             ;; portion following the <summary> too, using "details
             ;; .details".
             (str2 (replace-regexp-in-string
                    "\\(<summary>\\(.\\|\n\\)*</summary>\\)"
                    "\\1<p class=\"details\">"
                    str1)))
        ;; (message "[DBG details/summary]: is-open:%S `%s' `%s'" is-open
str1 str2)
        (if is-open
            (replace-regexp-in-string "\\`\\(<details\\)>" "\\1 open>" str2)
          str2)))

Also, thinking about HTML4 support, as details and summary tags were added
in HTML5, the "support" would be simply to and the (string= block-type
"details") condition with org-html-html5-fancy or equivalent. So then for
HTML4, details and summary Special Blocks will simply be div tags.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 4037 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Aaron Ecay]

Hi Kaushal,

2018ko maiatzak 24an, Kaushal Modi-ek idatzi zuen:

> 
> HTML allows multi-paragraph summaries.. (see my test link above). Expanding
> on your idea.. what if we had another Special block with name summary?

I wasnʼt thinking about multiparagraph summaries.  But actually, I
realize that your suggestion in response would work with only one small
change to a variable in ox-html.  Specifically, the following document
already yields the desired behavior with vanilla org (both giving the
proper behavior in html5, and having a fallback for earlier versions):

=====
* setup
#+begin_src emacs-lisp
  (add-to-list 'org-html-html5-elements "details")
  (add-to-list 'org-html-html5-elements "summary")
  (setq-local org-html-html5-fancy t)
  (setq-local org-html-doctype "html5")
#+end_src

* subtree
#+begin_details
#+begin_summary
Open for details

More summary.
#+end_summary
Many details here.
#+end_details
=====

Assuming there are no objections, we could make this tweak to
‘org-html-html5-elements’ in org core.  I think “syntactic sugar” like
your original proposal could be implemented as a before-parsing-hook
that transforms the --- into the above syntax.

Aaron

PS Once upon a time, I created ox-extras in contrib as a home for little
snippets like this hypothetical before-parsing-hook.  It never really
took off, but maybe itʼs just biding its time...

-- 
Aaron Ecay

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

Hi Aaron,

On Thu, May 24, 2018 at 2:36 PM Aaron Ecay <aaronecay@gmail.com> wrote:

>
> I wasnʼt thinking about multiparagraph summaries.  But actually, I
> realize that your suggestion in response would work with only one small
> change to a variable in ox-html.


Yes, the proposal though supports these 2 things specific to details
element:

1. Detecting "open" attribute via "#+attr_html: open".
2. Adding a wrapper <p class="details"> tag around the details portion
following the <summary> tag. Useful if user wants to set CSS rules for
"details .details".


> Assuming there are no objections, we could make this tweak to
> ‘org-html-html5-elements’ in org core.


I don't mind adding that to ox-html. Just that the details disclosure
implementation will be /slightly/ incomplete, but still very functional.

  I think “syntactic sugar” like
> your original proposal could be implemented as a before-parsing-hook
> that transforms the --- into the above syntax.
>

+1


> PS Once upon a time, I created ox-extras in contrib as a home for little
> snippets like this hypothetical before-parsing-hook.  It never really
> took off, but maybe itʼs just biding its time...
>

Sure thing :)
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2182 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Aaron Ecay]

Hi Kaushal,

2018ko maiatzak 24an, Kaushal Modi-ek idatzi zuen:
> 
> Yes, the proposal though supports these 2 things specific to details
> element:
> 
> 1. Detecting "open" attribute via "#+attr_html: open".

Is it important for open to be a “bare” attribute (not sure of the
official name) like “<details open>” as opposed to “<details
open="open">”?  The latter form is already supported (in fact the second
“open” can be any string).  Just add

#+attr_html: :open any-string-you-want

above the block.

> > 2. Adding a wrapper <p class="details"> tag around the details portion
> following the <summary> tag. Useful if user wants to set CSS rules for
> "details .details".

The only thing I know off the top of my head is that CSS can be quite
hairy, so I would not be surprised if this wrapper is sometimes needed.
And indeed, support for it would be missing.  Is it something that is
likely to crop up for other elements as well, such that it would be
desirable to support it in org core?  (This is at least partly me
wondering out loud, no need to specifically answer unless something
particularly occurs to you one way or the other.)

-- 
Aaron Ecay

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]

Hi Aaron,

I just went ahead and removed that "---" syntax from ox-hugo. Updated links:

1. https://ox-hugo.scripter.co/doc/details-and-summary/
2. https://ox-hugo.scripter.co/test/posts/details-and-summary/

(it was easy as I had committed that feature just yesterday.)

On Thu, May 24, 2018 at 3:08 PM Aaron Ecay <aaronecay@gmail.com> wrote:

>
> Is it important for open to be a “bare” attribute (not sure of the
> official name) like “<details open>” as opposed to “<details
> open="open">”?


I'm not so sure.. I just followed the documentation here:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details#Creating_an_open_disclosure_box


> The latter form is already supported (in fact the second
> “open” can be any string).  Just add
>
> #+attr_html: :open any-string-you-want
>

I know, but I did not test that.. just thought of doing as in that example
in the mozilla documentation.

That's why I am using (org-element-property :attr_html special-block) in
the code to get the raw values to #+attr_html.


> The only thing I know off the top of my head is that CSS can be quite
> hairy,


Heh, of course :)


> so I would not be surprised if this wrapper is sometimes needed.
> And indeed, support for it would be missing.



> Is it something that is
> likely to crop up for other elements as well, such that it would be
> desirable to support it in org core?


That would be great. I myself wasn't sure if I should bring that up to Org
core.. I had just implemented #+attr_css support for ox-hugo.

It looks like this:

#+attr_html: :class red-text
#+attr_css: :color red
- Red list item 1
- Red list item 2

Above will generate <style .red-text { color: red; } </style><div
class="red-text">..</div>, with that list in the div.

You can find many such examples in
https://kaushalmodi@github.com/kaushalmodi/ox-hugo/blob/master/test/site/content-org/all-posts.org,
and the implementation in
https://github.com/kaushalmodi/ox-hugo/blob/master/ox-blackfriday.el. Let
me know what you think if you have a chance to review those.


-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 3890 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Aaron Ecay]

Hi Kaushal,

2018ko maiatzak 24an, Kaushal Modi-ek idatzi zuen:

> That's why I am using (org-element-property :attr_html special-block) in
> the code to get the raw values to #+attr_html.

Iʼm not sure I made myself clear in the previous message.  In any case,
this org:

╭────
│ #+attr_html: :open t
│ #+begin_details
│ #+begin_summary
│ Open for details
│ 
│ More summary.
│ #+end_summary
│ Many details here.
│ #+end_details
╰────

exports to this HTML (using current-ish master with no additional
modifications beyond the tweak to org-html-html5-elements):

╭────
│ <details open="t">
│ <summary>
│ <p>
│ Open for details
│ </p>
│ 
│ <p>
│ More summary.
│ </p>
│ </summary>
│ <p>
│ Many details here.
│ </p>
│ </details>
╰────

which displays in the open state in a browser (in any event, in Chromium
66).  So I think what you want already exists for this feature.

> That would be great. I myself wasn't sure if I should bring that up to Org
> core.. I had just implemented #+attr_css support for ox-hugo.
> 
> It looks like this:
> 
> #+attr_html: :class red-text
> #+attr_css: :color red
> - Red list item 1
> - Red list item 2
> 
> Above will generate <style .red-text { color: red; } </style><div
> class="red-text">..</div>, with that list in the div.

What is wrong with:

#+attr_html: :style color:red;
- red list 1
- red list 2

?

That works in vanilla org today (exporting to html), without needing to
generate extra “style” and “div” elements.

> 
> You can find many such examples in
> https://kaushalmodi@github.com/kaushalmodi/ox-hugo/blob/master/test/site/content-org/all-posts.org,
> and the implementation in
> https://github.com/kaushalmodi/ox-hugo/blob/master/ox-blackfriday.el. Let
> me know what you think if you have a chance to review those.

I looked at the file briefly.  I had trouble determining what might
represent missing features in ox-{html,md}, and what was included to
work around quirks in a particular implementation of markdown.

On a broader level, both org and html have well-defined syntax, and
easy-to-work-with programmatic representations in lisp.  Markdown...does
not.  Since markdown-ing a valid HTML document should be a no-op
(AFAIK), I donʼt understand why you are bothering with markdown at all
in your usecase.  If it was me, I would just generate HTML from org and
skip the markdown step entirely.

It seems to me like you could get rid of ox-blackfriday, leaving behind
only a small ox-html-plus-plus containing whatever QOL improvements to
the vanilla html backend suited your taste (and that of ox-hugoʼs users).
If the set of QOL improvements is empty, then the custom backend would
disappear entirely.  Less code for the same features = maintenance win in
my book.  YMMV of course...

-- 
Aaron Ecay

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

Hi Aaron,

On Fri, May 25, 2018 at 10:19 AM Aaron Ecay <aaronecay@gmail.com>

> Iʼm not sure I made myself clear in the previous message.  In any case,
> this org:
>
> ╭────
> │ #+attr_html: :open t
> │ #+begin_details
> │ #+begin_summary
> │ Open for details
> │
> │ More summary.
> │ #+end_summary
> │ Many details here.
> │ #+end_details
> ╰────
>
> exports to this HTML (using current-ish master with no additional
> modifications beyond the tweak to org-html-html5-elements):
>
> ╭────
> │ <details open="t">
> │ <summary>
> ..│ </details>
> ╰────
>

No, sorry, you were clear. I just wanted to have the exported HTML to be
clean, i.e. not have that '="t"' portion. That said, I agree that it is
just easier to just do that in ox-html.


> which displays in the open state in a browser (in any event, in Chromium
> 66).  So I think what you want already exists for this feature.
>

Thanks for checking that.


What is wrong with:
>
> #+attr_html: :style color:red;
> - red list 1
> - red list 2
>
> ?
>

Nothing :) My implementation though needs those divs as I want to assign
classes and styles to Markdown blobs.


> That works in vanilla org today (exporting to html), without needing to
> generate extra “style” and “div” elements.
>

Agree.


I looked at the file briefly.  I had trouble determining what might
> represent missing features in ox-{html,md}, and what was included to
> work around quirks in a particular implementation of markdown.
>

Your assessment is correct.. ox-blackfriday is mainly to make things Just
Work with the Go Blackfriday markdown parser.. and thus all that lives in
that separate library. In addition, it adds the missing pieces that raw
HTML provides like assigning styling to various blocks, etc.


> I donʼt understand why you are bothering with markdown at all
> in your usecase.  If it was me, I would just generate HTML from org and
> skip the markdown step entirely.
>

I gave a lot of thought whether I should export to HTML or Markdown. In the
end, I am a fan of committing stuff that is readable. So it boiled down to
my personal preference. I prefer to commit Markdown instead of HTML. It's
very easy to track the corresponding changes from Org to Markdown, than
rather from Org to HTML.

Also, the ox-hugo exported Markdown is easy for collaborating documentation
with other folks (e.g. I contributed to official Hugo documentation using
ox-hugo exported Mardown.. I get to write the doc in Org, and they get the
Markdown doc source.. everyone is happy).

But then I also wanted to mask all the shortcoming of Markdown. So ox-hugo
helps bridge that gap by filling in HTML snippets only where needed. I hope
that makes sense.

It seems to me like you could get rid of ox-blackfriday, leaving behind
> only a small ox-html-plus-plus containing whatever QOL improvements to
> the vanilla html backend suited your taste (and that of ox-hugoʼs users).
>

That's a possibility. But as I said above, there isn't a strong reason to
have Hugo blog post source in HTML.

If the set of QOL improvements is empty, then the custom backend would
> disappear entirely.  Less code for the same features = maintenance win in
> my book.  YMMV of course...
>

Thanks.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 5028 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

Hi Aaron,

I really appreciate your feedback. That helped fine-tune my implementation
of details/summary.

I now moved that code to the ox-blackfriday library when I have this in
org-blackfriday-special-block:

It's basically QOL code:

1. Wraps the "details" portion in "<p class="details"> .. </p>".
2. Accepts only ":open t" as a "true condition" to keep the details widget
open by default. Detecting that inserts just "open" in the "details"
element. So ":open nil" or ":open foo" would not result in inserting any
open attr at all.

The modified contents and attr-str in the below snippet are later used as:
(format "<%s%s>\n%s\n</%s>"
                block-type attr-str contents block-type)

         ((string= block-type "details")
          (setq contents
                (let* ((str1 (concat contents "\n</p>"))
                       (str2 (replace-regexp-in-string
                              "<summary>\\(?:.\\|\n\\)*</summary>"
                              "\\&\n<p class=\"details\">"
                              str1))
                       (has-summary (not (string= str1 str2))))
                  (unless has-summary
                    (setq str2 (concat "<p class=\"details\">" str1)))
                  str2))
          ;; Insert the "open" attribute only if user has ":open t" in
          ;; "#+attr_html".
          (when (org-string-nw-p attr-str)
            (when (string-match "\\(?1:open\\(?2:=\"\\(?3:t\\)\"\\)\\)"
attr-str)
              (if (match-string 3 attr-str) ;if attr-str contains `open="t"'
                  (setq attr-str (replace-match "" nil nil attr-str 2))
                (setq attr-str (replace-match "" nil nil attr-str 1))))))

-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2702 bytes --]

Re: RFC: Proposal for an Org Special Block for ox-html

[Aaron Ecay]

Hi Kaushal,

[...]

> But then I also wanted to mask all the shortcoming of Markdown. So ox-hugo
> helps bridge that gap by filling in HTML snippets only where needed. I hope
> that makes sense.

It does indeed make sense, thanks for the explanation.  Now I understand
the context a bit better.

I added “summary” to ‘org-html-html5-elements’ in commit 5192e810a.  It
seems obviously correct (in fact, “details” was already included in that
variable).  So now summary/details should work for HTML5 export.  Thank
you for raising this issue.

It sounds like you have things well in hand for ox-{hugo,blackfriday}.
And I do not think there are any other shortcomings in org core that are
left outstanding from this discussion.

Thanks again,

-- 
Aaron Ecay