[BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[YE]

Exporting HTML file from Org-mode, f.i. with `org-html-export-to-html',
I expect to get a newline after the closing </html> tag, but it's not
added. I believe this used to work with previous versions of Org-mode.
Am I missing something?

Thanks.

Emacs  : GNU Emacs 29.1 (build 2, aarch64-apple-darwin22.6.0, NS appkit-2299.70 Version 13.5 (Build 22G74))
 of 2023-08-02
Package: Org mode version 9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

YE <yet@ego.team> writes:

> Exporting HTML file from Org-mode, f.i. with `org-html-export-to-html',
> I expect to get a newline after the closing </html> tag, but it's not
> added. I believe this used to work with previous versions of Org-mode.
> Am I missing something?

`org-html-template' never ever had a newline at the end of the generated
html file. May you please explain why missing newline is a problem?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[YE]

>
>> Exporting HTML file from Org-mode, f.i. with `org-html-export-to-html',
>> I expect to get a newline after the closing </html> tag, but it's not
>> added. I believe this used to work with previous versions of Org-mode.
>> Am I missing something?
>
> `org-html-template' never ever had a newline at the end of the generated
> html file.

Yes, I see that. It looks like there was something else that resulted in
a newline added at the end of the exported, including published, HTML
files (in previous Emacs versions).

> May you please explain why missing newline is a problem?

It's not necessarily a problem, since HTML parser implementations are
permissive. Still, a couple of arguments for adding it by default:

1. diff tools output "\ No newline at end of file" message when a
newline is missing. Supposedly, it's not optimal for default behavior.

2. POSIX.1-2017 states:
"3.206 Line
A sequence of zero or more non- <newline> characters plus a terminating
<newline> character."

3. Ending a line of text with a particular identifier is rather a
canonical approach.

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

YE <yet@ego.team> writes:

>> May you please explain why missing newline is a problem?
>
> It's not necessarily a problem, since HTML parser implementations are
> permissive. Still, a couple of arguments for adding it by default:
>
> 1. diff tools output "\ No newline at end of file" message when a
> newline is missing. Supposedly, it's not optimal for default behavior.

IMHO, I see no problem with that message.

> 2. POSIX.1-2017 states:
> "3.206 Line
> A sequence of zero or more non- <newline> characters plus a terminating
> <newline> character."

Does POSIX has anything to do with HTML?
Were you to show us HTML standard, it would be more convincing.

> 3. Ending a line of text with a particular identifier is rather a
> canonical approach.

Not ending is also not uncommon though. And causes no harm (no bug
reports during all the >10 years when ox-html is a thing).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Max Nikulin]

On 09/10/2023 18:29, Ihor Radchenko wrote:
> Not ending is also not uncommon though. And causes no harm (no bug 
> reports during all the >10 years when ox-html is a thing).

Ihor, it changed between Org-9.5 and 9.6.

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 09/10/2023 18:29, Ihor Radchenko wrote:
>> Not ending is also not uncommon though. And causes no harm (no bug 
>> reports during all the >10 years when ox-html is a thing).
>
> Ihor, it changed between Org-9.5 and 9.6.

Are you sure? I tried Org 9.5 and there is no newline.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Max Nikulin]

On 09/10/2023 18:46, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
> 
>> On 09/10/2023 18:29, Ihor Radchenko wrote:
>>> Not ending is also not uncommon though. And causes no harm (no bug
>>> reports during all the >10 years when ox-html is a thing).
>>
>> Ihor, it changed between Org-9.5 and 9.6.
> 
> Are you sure? I tried Org 9.5 and there is no newline.

Org mode version 9.5.5 (release_9.5.5 @ /usr/share/emacs/28.2/lisp/org/)

and the same for uncompiled release_9.5.5 from git

a test file containing just "test", C-c C-e h h.

     tail /tmp/test.org

next shell prompt starts at the beginning of next line. For a file 
exported by 9.6 shell prompt is placed after </html> on the same line.

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> Ihor, it changed between Org-9.5 and 9.6.
>> 
>> Are you sure? I tried Org 9.5 and there is no newline.
>
> Org mode version 9.5.5 (release_9.5.5 @ /usr/share/emacs/28.2/lisp/org/)
>
> and the same for uncompiled release_9.5.5 from git
>
> a test file containing just "test", C-c C-e h h.
>
>      tail /tmp/test.org
>
> next shell prompt starts at the beginning of next line. For a file 
> exported by 9.6 shell prompt is placed after </html> on the same line.

I can reproduce, but not with C-c C-e h H - there is no newline in the
export HTML buffer. If you have time, you may try to bisect.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Bruno Barbier]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]


Hi Ye, Ihor, Max,


The change seems to come from this commit:

   commit d7a55bbd537314d2776b082bd92a1a08b3edc84e
   Date:   Wed Sep 28 12:07:14 2022 +0800
   org-latex-export-to-latex: Do not suppress major modes in babel

It replaces 'write-file' with 'write-region', but, according to the
documentation of 'require-final-newline', 'write-region' ignores
'require-final-newline'.

See attached diff that reverts the problematic change.

In summary, before this commit, Emacs was fixing exported text files
(HTML, markdown, etc.) for free, adding a newline if needed for text
files.  It's not anymore.  I'm not sure what would be the best way to
fix this though.



Bruno


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Revert bug --]
[-- Type: text/x-patch, Size: 489 bytes --]

diff --git a/lisp/ox.el b/lisp/ox.el
index f8ccd2a9f..5ff105d3b 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -6893,7 +6893,7 @@ (defun org-export-to-file
           (with-temp-buffer
             (insert output)
             (let ((coding-system-for-write encoding))
-	      (write-region nil nil file)))
+	      (write-file file)))
           (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p output))
             (org-kill-new output))
           ;; Get proper return value.

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> The change seems to come from this commit:
>
>    commit d7a55bbd537314d2776b082bd92a1a08b3edc84e
>    Date:   Wed Sep 28 12:07:14 2022 +0800
>    org-latex-export-to-latex: Do not suppress major modes in babel
>
> It replaces 'write-file' with 'write-region', but, according to the
> documentation of 'require-final-newline', 'write-region' ignores
> 'require-final-newline'.

Thanks for narrowing down the cause!
We indeed cannot revert the commit - the change was made for a reason.

> See attached diff that reverts the problematic change.
>
> In summary, before this commit, Emacs was fixing exported text files
> (HTML, markdown, etc.) for free, adding a newline if needed for text
> files.  It's not anymore.  I'm not sure what would be the best way to
> fix this though.

I do not think that we need to honour `require-final-newline' during
export - this was (1) not documented; (2) caused inconsistency when
exporting with different user configs.

However, we indeed changed the existing behaviour.

AFAIU, `require-final-newline' is t by default, except binary data.
Also, Org export is not designed to generate binary output.
So, it should be OK to ensure the final newline manually, in
`org-export-to-file' - to minimize possible breakage.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> AFAIU, `require-final-newline' is t by default, except binary data.
> Also, Org export is not designed to generate binary output.
> So, it should be OK to ensure the final newline manually, in
> `org-export-to-file' - to minimize possible breakage.

See the attached tentative patch.
I am not merging it yet upstream (onto bugfix) in case if there are
objections.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-ox.el-org-export-to-file-Ensure-final-newline.patch --]
[-- Type: text/x-patch, Size: 1914 bytes --]

From 5bfe7dd32fb10af0540b271d1de27849507cb6d3 Mon Sep 17 00:00:00 2001
Message-ID: <5bfe7dd32fb10af0540b271d1de27849507cb6d3.1696936855.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 10 Oct 2023 14:19:53 +0300
Subject: [PATCH] * lisp/ox.el (org-export-to-file): Ensure final newline

The newline were added by default in the past, before d7a55bbd5.
Continue to add it to avoid changed behavior.

Reported-by: YE <yet@ego.team>
Link: https://orgmode.org/list/87wmvwklnq.fsf@localhost
---
 lisp/ox.el | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lisp/ox.el b/lisp/ox.el
index 94cc5a228..b63507606 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -6691,6 +6691,11 @@   (defun org-latex-export-to-latex
 		     ',ext-plist)))
 	       (with-temp-buffer
 		 (insert output)
+                 ;; Ensure final newline.  This is what was done
+                 ;; historically, when we used `write-file'.
+                 ;; Note that adding a newline is only safe for
+                 ;; non-binary data.
+                 (unless (bolp) (insert "\n"))
 		 (let ((coding-system-for-write ',encoding))
 		   (write-region (point-min) (point-max) ,file)))
 	       (or (ignore-errors (funcall ',post-process ,file)) ,file)))
@@ -6698,6 +6703,11 @@   (defun org-latex-export-to-latex
                        backend subtreep visible-only body-only ext-plist)))
           (with-temp-buffer
             (insert output)
+            ;; Ensure final newline.  This is what was done
+            ;; historically, when we used `write-file'.
+            ;; Note that adding a newline is only safe for
+            ;; non-binary data.
+            (unless (bolp) (insert "\n"))
             (let ((coding-system-for-write encoding))
 	      (write-region (point-min) (point-max) file)))
           (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p output))
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[YE]

> Bruno Barbier <brubar.cs@gmail.com> writes:

> The change seems to come from this commit:
>
>    commit d7a55bbd537314d2776b082bd92a1a08b3edc84e
>    Date:   Wed Sep 28 12:07:14 2022 +0800
>    org-latex-export-to-latex: Do not suppress major modes in babel
>
Thank you for finding the culprit.

> Ihor Radchenko <yantar92@posteo.net> writes:
> See the attached tentative patch.

Thanks, LGTM so far.

Re: [BUG] No newline at end of exported HTML file [9.6.6 (release_9.6.6 @ /Applications/MacPorts/Emacs.app/Contents/Resources/lisp/org/)]

[Ihor Radchenko]

YE <yet@ego.team> writes:

>> Ihor Radchenko <yantar92@posteo.net> writes:
>> See the attached tentative patch.
>
> Thanks, LGTM so far.

Applied, onto bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=51937d4b1

Fixed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>