[bug#48205] [PATCH] gnu: emacs-org: Make build reproducible

[bug#48205] [PATCH] gnu: emacs-org: Make build reproducible

[Morgan.J.Smith]

From: Morgan Smith <Morgan.J.Smith@outlook.com>

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use
autoloads instead
---
 gnu/packages/emacs-xyz.scm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index c950f116f9..a73b7959ef 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11243,6 +11243,12 @@ (define-public emacs-org
                (("org-release \"\"")
                 (string-append "org-release \"" ,version "\"")))
              #t))
+         (add-after 'fix-org-version 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." ".*.el")
+                 (("org-loaddefs.el") "org-autoloads.el"))
+             #t))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.31.1

[bug#48205] [PATCH] gnu: emacs-org: Make build reproducible

[Ludovic Courtès]

Hi,

Morgan.J.Smith@outlook.com skribis:

> From: Morgan Smith <Morgan.J.Smith@outlook.com>
>
> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use
> autoloads instead

[...]

> +         (add-after 'fix-org-version 'use-autoloads
> +           (lambda _
> +             (delete-file "org-loaddefs.el")
> +             (substitute* (find-files "." ".*.el")
> +                 (("org-loaddefs.el") "org-autoloads.el"))
> +             #t))

You can drop the trailing #t.

Could you explain in a comment in the phase what it does?  It’s not
clear to me how this relates to making the build bit-reproducible.

Remember that ‘find-files’ takes a regexp as its second argument; thus,
to match .el files, you need to write "\\.el$".

Could you send an updated patch?

Thanks,
Ludo’.

[bug#48205] [PATCH v2] gnu: emacs-org: Make build reproducible

[Morgan.J.Smith]

From: Morgan Smith <Morgan.J.Smith@outlook.com>

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use
autoloads instead. Copy autoloads into build directory.
---

Thanks for the suggestions! Sorry about the delay. Also sorry about the
comment. It's probably a little too long :/. I need to work on my english and
my ability to be concise.

 gnu/packages/emacs-xyz.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d954ec5bd..180b2709fe 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11761,6 +11761,27 @@ (define-public emacs-org
     (arguments
      `(#:phases
        (modify-phases %standard-phases
+         ;; Org ships with the file org-loaddefs.el that functions like our
+         ;; autoloads. Something about the interaction between the loaddefs
+         ;; and autoloads makes the build non-deterministic. We should opt for
+         ;; our autoloads system over the loaddefs since the loaddefs are
+         ;; autogenerated for the release tarball. org-loaddefs.el is
+         ;; mentioned by name in the source files so we have to do a blanket
+         ;; regex search and replace.
+         (add-after 'unpack 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." "\\.el$")
+               (("org-loaddefs.el") "org-autoloads.el"))))
+         ;; We create the autoloads directly in the install folder. During
+         ;; compilation org checks for the autoloads file so we need to copy
+         ;; it to the build directory.
+         (add-after 'make-autoloads 'copy-autoload-to-build-dir
+           (lambda* (#:key outputs #:allow-other-keys)
+             (copy-file (string-append
+                         (elpa-directory (assoc-ref outputs "out"))
+                         "/org-autoloads.el")
+                        "org-autoloads.el")))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.32.0

[bug#48205] [PATCH v2] gnu: emacs-org: Make build reproducible

[Ludovic Courtès]

Hi,

Morgan.J.Smith@outlook.com skribis:

> From: Morgan Smith <Morgan.J.Smith@outlook.com>
>
> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use
> autoloads instead. Copy autoloads into build directory.

[...]

>         (modify-phases %standard-phases
> +         ;; Org ships with the file org-loaddefs.el that functions like our
> +         ;; autoloads. Something about the interaction between the loaddefs
> +         ;; and autoloads makes the build non-deterministic. We should opt for
> +         ;; our autoloads system over the loaddefs since the loaddefs are
> +         ;; autogenerated for the release tarball. org-loaddefs.el is
> +         ;; mentioned by name in the source files so we have to do a blanket
> +         ;; regex search and replace.

I wonder about the interaction.

Are you sure using org-autoloads.el everywhere is enough to fix the
problem?  With current master, I see:

--8<---------------cut here---------------start------------->8---
$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org'
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
  differing files:
    /share/emacs/site-lisp/org-9.4.6/org-loaddefs.el
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.elc
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.el
--8<---------------cut here---------------end--------------->8---

And with diffoscope:

--8<---------------cut here---------------start------------->8---
$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org' --diff=diffoscope
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
 bayfront.guix.gnu.org  1.7MiB                          7.9MiB/s 00:00 [##################] 100.0%--- /tmp/guix-directory.vqBGkT
+++ /tmp/guix-directory.H4HEMX
│   --- /tmp/guix-directory.vqBGkT/share
├── +++ /tmp/guix-directory.H4HEMX/share
│ │   --- /tmp/guix-directory.vqBGkT/share/emacs
│ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs
│ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp
│ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp
│ │ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6
│ │ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6
│ │ │ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
│ │ │ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
│ │ │ │ │ @@ -1071,28 +1071,52 @@
│ │ │ │ │  ;;;### (autoloads nil "org-mouse" "org-mouse.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-mouse.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-mouse" '("org-mouse-")))
│ │ │ │ │  
│ │ │ │ │  ;;;***
│ │ │ │ │  
│ │ │ │ │ +;;;### (autoloads "actual autoloads are elsewhere" "org-num" "org-num.el"
│ │ │ │ │ +;;;;;;  (0 1 0 0))
│ │ │ │ │ +;;; Generated autoloads from org-num.el
│ │ │ │ │ +
│ │ │ │ │ +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-num" '("org-num-")))
│ │ │ │ │ +
│ │ │ │ │ +;;;***
│ │ │ │ │ +
│ │ │ │ │  ;;;### (autoloads nil "org-pcomplete" "org-pcomplete.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-pcomplete.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-pcomplete" '("org-" "pcomplete/org-mode/")))
│ │ │ │ │  
│ │ │ │ │  ;;;***
│ │ │ │ │  
│ │ │ │ │ +;;;### (autoloads "actual autoloads are elsewhere" "org-plot" "org-plot.el"
│ │ │ │ │ +;;;;;;  (0 1 0 0))
│ │ │ │ │ +;;; Generated autoloads from org-plot.el
│ │ │ │ │ +
│ │ │ │ │ +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-plot" '("org-plot")))
│ │ │ │ │ +
│ │ │ │ │ +;;;***
│ │ │ │ │ +

│ │ │ │ │  ;;;### (autoloads nil "org-protocol" "org-protocol.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-protocol.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-protocol" '("org-protocol-")))
│ │ │ │ │  
│ │ │ │ │  ;;;***
--8<---------------cut here---------------end--------------->8---

Thanks,
Ludo’.

[bug#48205] [PATCH] gnu: emacs-org: Don't use release tar

[Morgan.J.Smith]

From: Morgan Smith <Morgan.J.Smith@outlook.com>

* gnu/packages/emacs-xyz.scm (emacs-org):
[source]: Use git repository
[arguments]: Added phases to help with building
[native-inputs]: Added texinfo
---

So I still have no clue why the current emacs-org package is not reproducible,
but I made a package that is reproducible.  It also is building from the
development source so people can use package transformations easier and it
actually runs the tests.


 gnu/packages/emacs-xyz.scm | 46 +++++++++++++++++++++++++++-----------
 1 file changed, 33 insertions(+), 13 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 0cc66b6787..ab7b6fcf6a 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12439,24 +12439,44 @@ (define-public emacs-org
     (version "9.5")
     (source
      (origin
-       (method url-fetch)
-       (uri (string-append "https://elpa.gnu.org/packages/org-" version ".tar"))
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://git.savannah.gnu.org/git/emacs/org-mode.git")
+             (commit (string-append "release_" version))))
+       (file-name (git-file-name name version))
        (sha256
-        (base32 "16cflg5nms5nb8w86nvwkg49zkl0rvdhigkf4xpvbs0v7zb5y3ky"))))
+        (base32
+         "1gnzx53gkg2c6ljam31bvbibkra66lfx0w04jqmsv2gk8613527r"))))
     (build-system emacs-build-system)
     (arguments
-     `(#:phases
+     `(#:tests? #t
+       #:test-command '("make" "test-dirty")
+       #:phases
        (modify-phases %standard-phases
-         (add-after 'install 'install-documentation
+         (delete 'build)
+         (add-before 'check 'make
+           (lambda _
+             (invoke "make" (string-append "ORGVERSION=" ,version))))
+         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-                    (info-dir (string-append share "/info"))
-                    (doc-dir (string-append share "/doc/" ,name "-" ,version)))
-               (install-file "org.info" info-dir)
-               (install-file "orgguide.info" info-dir)
-               ;; XXX: "orgcard.pdf" is not built in Org 9.5.
-               ;; (install-file "orgcard.pdf" doc-dir)
-               ))))))
+             (substitute* "local.mk"
+               (("^prefix.*") (string-append "prefix = " (assoc-ref outputs "out")))
+               (("^lispdir.*")
+                (string-append "lispdir = " (elpa-directory (assoc-ref outputs "out")))))
+             (invoke "make" "install")))
+         (add-after 'unpack 'fix-tests
+           (lambda _
+             ;; These files are modified during testing
+             (make-file-writable "testing/examples/babel.org")
+             (make-file-writable "testing/examples/ob-awk-test.org")
+             (make-file-writable "testing/examples/ob-sed-test.org")
+             (make-file-writable "testing/examples/ob-shell-test.org")
+
+             ;; Specify where sh is
+             (substitute* "testing/examples/babel.org"
+               (("/bin/sh") (which "sh"))))))))
+    (native-inputs
+     `(("texinfo" ,texinfo)))
     (home-page "https://orgmode.org/")
     (synopsis "Outline-based notes management and organizer")
     (description "Org is an Emacs mode for keeping notes, maintaining TODO
-- 
2.34.0

bug#48205: [PATCH] gnu: emacs-org: Don't use release tar

[Nicolas Goaziou]

Hello,

Morgan.J.Smith@outlook.com writes:

> * gnu/packages/emacs-xyz.scm (emacs-org):
> [source]: Use git repository
> [arguments]: Added phases to help with building
> [native-inputs]: Added texinfo
> ---

I tweaked the definition a bit and applied your patch. Thank you.

There's an issue with the M-x org-version output, tho.

Regards,
-- 
Nicolas Goaziou

[bug#48205] [PATCH] gnu: emacs-org: Don't use release tar

[Michael Rohleder]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> There's an issue with the M-x org-version output, tho.

Looks like, this issue breaks emacs-org-re-reveal:

...
starting phase `build'
Checking /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/...
Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal-autoloads.el...
Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal.el...
Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/ox-re-reveal.el...
Invalid version syntax: ‘N/A’ (must start with a number)
command "/gnu/store/4w5rkbxb4l0q06bn07qjm5wps316s3sf-emacs-minimal-27.2/bin/emacs" "--quick" "--batch" "--eval=(eval '(progn (setq byte-compile-debug t) (byte-recompile-directory (file-name-as-directory \"/gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4\") 0 1)) t)" failed with status 255


-- 
There are only three things certain in programmers life: taxes, death and parsing text.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

[bug#48205] [PATCH] gnu: emacs-org: Preserve version information

[Morgan.J.Smith]

From: Morgan Smith <Morgan.J.Smith@outlook.com>

* gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from
deleting the version information
---

Fixed it!


 gnu/packages/emacs-xyz.scm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d4c682482..9ced8198c7 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12463,6 +12463,10 @@ (define-public emacs-org
                (("^lispdir.*")
                 (string-append "lispdir = "
                                (elpa-directory (assoc-ref outputs "out")))))
+             ;; The dependent targets for install perform cleanup that
+             ;; partially undoes our make phase
+             (substitute* "lisp/Makefile"
+               (("^install:.*") "install:\n"))
              (invoke "make" "install")))
          (add-after 'unpack 'fix-tests
            (lambda* (#:key inputs #:allow-other-keys)
-- 
2.34.0

[bug#48205] [PATCH] gnu: emacs-org: Preserve version information

[Nicolas Goaziou]

Hello,

Morgan.J.Smith@outlook.com writes:

> * gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from
> deleting the version information

Applied. Thank you.

Regards,
-- 
Nicolas Goaziou