bug#74413: [PATCH] Allow to store and read repository information of VCS builds

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 3713 bytes --]

Tags: patch


Emacs has the feature to read the repository version and
branch if git is installed during the build and afterwards if also
the sources including the VCS repository is present.

For the Android builds the feature was added to store and read the
information mentioned above in a special version file.

This patch reuses that mechanism so it can be reused on other platforms
to use for the same reasons its available for Android and to also be
able to use the information on CI workers without git installed and/or a
full source checkout.

The things I'm not sure about for this patch are:
- Is the generating of the version file in the right place in
Makefile.in
- Is the data directory the right place to store the file
- Should the creation of the version file be shared between the Android
  builds and the other platforms



In GNU Emacs 31.0.50 (build 1, x86_64-suse-linux-gnu, GTK+ Version
3.24.43, cairo version 1.18.2)
Repository revision: eee0ed8442aa78320a3e578ab290df145fb49624
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101014
System Description: openSUSE Tumbleweed

Configured using:
 'configure --disable-build-details --without-pop --with-mailutils
 --without-hesiod --with-gameuser=:games --with-kerberos
 --with-kerberos5 --with-file-notification=inotify --with-modules
 --enable-autodepend --enable-link-time-optimization --prefix=/usr
 --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
 --localstatedir=/var --sharedstatedir=/var/lib
 --libexecdir=/usr/libexec --with-file-notification=yes
 --libdir=/usr/lib64 --with-native-compilation=aot
 --enable-locallisppath=/usr/share/emacs/31.0.50/site-lisp:/usr/share/emacs/site-lisp
 --with-x --with-xim --with-sound --with-xpm --with-jpeg --with-tiff
 --with-gif --with-png --with-rsvg --with-dbus --with-xft --without-gpm
 --with-tree-sitter --with-x-toolkit=gtk --without-pgtk
 --with-toolkit-scroll-bars --x-includes=/usr/include
 --x-libraries=/usr/lib64 --with-libotf --with-m17n-flt --with-cairo
 --build=x86_64-suse-linux --with-dumping=pdumper
 build_alias=x86_64-suse-linux 'CC=sccache cc' 'CFLAGS=-O2 -Wall
 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong
 -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
 -Werror=return-type -flto=auto -march=znver3 -mmmx -mpopcnt -msse
 -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4
 -mno-xop -mfma -mbmi -mbmi2 -maes -mpclmul -mno-gfni -mvpclmulqdq
 -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero
 -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp
 -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mpku
 -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize
 -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg
 -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile
 -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl
 -mno-avxvnni -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert
 -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint
 -mno-amx-complex --param l1-cache-size=32 --param l1-cache-line-size=64
 --param l2-cache-size=512 -mtune=znver3 -fno-optimize-sibling-calls -O2
 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong
 -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
 -Werror=return-type -flto=auto -g -D_GNU_SOURCE
 -DGDK_DISABLE_DEPRECATION_WARNINGS -DGLIB_DISABLE_DEPRECATION_WARNINGS
 -pipe -Wno-pointer-sign -Wno-unused-variable -Wno-unused-label
 -DPDMP_BASE='\''"emacs-gtk"'\''' LDFLAGS=-Wl,-O2 'CXX=sccache c++'
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-to-store-and-read-repository-information-of-VC.patch --]
[-- Type: text/patch, Size: 8336 bytes --]

From c5332d1a4a65ad6b124d0919d275c0ddde045842 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Bj=C3=B6rn=20Bidar?= <bjorn.bidar@thaodan.de>
Date: Sun, 17 Nov 2024 13:37:21 +0200
Subject: [PATCH] Allow to store and read repository information of VCS builds

Store repository information in version file if Emacs is build from
source while VCS is present. The version file can also be stored in the Emacs
sources prior built to indicate if Emacs was built from VCS sources.

An example use case could be if Emacs runs on a system where
the version control system isn't available, e.g. similarly
as it is intended for Android builds.
Another one is to be able to set the VCS information for workers
in a CI environment where sources are generated separately from VCS
but it or the VCS repository isn't present on the worker.

Reuse the same mechanism that exist for Android builds if the version
file is present.

* Makefile.in (etc-emacsver):
Generate etc/version file with revision and branch if git is installed
and Emacs sources are VCS sources.

* lisp/version.el (emacs-repository-get-branch)
(emacs-repository-get-version, emacs-repository-branch-static)
(emacs-repository-version-static):
Implement static versions that can use the information generated during
build if present.
---
 .gitignore      |  1 +
 Makefile.in     | 20 ++++++++++++++++----
 lisp/version.el | 44 ++++++++++++++++++++++++++++++++------------
 3 files changed, 49 insertions(+), 16 deletions(-)

diff --git a/.gitignore b/.gitignore
index c1f31514d06..d3b737d590c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -314,6 +314,7 @@ doc/misc/modus-themes.texi
 doc/misc/org.texi
 etc/DOC
 etc/refcards/emacsver.tex
+etc/version
 gnustmp*
 /info/
 
diff --git a/Makefile.in b/Makefile.in
index 30a762ed03b..202b0417282 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -452,7 +452,18 @@ etc-emacsver:
 	sed "s/[@]majorversion@/$${majorversion}/" \
 	  ${srcdir}/etc/refcards/emacsver.tex.in > emacsver.tex.$$$$ && \
 	  ${srcdir}/build-aux/move-if-change emacsver.tex.$$$$ \
-	  ${srcdir}/etc/refcards/emacsver.tex
+	  ${srcdir}/etc/refcards/emacsver.tex; \
+	  if [ -e $(srcdir)/.git ]  && \
+	  which git > /dev/null ; then \
+	  { (cd $(srcdir) \
+	     && git rev-parse HEAD || echo "Unknown") \
+	     && (git rev-parse --abbrev-ref HEAD \
+	         || echo "Unknown") } 2> /dev/null > \
+	     ${top_builddir}/etc/version.$$$$; \
+	     ${srcdir}/build-aux/move-if-change \
+		${top_builddir}/etc/version.$$$$ \
+		${top_builddir}/etc/version; \
+          else : ;fi
 
 # The shared gamedir name as a C string literal, or a null ptr if not in use.
 PATH_GAME = $(if $(use_gamedir),"$(gamedir)",((char const *) 0))
@@ -627,7 +638,7 @@ install-arch-dep:
 	umask 022; ${MKDIR_P} "$(DESTDIR)${bindir}"
 	$(MAKE) -C lib-src install
 ifeq (${ns_self_contained},no)
-	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACSFULL)"
+	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACS)"
 ifeq (${HAVE_BE_APP},yes)
 	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/Emacs "$(DESTDIR)${prefix}/apps/Emacs"
 endif
@@ -637,7 +648,7 @@ install-arch-dep:
 endif
 	${INSTALL_DATA} src/emacs.pdmp "$(DESTDIR)${libexecdir}/emacs/${version}/${configuration}"/emacs-${EMACS_PDMP}
 endif
-	-chmod 755 "$(DESTDIR)${bindir}/$(EMACSFULL)"
+	-chmod 755 "$(DESTDIR)${bindir}/$(EMACS)"
 ifndef NO_BIN_LINK
 	rm -f "$(DESTDIR)${bindir}/$(EMACS)"
 	cd "$(DESTDIR)${bindir}" && $(LN_S_FILEONLY) "$(EMACSFULL)" "$(EMACS)"
@@ -826,6 +837,7 @@ install-man:
 	umask 022; ${MKDIR_P} "$(DESTDIR)${man1dir}"
 	thisdir=`pwd -P`; \
 	cd ${mansrcdir}; \
+	cp ctags.1 gnuctags.1; \
 	for page in *.1; do \
 	  test "$$page" = ChangeLog.1 && continue; \
 	  dest=`echo "$${page}" | sed -e 's/\.1$$//' -e '$(TRANSFORM)'`.1; \
@@ -969,7 +981,7 @@ uninstall:
 	   for page in *.1; do \
 	     rm -f "$(DESTDIR)${man1dir}"/`echo "$${page}" | sed -e 's/\.1$$//' -e '$(TRANSFORM)'`.1$$ext; done; \
 	 fi)
-	rm -f "$(DESTDIR)${bindir}/$(EMACS)" "$(DESTDIR)${bindir}/$(EMACSFULL)"
+	rm -f "$(DESTDIR)${bindir}/$(EMACS)"
 	(if cd "$(DESTDIR)${icondir}"; then \
 	   rm -f hicolor/*x*/apps/"${EMACS_NAME}.png" \
 	     "hicolor/scalable/apps/${EMACS_NAME}.svg" \
diff --git a/lisp/version.el b/lisp/version.el
index db2afd55694..3b2c91c03dc 100644
--- a/lisp/version.el
+++ b/lisp/version.el
@@ -171,15 +171,21 @@ emacs-repository-version-git
 		  (looking-at "[[:xdigit:]]\\{40\\}"))
 	   (match-string 0)))))
 
-(defun emacs-repository-version-android ()
+(defun emacs-repository-version-static (dir)
   "Return the Emacs repository revision Emacs was built from.
 Value is nil if Emacs was not built from a repository checkout.
-Use information from the `/assets/version' special file."
+Use information from the `DIR/version' special file."
   (with-temp-buffer
-    (insert-file-contents "/assets/version")
+    (insert-file-contents (expand-file-name "version" dir))
     (let ((string (buffer-substring 1 (line-end-position))))
       (and (not (equal string "Unknown")) string))))
 
+(defun emacs-repository-version-android ()
+  "Return the Emacs repository revision Emacs was built from.
+Value is nil if Emacs was not built from a repository checkout.
+Use information from the `/assets/version' special file."
+  (emacs-repository-version-static "/assets"))
+
 (defun emacs-repository-get-version (&optional dir _external)
   "Try to return as a string the repository revision of the Emacs sources.
 The format of the returned string is dependent on the VCS in use.
@@ -194,9 +200,13 @@ emacs-repository-get-version
 
 Optional argument DIR is a directory to use instead of `source-directory'.
 Optional argument EXTERNAL is ignored."
-  (cond ((and (featurep 'android)
-              (eq system-type 'android))
-         (emacs-repository-version-android))
+  (cond ((and (or (and (featurep 'android)
+                       (eq system-type 'android)
+                       (setq dir "/assets"))
+                  (and (not dir)
+                       (file-exists-p (expand-file-name  "version" data-directory))
+                       (setq dir data-directory)))
+              (emacs-repository-version-static dir)))
         (t (emacs-repository-version-git
             (or dir source-directory)))))
 
@@ -209,8 +219,14 @@ emacs-repository-branch-android
   "Return the Emacs repository branch Emacs was built from.
 Value is nil if Emacs was not built from a repository checkout.
 Use information from the `/assets/version' special file."
+  (emacs-repository-branch-static "/assets"))
+
+(defun emacs-repository-branch-static (dir)
+  "Return the Emacs repository branch Emacs was built from.
+Value is nil if Emacs was not built from a repository checkout.
+Use information from the `DIR/version' special file."
   (with-temp-buffer
-    (insert-file-contents "/assets/version")
+    (insert-file-contents (expand-file-name "version" dir))
     (end-of-line)
     (forward-char)
     (let ((string (buffer-substring (point) (line-end-position))))
@@ -232,8 +248,8 @@ emacs-repository-get-branch
   "Try to return as a string the repository branch of the Emacs sources.
 The format of the returned string is dependent on the VCS in use.
 
-If Emacs is built for Android, use the version information
-embedded in the Emacs installation package.
+If Emacs is built for Android or contains version file,
+use the version information embedded in the Emacs installation package.
 
 Value is nil if the sources do not seem to be under version
 control, or if we could not determine the branch.  Note that
@@ -241,9 +257,13 @@ emacs-repository-get-branch
 correspond to the running Emacs.
 
 Optional argument DIR is a directory to use instead of `source-directory'."
-  (cond ((and (featurep 'android)
-              (eq system-type 'android))
-         (emacs-repository-branch-android))
+  (cond ((and (or (and (featurep 'android)
+                       (eq system-type 'android)
+                       (setq dir "/assets"))
+                  (and (not dir)
+                       (file-exists-p (expand-file-name  "version" data-directory))
+                       (setq dir data-directory)))
+              (emacs-repository-branch-static dir)))
         (t (emacs-repository-branch-git
             (or dir source-directory)))))
 
-- 
2.45.2

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> Date: Mon, 18 Nov 2024 10:18:11 +0200
> From:  Björn Bidar via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Emacs has the feature to read the repository version and
> branch if git is installed during the build and afterwards if also
> the sources including the VCS repository is present.
> 
> For the Android builds the feature was added to store and read the
> information mentioned above in a special version file.

Po Lu, why is that needed in the Android build, and how is it used
there?

> This patch reuses that mechanism so it can be reused on other platforms
> to use for the same reasons its available for Android and to also be
> able to use the information on CI workers without git installed and/or a
> full source checkout.

Doesn't that go against the tendency to have _less_ detailed/private
information in the build?  We've lately removed some relatively useful
infos from what we report in commands that use the build information.

More generally, could you present the motivation and the rationale for
making this information available in production builds?

> The things I'm not sure about for this patch are:
> - Is the generating of the version file in the right place in
> Makefile.in

It should be in the build tree, yes.

> - Is the data directory the right place to store the file

Not sure, but I don't see why not.

> - Should the creation of the version file be shared between the Android
>   builds and the other platforms

Let's first discuss whether this is at all needed and a good idea,
okay?

> -	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACSFULL)"
> +	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACS)"

Why this change (and other similar ones)?  What does EMACSFULL have to
do with the repository version data?

> @@ -826,6 +837,7 @@ install-man:
>  	umask 022; ${MKDIR_P} "$(DESTDIR)${man1dir}"
>  	thisdir=`pwd -P`; \
>  	cd ${mansrcdir}; \
> +	cp ctags.1 gnuctags.1; \

This hunk also looks unrelated.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 3354 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 18 Nov 2024 10:18:11 +0200
>> From:  Björn Bidar via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> Emacs has the feature to read the repository version and
>> branch if git is installed during the build and afterwards if also
>> the sources including the VCS repository is present.
>> 
>> For the Android builds the feature was added to store and read the
>> information mentioned above in a special version file.
>
> Po Lu, why is that needed in the Android build, and how is it used
> there?
>
>> This patch reuses that mechanism so it can be reused on other platforms
>> to use for the same reasons its available for Android and to also be
>> able to use the information on CI workers without git installed and/or a
>> full source checkout.
>
> Doesn't that go against the tendency to have _less_ detailed/private
> information in the build?  We've lately removed some relatively useful
> infos from what we report in commands that use the build information.

The information added is only the branch and the repository similarly as
used by the Android builds. There's no private information there unless
the exact change reference Emacs was built on is private.

> More generally, could you present the motivation and the rationale for
> making this information available in production builds?

The information wouldn't be only available to production builds but also
testing/developer builds that are builtin in a CI environment to
e.g. provide test builds for developers to use or to instruct user to
use to try to reproduce a bug.

Even for production builds it could be useful for convenience to track
down the exact reference/branch a build came from from, that's
side effect only thou.

>> The things I'm not sure about for this patch are:
>> - Is the generating of the version file in the right place in
>> Makefile.in
>
> It should be in the build tree, yes.

I was more talking if the section for the file should be in a separate
recipe or if etc-emacsver fits this purpose, I think the usecase is
quite close so it does sound ok to me.

>> - Is the data directory the right place to store the file
>
> Not sure, but I don't see why not.

OK I was just mostly wondering about the macOS builds who don't ship the
etc dir but since the information is be present also during dumping if
so desired it shouldn't be a big issue anyway.

>> - Should the creation of the version file be shared between the Android
>>   builds and the other platforms
>
> Let's first discuss whether this is at all needed and a good idea,
> okay?

Sure np, I was mostly speaking out load.

>
>> -	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACSFULL)"
>> +	${INSTALL_PROGRAM} $(INSTALL_STRIP) src/emacs${EXEEXT} "$(DESTDIR)${bindir}/$(EMACS)"
>
> Why this change (and other similar ones)?  What does EMACSFULL have to
> do with the repository version data?
>
>> @@ -826,6 +837,7 @@ install-man:
>>  	umask 022; ${MKDIR_P} "$(DESTDIR)${man1dir}"
>>  	thisdir=`pwd -P`; \
>>  	cd ${mansrcdir}; \
>> +	cp ctags.1 gnuctags.1; \
>
> This hunk also looks unrelated.

I'm sorry these changes came in accidentally. I attached a fixed patch below:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-to-store-and-read-repository-information-of-VC.patch --]
[-- Type: text/x-patch, Size: 6780 bytes --]

From d522173a61a84237d098690ee5289d2c11307306 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Bj=C3=B6rn=20Bidar?= <bjorn.bidar@thaodan.de>
Date: Mon, 18 Nov 2024 16:18:40 +0200
Subject: [PATCH] Allow to store and read repository information of VCS builds

Store repository information in version file if Emacs is build from
source while VCS is present. The version file can also be stored in the Emacs
sources prior built to indicate if Emacs was built from VCS sources.

An example use case could be if Emacs runs on a system where
the version control system isn't available, e.g. similarly
as it is intended for Android builds.
Another one is to be able to set the VCS information for workers
in a CI environment where sources are generated separately from VCS
but it or the VCS repository isn't present on the worker.

Reuse the same mechanism that exist for Android builds if the version
file is present.

* Makefile.in (etc-emacsver):
Generate etc/version file with revision and branch if git is installed
and Emacs sources are VCS sources.

* lisp/version.el (emacs-repository-get-branch)
(emacs-repository-get-version, emacs-repository-branch-static)
(emacs-repository-version-static):
Implement static versions that can use the information generated during
build if present.
---
 .gitignore      |  1 +
 Makefile.in     | 13 ++++++++++++-
 lisp/version.el | 44 ++++++++++++++++++++++++++++++++------------
 3 files changed, 45 insertions(+), 13 deletions(-)

diff --git a/.gitignore b/.gitignore
index c1f31514d06..d3b737d590c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -314,6 +314,7 @@ doc/misc/modus-themes.texi
 doc/misc/org.texi
 etc/DOC
 etc/refcards/emacsver.tex
+etc/version
 gnustmp*
 /info/
 
diff --git a/Makefile.in b/Makefile.in
index 30a762ed03b..2817742af86 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -452,7 +452,18 @@ etc-emacsver:
 	sed "s/[@]majorversion@/$${majorversion}/" \
 	  ${srcdir}/etc/refcards/emacsver.tex.in > emacsver.tex.$$$$ && \
 	  ${srcdir}/build-aux/move-if-change emacsver.tex.$$$$ \
-	  ${srcdir}/etc/refcards/emacsver.tex
+	  ${srcdir}/etc/refcards/emacsver.tex; \
+	  if [ -e $(srcdir)/.git ]  && \
+	  which git > /dev/null ; then \
+	  { (cd $(srcdir) \
+	     && git rev-parse HEAD || echo "Unknown") \
+	     && (git rev-parse --abbrev-ref HEAD \
+	         || echo "Unknown") } 2> /dev/null > \
+	     ${top_builddir}/etc/version.$$$$; \
+	     ${srcdir}/build-aux/move-if-change \
+		${top_builddir}/etc/version.$$$$ \
+		${top_builddir}/etc/version; \
+          else : ;fi
 
 # The shared gamedir name as a C string literal, or a null ptr if not in use.
 PATH_GAME = $(if $(use_gamedir),"$(gamedir)",((char const *) 0))
diff --git a/lisp/version.el b/lisp/version.el
index db2afd55694..3b2c91c03dc 100644
--- a/lisp/version.el
+++ b/lisp/version.el
@@ -171,15 +171,21 @@ emacs-repository-version-git
 		  (looking-at "[[:xdigit:]]\\{40\\}"))
 	   (match-string 0)))))
 
-(defun emacs-repository-version-android ()
+(defun emacs-repository-version-static (dir)
   "Return the Emacs repository revision Emacs was built from.
 Value is nil if Emacs was not built from a repository checkout.
-Use information from the `/assets/version' special file."
+Use information from the `DIR/version' special file."
   (with-temp-buffer
-    (insert-file-contents "/assets/version")
+    (insert-file-contents (expand-file-name "version" dir))
     (let ((string (buffer-substring 1 (line-end-position))))
       (and (not (equal string "Unknown")) string))))
 
+(defun emacs-repository-version-android ()
+  "Return the Emacs repository revision Emacs was built from.
+Value is nil if Emacs was not built from a repository checkout.
+Use information from the `/assets/version' special file."
+  (emacs-repository-version-static "/assets"))
+
 (defun emacs-repository-get-version (&optional dir _external)
   "Try to return as a string the repository revision of the Emacs sources.
 The format of the returned string is dependent on the VCS in use.
@@ -194,9 +200,13 @@ emacs-repository-get-version
 
 Optional argument DIR is a directory to use instead of `source-directory'.
 Optional argument EXTERNAL is ignored."
-  (cond ((and (featurep 'android)
-              (eq system-type 'android))
-         (emacs-repository-version-android))
+  (cond ((and (or (and (featurep 'android)
+                       (eq system-type 'android)
+                       (setq dir "/assets"))
+                  (and (not dir)
+                       (file-exists-p (expand-file-name  "version" data-directory))
+                       (setq dir data-directory)))
+              (emacs-repository-version-static dir)))
         (t (emacs-repository-version-git
             (or dir source-directory)))))
 
@@ -209,8 +219,14 @@ emacs-repository-branch-android
   "Return the Emacs repository branch Emacs was built from.
 Value is nil if Emacs was not built from a repository checkout.
 Use information from the `/assets/version' special file."
+  (emacs-repository-branch-static "/assets"))
+
+(defun emacs-repository-branch-static (dir)
+  "Return the Emacs repository branch Emacs was built from.
+Value is nil if Emacs was not built from a repository checkout.
+Use information from the `DIR/version' special file."
   (with-temp-buffer
-    (insert-file-contents "/assets/version")
+    (insert-file-contents (expand-file-name "version" dir))
     (end-of-line)
     (forward-char)
     (let ((string (buffer-substring (point) (line-end-position))))
@@ -232,8 +248,8 @@ emacs-repository-get-branch
   "Try to return as a string the repository branch of the Emacs sources.
 The format of the returned string is dependent on the VCS in use.
 
-If Emacs is built for Android, use the version information
-embedded in the Emacs installation package.
+If Emacs is built for Android or contains version file,
+use the version information embedded in the Emacs installation package.
 
 Value is nil if the sources do not seem to be under version
 control, or if we could not determine the branch.  Note that
@@ -241,9 +257,13 @@ emacs-repository-get-branch
 correspond to the running Emacs.
 
 Optional argument DIR is a directory to use instead of `source-directory'."
-  (cond ((and (featurep 'android)
-              (eq system-type 'android))
-         (emacs-repository-branch-android))
+  (cond ((and (or (and (featurep 'android)
+                       (eq system-type 'android)
+                       (setq dir "/assets"))
+                  (and (not dir)
+                       (file-exists-p (expand-file-name  "version" data-directory))
+                       (setq dir data-directory)))
+              (emacs-repository-branch-static dir)))
         (t (emacs-repository-branch-git
             (or dir source-directory)))))
 
-- 
2.45.2

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Po Lu <luangruo@yahoo.com>,  74413@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 16:21:28 +0200
> 
> > Doesn't that go against the tendency to have _less_ detailed/private
> > information in the build?  We've lately removed some relatively useful
> > infos from what we report in commands that use the build information.
> 
> The information added is only the branch and the repository similarly as
> used by the Android builds. There's no private information there unless
> the exact change reference Emacs was built on is private.

The branch name could be private.

Stefan, WDYT about this feature suggestion?

> > More generally, could you present the motivation and the rationale for
> > making this information available in production builds?
> 
> The information wouldn't be only available to production builds but also
> testing/developer builds that are builtin in a CI environment to
> e.g. provide test builds for developers to use or to instruct user to
> use to try to reproduce a bug.
> 
> Even for production builds it could be useful for convenience to track
> down the exact reference/branch a build came from from, that's
> side effect only thou.

This is already available if Emacs is built in a Git repository, and
the information is stored in the dumped Emacs.  So what is gained by
also recording the repository version on a disk file external to
Emacs?

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Po Lu <luangruo@yahoo.com>,  74413@debbugs.gnu.org
>> Date: Mon, 18 Nov 2024 16:21:28 +0200
>> 
>> > Doesn't that go against the tendency to have _less_ detailed/private
>> > information in the build?  We've lately removed some relatively useful
>> > infos from what we report in commands that use the build information.
>> 
>> The information added is only the branch and the repository similarly as
>> used by the Android builds. There's no private information there unless
>> the exact change reference Emacs was built on is private.
>
> The branch name could be private.

That could be but at that point you wouldn't have access to the binary
that contains the information and probably wouldn't report bugs to this
tracker either I think.

> Stefan, WDYT about this feature suggestion?
>
>> > More generally, could you present the motivation and the rationale for
>> > making this information available in production builds?
>> 
>> The information wouldn't be only available to production builds but also
>> testing/developer builds that are builtin in a CI environment to
>> e.g. provide test builds for developers to use or to instruct user to
>> use to try to reproduce a bug.
>> 
>> Even for production builds it could be useful for convenience to track
>> down the exact reference/branch a build came from from, that's
>> side effect only thou.
>
> This is already available if Emacs is built in a Git repository, and
> the information is stored in the dumped Emacs.  So what is gained by
> also recording the repository version on a disk file external to
> Emacs?

The information is only stored if the worker already had git installed
and checked out the sources with git inside the worker.
Also the function currently fails unless the system the user uses also happens to
have git installed and the sources if the are installed also contain the
VCS metadata.
Storing the VCS metadata in the sources doesn't happen usually as it
increases the size of a good chunk. In my case e.g. from 188MB to 788MB.
Why not have the same feature for other platforms too?

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  luangruo@yahoo.com,
>   74413@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 18:54:26 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This is already available if Emacs is built in a Git repository, and
> > the information is stored in the dumped Emacs.  So what is gained by
> > also recording the repository version on a disk file external to
> > Emacs?
> 
> The information is only stored if the worker already had git installed
> and checked out the sources with git inside the worker.
> Also the function currently fails unless the system the user uses also happens to
> have git installed and the sources if the are installed also contain the
> VCS metadata.
> Storing the VCS metadata in the sources doesn't happen usually as it
> increases the size of a good chunk. In my case e.g. from 188MB to 788MB.
> Why not have the same feature for other platforms too?

Sorry, I don't understand this explanation; I'm probably missing
something.

The feature you propose requires to build Emacs inside a Git
repository, is that correct?  Because otherwise "git rev-parse" will
not work, right?  If that is correct, then building Emacs inside a Git
repository already calls this Git command and records the result in 2
Emacs Lisp variables.  So why do you also want to record the same
information on a file?  What kind of scenario do you have in mind in
which building Emacs with its current code will not record the branch
and the revision, but your additions to Makefile will record that?

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Stefan Kangas <stefankangas@gmail.com>,  luangruo@yahoo.com,
>>   74413@debbugs.gnu.org
>> Date: Mon, 18 Nov 2024 18:54:26 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > This is already available if Emacs is built in a Git repository, and
>> > the information is stored in the dumped Emacs.  So what is gained by
>> > also recording the repository version on a disk file external to
>> > Emacs?
>> 
>> The information is only stored if the worker already had git installed
>> and checked out the sources with git inside the worker.
>> Also the function currently fails unless the system the user uses also happens to
>> have git installed and the sources if the are installed also contain the
>> VCS metadata.
>> Storing the VCS metadata in the sources doesn't happen usually as it
>> increases the size of a good chunk. In my case e.g. from 188MB to 788MB.
>> Why not have the same feature for other platforms too?
>
> Sorry, I don't understand this explanation; I'm probably missing
> something.
>
> The feature you propose requires to build Emacs inside a Git
> repository, is that correct?  Because otherwise "git rev-parse" will
> not work, right?  If that is correct, then building Emacs inside a Git
> repository already calls this Git command and records the result in 2
> Emacs Lisp variables.
>
> So why do you also want to record the same
> information on a file?  What kind of scenario do you have in mind in
> which building Emacs with its current code will not record the branch
> and the revision, but your additions to Makefile will record that?

The additions to the make file are so that if the worker contains git
the file can be generated so that the related functions will still work
after the built or to generate them prior the built. The latter probably
makes less sense except to maybe avoid having autotools in the built
dependency chain.

If the Make recipe isn't used to generate the version file it can be
generated by the CI, e.g. in my case I take the information from the
open build source service. For others such as Fedora the sources can be
retrieved in a similar manner.

The file can be added before the built starts and package  so that the pdump
will contain the repository information and the VCS function will also
work afterwards. 

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 21:31:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Björn Bidar <bjorn.bidar@thaodan.de>
> >> Cc: Stefan Kangas <stefankangas@gmail.com>,  luangruo@yahoo.com,
> >>   74413@debbugs.gnu.org
> >> Date: Mon, 18 Nov 2024 18:54:26 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > This is already available if Emacs is built in a Git repository, and
> >> > the information is stored in the dumped Emacs.  So what is gained by
> >> > also recording the repository version on a disk file external to
> >> > Emacs?
> >> 
> >> The information is only stored if the worker already had git installed
> >> and checked out the sources with git inside the worker.
> >> Also the function currently fails unless the system the user uses also happens to
> >> have git installed and the sources if the are installed also contain the
> >> VCS metadata.
> >> Storing the VCS metadata in the sources doesn't happen usually as it
> >> increases the size of a good chunk. In my case e.g. from 188MB to 788MB.
> >> Why not have the same feature for other platforms too?
> >
> > Sorry, I don't understand this explanation; I'm probably missing
> > something.
> >
> > The feature you propose requires to build Emacs inside a Git
> > repository, is that correct?  Because otherwise "git rev-parse" will
> > not work, right?  If that is correct, then building Emacs inside a Git
> > repository already calls this Git command and records the result in 2
> > Emacs Lisp variables.
> >
> > So why do you also want to record the same
> > information on a file?  What kind of scenario do you have in mind in
> > which building Emacs with its current code will not record the branch
> > and the revision, but your additions to Makefile will record that?
> 
> The additions to the make file are so that if the worker contains git
> the file can be generated so that the related functions will still work
> after the built or to generate them prior the built. The latter probably
> makes less sense except to maybe avoid having autotools in the built
> dependency chain.
> 
> If the Make recipe isn't used to generate the version file it can be
> generated by the CI, e.g. in my case I take the information from the
> open build source service. For others such as Fedora the sources can be
> retrieved in a similar manner.
> 
> The file can be added before the built starts and package  so that the pdump
> will contain the repository information and the VCS function will also
> work afterwards. 

I still don't think I understand, sorry.  Do you mean the file is
generated from a Git repository, but then Emacs is somehow built from
a directory that is not under Git?  But if this is the scenario, how
can you be sure the produced Emacs binary was made from that revision
on that branch?  This is only guaranteed if you actually build from
Git when you record this information.

What am I missing?

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
>> Date: Mon, 18 Nov 2024 21:31:43 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> >> Cc: Stefan Kangas <stefankangas@gmail.com>,  luangruo@yahoo.com,
>> >>   74413@debbugs.gnu.org
>> >> Date: Mon, 18 Nov 2024 18:54:26 +0200
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> > This is already available if Emacs is built in a Git repository, and
>> >> > the information is stored in the dumped Emacs.  So what is gained by
>> >> > also recording the repository version on a disk file external to
>> >> > Emacs?
>> >> 
>> >> The information is only stored if the worker already had git installed
>> >> and checked out the sources with git inside the worker.
>> >> Also the function currently fails unless the system the user uses also happens to
>> >> have git installed and the sources if the are installed also contain the
>> >> VCS metadata.
>> >> Storing the VCS metadata in the sources doesn't happen usually as it
>> >> increases the size of a good chunk. In my case e.g. from 188MB to 788MB.
>> >> Why not have the same feature for other platforms too?
>> >
>> > Sorry, I don't understand this explanation; I'm probably missing
>> > something.
>> >
>> > The feature you propose requires to build Emacs inside a Git
>> > repository, is that correct?  Because otherwise "git rev-parse" will
>> > not work, right?  If that is correct, then building Emacs inside a Git
>> > repository already calls this Git command and records the result in 2
>> > Emacs Lisp variables.
>> >
>> > So why do you also want to record the same
>> > information on a file?  What kind of scenario do you have in mind in
>> > which building Emacs with its current code will not record the branch
>> > and the revision, but your additions to Makefile will record that?
>> 
>> The additions to the make file are so that if the worker contains git
>> the file can be generated so that the related functions will still work
>> after the built or to generate them prior the built. The latter probably
>> makes less sense except to maybe avoid having autotools in the built
>> dependency chain.
>> 
>> If the Make recipe isn't used to generate the version file it can be
>> generated by the CI, e.g. in my case I take the information from the
>> open build source service. For others such as Fedora the sources can be
>> retrieved in a similar manner.
>> 
>> The file can be added before the built starts and package  so that the pdump
>> will contain the repository information and the VCS function will also
>> work afterwards. 
>
> I still don't think I understand, sorry.  Do you mean the file is
> generated from a Git repository, but then Emacs is somehow built from
> a directory that is not under Git?

The file can be generated from the git repository outside of the Emacs
builder.

E.g. in my case the obs source service store's the git revision used in
the service 

In my case the file looks like this:
name: emacs
version: 31.0.50.9794.eee0ed8442a
mtime: 1731883844
commit: eee0ed8442aa78320a3e578ab290df145fb49624

sed -n -e 's/^commit: \(.*\)/\1/p' emacs.obsinfo > etc/version

> But if this is the scenario, how can you be sure the produced Emacs binary was made from that revision
> on that branch?  This is only guaranteed if you actually build from
> Git when you record this information.
>
> What am I missing?

If the source is generated by the CI it can also store this information
in the build source which then can be extracted from the ci metadata to
the Emacs sources on the builder.

I can be sure that Emacs was built from that revision as much as I can
trust the CI to use the sources I told it to use. If I can't trust one,
I can't trust the other.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Po Lu <luangruo@yahoo.com>,  74413@debbugs.gnu.org
>> Date: Mon, 18 Nov 2024 16:21:28 +0200
>>
>> > Doesn't that go against the tendency to have _less_ detailed/private
>> > information in the build?  We've lately removed some relatively useful
>> > infos from what we report in commands that use the build information.
>>
>> The information added is only the branch and the repository similarly as
>> used by the Android builds. There's no private information there unless
>> the exact change reference Emacs was built on is private.
>
> The branch name could be private.
>
> Stefan, WDYT about this feature suggestion?

The privacy risk here is that if a user is building their own private
branch, announcing the sha or branch name to the world can be used to
uniquely identify that user.  It would be a serious privacy issue if we,
for example, included that information in User-Agent headers sent by EWW
or other kinds of network traffic.  AFAIK, we don't do that.

IIUC, we use this information only when submitting bug reports.  I think
this is harmless, if we assume privacy threat models where it can also
be considered safe to report bugs.  The few users that have more strict
privacy requirements, and are eager to report bugs, will just have to
think about this detail themselves; it's a rather specialized use case.

IOW, I don't think I see a reason to object on these grounds.

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> The additions to the make file are so that if the worker contains git
> the file can be generated so that the related functions will still work
> after the built or to generate them prior the built. The latter probably
> makes less sense except to maybe avoid having autotools in the built
> dependency chain.
>
> If the Make recipe isn't used to generate the version file it can be
> generated by the CI, e.g. in my case I take the information from the
> open build source service. For others such as Fedora the sources can be
> retrieved in a similar manner.
>
> The file can be added before the built starts and package  so that the pdump
> will contain the repository information and the VCS function will also
> work afterwards.

I don't fully understand how you can have a situation where you can get
this information in the Makefile, but you can't also get it when dumping
using `emacs-repository-get-version` and `emacs-repository-get-branch`
(lisp/loadup.el:474).  Could you please elaborate on this?

Do you mean that you have one containerized process with Git that clones
emacs.git into a directory, and then an entirely separate containerized
process, without Git, builds Emacs from that very same directory?  Or
something along those lines?

In this very particular scenario, isn't it enough to add this
additional step to your CI pipeline:

    ./src/emacs -Q --batch --eval '(princ (format "%s:%s" \
    emacs-repository-version emacs-repository-branch))' > version.info

?

In other words, is it really necessary for us to support this use case
in our Makefile?  Do we expect that building Emacs in such CI pipelines
using non-released development version of Emacs will be very common?

BTW, what is the name of that CI system that you're using here?  For
what purpose are you building Emacs: to test Elisp packages, to test
Emacs, or something else?  Finally, why are you not using officially
tagged versions, either from us or from some distro, in this context?

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>>> Cc: Po Lu <luangruo@yahoo.com>,  74413@debbugs.gnu.org
>>> Date: Mon, 18 Nov 2024 16:21:28 +0200
>>>
>>> > Doesn't that go against the tendency to have _less_ detailed/private
>>> > information in the build?  We've lately removed some relatively useful
>>> > infos from what we report in commands that use the build information.
>>>
>>> The information added is only the branch and the repository similarly as
>>> used by the Android builds. There's no private information there unless
>>> the exact change reference Emacs was built on is private.
>>
>> The branch name could be private.
>>
>> Stefan, WDYT about this feature suggestion?
>
> The privacy risk here is that if a user is building their own private
> branch, announcing the sha or branch name to the world can be used to
> uniquely identify that user.  It would be a serious privacy issue if we,
> for example, included that information in User-Agent headers sent by EWW
> or other kinds of network traffic.  AFAIK, we don't do that.
>
> IIUC, we use this information only when submitting bug reports.  I think
> this is harmless, if we assume privacy threat models where it can also
> be considered safe to report bugs.  The few users that have more strict
> privacy requirements, and are eager to report bugs, will just have to
> think about this detail themselves; it's a rather specialized use case.
>
> IOW, I don't think I see a reason to object on these grounds.
>
> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> The additions to the make file are so that if the worker contains git
>> the file can be generated so that the related functions will still work
>> after the built or to generate them prior the built. The latter probably
>> makes less sense except to maybe avoid having autotools in the built
>> dependency chain.
>>
>> If the Make recipe isn't used to generate the version file it can be
>> generated by the CI, e.g. in my case I take the information from the
>> open build source service. For others such as Fedora the sources can be
>> retrieved in a similar manner.
>>
>> The file can be added before the built starts and package  so that the pdump
>> will contain the repository information and the VCS function will also
>> work afterwards.
>
> I don't fully understand how you can have a situation where you can get
> this information in the Makefile, but you can't also get it when dumping
> using `emacs-repository-get-version` and `emacs-repository-get-branch`
> (lisp/loadup.el:474).  Could you please elaborate on this?

I added the Makefile part so it would work outside of my environment if
one builds Emacs and then distributes/install it without the sources or
without them containing the VCS information while git is also installed
in their build environment.

Since Emacs Makefiles already have that target for the Android build it
could make sense to share this target outside of the Android build.

> Do you mean that you have one containerized process with Git that clones
> emacs.git into a directory, and then an entirely separate containerized
> process, without Git, builds Emacs from that very same directory?  Or
> something along those lines?

Yes more or less.
I'm using an obs source service that generates a tarball from VCS's and
then creates a tarball. [1] 

> In this very particular scenario, isn't it enough to add this
> additional step to your CI pipeline:
>
>     ./src/emacs -Q --batch --eval '(princ (format "%s:%s" \
>     emacs-repository-version emacs-repository-branch))' > version.info
The obs source service already contains that information see the
emacs.obsinfo file I mentioned earlier.

> In other words, is it really necessary for us to support this use case
> in our Makefile?  Do we expect that building Emacs in such CI pipelines
> using non-released development version of Emacs will be very common?

The Makefile target is there already for the Android builds all I did I
added it to other platforms too. If the target would be shared then the
hole process is reused.

I think it would become more common to build from VCS sources would be
become more common where the information is useful.
To get more users to test earlier development builds to have faster
general feedback and bug reports reports it is very useful to have
non-released development builds. There are other instances outside my
own usecase where this is frequent such as for example the AUR emacs-git, guix
emacs-next or the Android Fdroid builds.

> BTW, what is the name of that CI system that you're using here?  For
> what purpose are you building Emacs: to test Elisp packages, to test
> Emacs, or something else?  Finally, why are you not using officially
> tagged versions, either from us or from some distro, in this context?

I'm build Emacs on the open build service. I'm building Emacs on the obs
to provide development builds that can be used to test Emacs and submit
bug reports.
The package is build using the official RPM packaging just
with the newer sources from git.

The initial reason reason was that I noticed that even if you build from
git directly without a CI/or something similar to the source service
mentioned earlier that the functions to retrieve the repository
information don't work since the source path either doesn't exist on any
the machine that installs the final package or that it doesn't contain
the VCS information anymore.
The setup is very similar to the Android builds.



---
[1] https://github.com/openSUSE/obs-service-tar_scm

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 23:48:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I still don't think I understand, sorry.  Do you mean the file is
> > generated from a Git repository, but then Emacs is somehow built from
> > a directory that is not under Git?
> 
> The file can be generated from the git repository outside of the Emacs
> builder.

So you mean someone will chdir to the Git repository, say

  $ make etc-emacsver

Then take the produced file and manually install it when Emacs is
built (in another directory) and installed, is that right?

> > But if this is the scenario, how can you be sure the produced Emacs binary was made from that revision
> > on that branch?  This is only guaranteed if you actually build from
> > Git when you record this information.
> >
> > What am I missing?
> 
> If the source is generated by the CI it can also store this information
> in the build source which then can be extracted from the ci metadata to
> the Emacs sources on the builder.
> 
> I can be sure that Emacs was built from that revision as much as I can
> trust the CI to use the sources I told it to use. If I can't trust one,
> I can't trust the other.

But CI builds from Git, doesn't it?  If so, the Emacs it produces
already records the revision and the branch.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Mon, 18 Nov 2024 18:48:31 -0500
> Cc: luangruo@yahoo.com, 74413@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The branch name could be private.
> >
> > Stefan, WDYT about this feature suggestion?
> 
> The privacy risk here is that if a user is building their own private
> branch, announcing the sha or branch name to the world can be used to
> uniquely identify that user.  It would be a serious privacy issue if we,
> for example, included that information in User-Agent headers sent by EWW
> or other kinds of network traffic.  AFAIK, we don't do that.
> 
> IIUC, we use this information only when submitting bug reports.  I think
> this is harmless, if we assume privacy threat models where it can also
> be considered safe to report bugs.  The few users that have more strict
> privacy requirements, and are eager to report bugs, will just have to
> think about this detail themselves; it's a rather specialized use case.

AFAIU, the intent is to use this for more than just bug reporting.

> I don't fully understand how you can have a situation where you can get
> this information in the Makefile, but you can't also get it when dumping
> using `emacs-repository-get-version` and `emacs-repository-get-branch`
> (lisp/loadup.el:474).  Could you please elaborate on this?

Yes, I still don't understand the utility of this feature, since it
needs Git for producing the information in the file.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
>> Date: Mon, 18 Nov 2024 23:48:38 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I still don't think I understand, sorry.  Do you mean the file is
>> > generated from a Git repository, but then Emacs is somehow built from
>> > a directory that is not under Git?
>> 
>> The file can be generated from the git repository outside of the Emacs
>> builder.
>
> So you mean someone will chdir to the Git repository, say
>
>   $ make etc-emacsver
>
> Then take the produced file and manually install it when Emacs is
> built (in another directory) and installed, is that right?

I extract the file from the obs service and generate file from the
services metadata. The make target is there to create the file if Emacs
is built with git installed or if the target is used in the way you
mentioned above. 

In both cases the emacs VCS functions will still work after the built
without the sources or git installed, in the case of the former
as long as in the packager has provided the metadata themselves. 

>> > But if this is the scenario, how can you be sure the produced Emacs binary was made from that revision
>> > on that branch?  This is only guaranteed if you actually build from
>> > Git when you record this information.
>> >
>> > What am I missing?
>> 
>> If the source is generated by the CI it can also store this information
>> in the build source which then can be extracted from the ci metadata to
>> the Emacs sources on the builder.
>> 
>> I can be sure that Emacs was built from that revision as much as I can
>> trust the CI to use the sources I told it to use. If I can't trust one,
>> I can't trust the other.
>
> But CI builds from Git, doesn't it?  If so, the Emacs it produces
> already records the revision and the branch.

The source generator, the source service runs git but the builder
doesn't.
Because the builder doesn't have to deal with git the rebuild chain is
smaller, build dependency changes can trigger rebuilds, and the worker
doesn't have to have git installed.

In most cases it's not required or wanted that worker have git as the
only allowed purpose would be to get access already existing metadata but
not change anything on the sources itself.

Providing the metadata to the VCS emacs-repository-branch and
emacs-repository-version with the previously generated metadata that
belongs to the package sources removes the need for git in the build worker.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Mon, 18 Nov 2024 18:48:31 -0500
>> Cc: luangruo@yahoo.com, 74413@debbugs.gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The branch name could be private.
>> >
>> > Stefan, WDYT about this feature suggestion?
>> 
>> The privacy risk here is that if a user is building their own private
>> branch, announcing the sha or branch name to the world can be used to
>> uniquely identify that user.  It would be a serious privacy issue if we,
>> for example, included that information in User-Agent headers sent by EWW
>> or other kinds of network traffic.  AFAIK, we don't do that.
>> 
>> IIUC, we use this information only when submitting bug reports.  I think
>> this is harmless, if we assume privacy threat models where it can also
>> be considered safe to report bugs.  The few users that have more strict
>> privacy requirements, and are eager to report bugs, will just have to
>> think about this detail themselves; it's a rather specialized use case.
>
> AFAIU, the intent is to use this for more than just bug reporting.

Where could leak the feature information? If it could it should be
adjusted as well for the Android builds.

>> I don't fully understand how you can have a situation where you can get
>> this information in the Makefile, but you can't also get it when dumping
>> using `emacs-repository-get-version` and `emacs-repository-get-branch`
>> (lisp/loadup.el:474).  Could you please elaborate on this?
>
> Yes, I still don't understand the utility of this feature, since it
> needs Git for producing the information in the file.

As explained Git doesn't have to be installed using that feature on the
machine that executes the built Emacs or on the builder.

The feature is the same as for the Android builds just for other
platforms, the reasons are the same as for Android.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
> Date: Tue, 19 Nov 2024 18:13:14 +0200
> 
> >> If the source is generated by the CI it can also store this information
> >> in the build source which then can be extracted from the ci metadata to
> >> the Emacs sources on the builder.
> >> 
> >> I can be sure that Emacs was built from that revision as much as I can
> >> trust the CI to use the sources I told it to use. If I can't trust one,
> >> I can't trust the other.
> >
> > But CI builds from Git, doesn't it?  If so, the Emacs it produces
> > already records the revision and the branch.
> 
> The source generator, the source service runs git but the builder
> doesn't.
> Because the builder doesn't have to deal with git the rebuild chain is
> smaller, build dependency changes can trigger rebuilds, and the worker
> doesn't have to have git installed.
> 
> In most cases it's not required or wanted that worker have git as the
> only allowed purpose would be to get access already existing metadata but
> not change anything on the sources itself.
> 
> Providing the metadata to the VCS emacs-repository-branch and
> emacs-repository-version with the previously generated metadata that
> belongs to the package sources removes the need for git in the build worker.

Thanks, but I still don't understand the problem you are trying to
solve.  I guess I'm missing something very fundamental here.  You are
talking about "builder", "source generator", "metadata", etc., and I
don't understand these terms.  The last sentence seems to confirm my
guess: you tage the Git revision from a repository, and then build
from another directory, which can easily cause a mismatch.  So this
feature looks completely redundant to me, based on the little I do
understand in these matters.  But I will now bow out of this
discussion; if Stefan (and others) think it's okay, I won't object.

Thank you for your patience.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  luangruo@yahoo.com,
>   74413@debbugs.gnu.org
> Date: Tue, 19 Nov 2024 18:16:28 +0200
> 
> > Yes, I still don't understand the utility of this feature, since it
> > needs Git for producing the information in the file.
> 
> As explained Git doesn't have to be installed using that feature on the
> machine that executes the built Emacs or on the builder.

Yes, and I don't understand the utility of this.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: stefankangas@gmail.com,  luangruo@yahoo.com,  74413@debbugs.gnu.org
>> Date: Tue, 19 Nov 2024 18:13:14 +0200
>> 
>> >> If the source is generated by the CI it can also store this information
>> >> in the build source which then can be extracted from the ci metadata to
>> >> the Emacs sources on the builder.
>> >> 
>> >> I can be sure that Emacs was built from that revision as much as I can
>> >> trust the CI to use the sources I told it to use. If I can't trust one,
>> >> I can't trust the other.
>> >
>> > But CI builds from Git, doesn't it?  If so, the Emacs it produces
>> > already records the revision and the branch.
>> 
>> The source generator, the source service runs git but the builder
>> doesn't.
>> Because the builder doesn't have to deal with git the rebuild chain is
>> smaller, build dependency changes can trigger rebuilds, and the worker
>> doesn't have to have git installed.
>> 
>> In most cases it's not required or wanted that worker have git as the
>> only allowed purpose would be to get access already existing metadata but
>> not change anything on the sources itself.
>> 
>> Providing the metadata to the VCS emacs-repository-branch and
>> emacs-repository-version with the previously generated metadata that
>> belongs to the package sources removes the need for git in the build worker.
>
> Thanks, but I still don't understand the problem you are trying to
> solve.  I guess I'm missing something very fundamental here.  You are
> talking about "builder", "source generator", "metadata", etc., and I
> don't understand these terms.  The last sentence seems to confirm my
> guess: you tage the Git revision from a repository, and then build
> from another directory, which can easily cause a mismatch.

It could be that I either assume some things or don't explain them well enough.

I generate a tarball from one repository, store the last commit's refernce the
snapshot of that repository which the tarball was generated from.
The configuration of the program that generates the tarball contains the
branch it pulls the sources from. Both of these then are put into the
the same file that is used by the Android builds.

> So this
> feature looks completely redundant to me, based on the little I do
> understand in these matters.  But I will now bow out of this
> discussion; if Stefan (and others) think it's okay, I won't object.

To reduce any possible I could make the make target shared with the
Android builds if that helps.

> Thank you for your patience.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Stefan Kangas]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

>> I don't fully understand how you can have a situation where you can get
>> this information in the Makefile, but you can't also get it when dumping
>> using `emacs-repository-get-version` and `emacs-repository-get-branch`
>> (lisp/loadup.el:474).  Could you please elaborate on this?
>
> I added the Makefile part so it would work outside of my environment if
> one builds Emacs and then distributes/install it without the sources or
> without them containing the VCS information while git is also installed
> in their build environment.

I don't think I understand what you're saying here.  Is it possible that
there are some typos above?  I'm thinking of the part that contains the
tautology "without the sources or without them".

There are also some words where the meaning is not entirely clear to me,
such as "distributes" and "installs".

Maybe it will be clarified by the answers to my below questions.

>> Do you mean that you have one containerized process with Git that clones
>> emacs.git into a directory, and then an entirely separate containerized
>> process, without Git, builds Emacs from that very same directory?  Or
>> something along those lines?
>
> Yes more or less.
> I'm using an obs source service that generates a tarball from VCS's and
> then creates a tarball. [1]

I took a look at the linked documentation, but it seems to be on a very
detailed level, and does not provide a good overview.  So I don't think
I understood much from it.

I'm not familiar with the Open Build System, so this is quite hard for
me to follow.  Please be patient with me.

In your description above, could you explain what is the difference
between the two steps "generates a tarball from VCS" and "creates a
tarball"?

Do you mean to say that it:
1) generates a tarball from emacs.git, excluding the .git directory
2) copies that tarball to a completely different environment
3) builds Emacs

The part that I do not understand is in which step you run `make`, and
how.  Is it run in the first step, the third, the first and the third,
or something else?  Is it run automatically by OBS, or do you have to
specify it?

Basically, I still don't understand the answer to my below question:

>> In this very particular scenario, isn't it enough to add this
>> additional step to your CI pipeline:
>>
>>     ./src/emacs -Q --batch --eval '(princ (format "%s:%s" \
>>     emacs-repository-version emacs-repository-branch))' > version.info
> The obs source service already contains that information see the
> emacs.obsinfo file I mentioned earlier.

Hmm.  Would an alternative solution be to read the version information
from that file then, if it exists?

>> In other words, is it really necessary for us to support this use case
>> in our Makefile?  Do we expect that building Emacs in such CI pipelines
>> using non-released development version of Emacs will be very common?
>
> The Makefile target is there already for the Android builds all I did I
> added it to other platforms too. If the target would be shared then the
> hole process is reused.

If we want this for other builds, then reusing what we do for the
Android build sounds prudent.  It is clear that this is needed for
Android.  What remains to clarify is why it is needed elsewhere.

> I think it would become more common to build from VCS sources would be
> become more common where the information is useful.

Building from VCS sources is already quite common.  I'm asking something
more specific: will it be increasingly common to build using an
environment very similar to OBS?

If it is not going to be very common, this should perhaps be fixed in
the OBS pipeline directly, instead of complicating our Makefile.  But
I'm not yet clear on whether that is true.

> To get more users to test earlier development builds to have faster
> general feedback and bug reports reports it is very useful to have
> non-released development builds. There are other instances outside my
> own usecase where this is frequent such as for example the AUR emacs-git, guix
> emacs-next or the Android Fdroid builds.

Are you saying that AUR emacs-git, guix emacs-next, and the Android
Fdroid builds also have no way to know which revision Emacs was built
from without your patch?

>> BTW, what is the name of that CI system that you're using here?  For
>> what purpose are you building Emacs: to test Elisp packages, to test
>> Emacs, or something else?  Finally, why are you not using officially
>> tagged versions, either from us or from some distro, in this context?
>
> I'm build Emacs on the open build service. I'm building Emacs on the obs
> to provide development builds that can be used to test Emacs and submit
> bug reports.
> The package is build using the official RPM packaging just
> with the newer sources from git.
>
> The initial reason reason was that I noticed that even if you build from
> git directly without a CI/or something similar to the source service
> mentioned earlier that the functions to retrieve the repository
> information don't work since the source path either doesn't exist on any
> the machine that installs the final package or that it doesn't contain
> the VCS information anymore.
> The setup is very similar to the Android builds.

Thanks.

bug#74413: [PATCH] Allow to store and read repository information of VCS builds

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefankangas@gmail.com> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>>> I don't fully understand how you can have a situation where you can get
>>> this information in the Makefile, but you can't also get it when dumping
>>> using `emacs-repository-get-version` and `emacs-repository-get-branch`
>>> (lisp/loadup.el:474).  Could you please elaborate on this?
>>
>> I added the Makefile part so it would work outside of my environment if
>> one builds Emacs and then distributes/install it without the sources or
>> without them containing the VCS information while git is also installed
>> in their build environment.
>
> I don't think I understand what you're saying here.  Is it possible that
> there are some typos above?  I'm thinking of the part that contains the
> tautology "without the sources or without them".

I meant to say that if a user builds with Git installed and Emacs sources 
checked out from git. Then packages Emacs but only puts the checkout
sources without the VCS information into the source package/tarball.

> There are also some words where the meaning is not entirely clear to me,
> such as "distributes" and "installs".

Distributed in this case meant to be the package build put into a
repository and then installed using a package manager.

> Maybe it will be clarified by the answers to my below questions.
>
>>> Do you mean that you have one containerized process with Git that clones
>>> emacs.git into a directory, and then an entirely separate containerized
>>> process, without Git, builds Emacs from that very same directory?  Or
>>> something along those lines?
>>
>> Yes more or less.
>> I'm using an obs source service that generates a tarball from VCS's and
>> then creates a tarball. [1]
>
> I took a look at the linked documentation, but it seems to be on a very
> detailed level, and does not provide a good overview.  So I don't think
> I understood much from it.
>
> I'm not familiar with the Open Build System, so this is quite hard for
> me to follow.  Please be patient with me.
>
> In your description above, could you explain what is the difference
> between the two steps "generates a tarball from VCS" and "creates a
> tarball"?

It should have be more detailed or used the proper word.
The program that I use to generate the sources fetches the sources from
a VCS such as git and then creates a cpio archive (for storing the
sources more efficiently but that's beyond this topic).

The cpio archive is then converted to a tar archive and compressed.

> Do you mean to say that it:
> 1) generates a tarball from emacs.git, excluding the .git directory
> 2) copies that tarball to a completely different environment
> 3) builds Emacs

Essentially yes. The point of this is that the environment that builds
the packages is disconnected from internet or other external factors for
security reasons.

The tarball is copied into the build environment while it prepares it
so that a minimal linux vm or chroot can start to build software.

> The part that I do not understand is in which step you run `make`, and
> how.  Is it run in the first step, the third, the first and the third,
> or something else?  Is it run automatically by OBS, or do you have to
> specify it?

I run make in in the 3 step right after the environment is setup up
inside rpmbuild.

Before make is called I parse the file that contains the reference hash
and the file that contains the configuration for the program that
generates the sources which contains the branch that the source where
generated from into the file that Emacs then can read during the build
process and after that when the package was installed.

The only part that's special here is that I generate the version
file from the information gathered while generating the source tarball
as opposed to using git to gather those.


> Basically, I still don't understand the answer to my below question:
>
>>> In this very particular scenario, isn't it enough to add this
>>> additional step to your CI pipeline:
>>>
>>>     ./src/emacs -Q --batch --eval '(princ (format "%s:%s" \
>>>     emacs-repository-version emacs-repository-branch))' > version.info
>> The obs source service already contains that information see the
>> emacs.obsinfo file I mentioned earlier.
>
> Hmm.  Would an alternative solution be to read the version information
> from that file then, if it exists?
>
>>> In other words, is it really necessary for us to support this use case
>>> in our Makefile?  Do we expect that building Emacs in such CI pipelines
>>> using non-released development version of Emacs will be very common?
>>
>> The Makefile target is there already for the Android builds all I did I
>> added it to other platforms too. If the target would be shared then the
>> hole process is reused.
>
> If we want this for other builds, then reusing what we do for the
> Android build sounds prudent.  It is clear that this is needed for
> Android.  What remains to clarify is why it is needed elsewhere.
>
>> I think it would become more common to build from VCS sources would be
>> become more common where the information is useful.
>
> Building from VCS sources is already quite common.  I'm asking something
> more specific: will it be increasingly common to build using an
> environment very similar to OBS?

I think so. It's already quite common to build software in a much
simpler CI such as Gitlab CI. Fedora has their own CI similar to OBS for example.

> If it is not going to be very common, this should perhaps be fixed in
> the OBS pipeline directly, instead of complicating our Makefile.  But
> I'm not yet clear on whether that is true.

The makefile only has the rule add or moved for the version file.
To me this just looks like refactoring so the logic that was added for
Android can used on other platforms.

>> To get more users to test earlier development builds to have faster
>> general feedback and bug reports reports it is very useful to have
>> non-released development builds. There are other instances outside my
>> own usecase where this is frequent such as for example the AUR emacs-git, guix
>> emacs-next or the Android Fdroid builds.
>
> Are you saying that AUR emacs-git, guix emacs-next, and the Android
> Fdroid builds also have no way to know which revision Emacs was built
> from without your patch?

The Android builds obviously have the revision already integrated
as the logic that I'm trying to reuse here have been already
implemented for Android.
I don't know about the Guix build process but for Arch git isn't
into the change-root when building Emacs from the tarball that has been
generated before hand.

The OBS can build packages for all kinds of Linux distributions such as
Flatpak, Debian, Fedora, SUSE, Ubuntu etc.

So this change can help building test builds for all these distributions
in theory. Personally just building for SUSE.

>>> BTW, what is the name of that CI system that you're using here?  For
>>> what purpose are you building Emacs: to test Elisp packages, to test
>>> Emacs, or something else?  Finally, why are you not using officially
>>> tagged versions, either from us or from some distro, in this context?
>>
>> I'm build Emacs on the open build service. I'm building Emacs on the obs
>> to provide development builds that can be used to test Emacs and submit
>> bug reports.
>> The package is build using the official RPM packaging just
>> with the newer sources from git.
>>
>> The initial reason reason was that I noticed that even if you build from
>> git directly without a CI/or something similar to the source service
>> mentioned earlier that the functions to retrieve the repository
>> information don't work since the source path either doesn't exist on any
>> the machine that installs the final package or that it doesn't contain
>> the VCS information anymore.
>> The setup is very similar to the Android builds.
>
> Thanks.

No problem.