[bug#47882] [PATCH] Makefile: Reimplement download-po target

[Julien Lepiller <julien@lepiller.eu>]

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]

Hi Guix!

The attached patch reimplements the download-po target in Makefile.am.
The previous implementation was based on the implementation for the TP
and would use the API of Weblate to get files.

However, the rate-limit is very close to the number of files we need to
download, which was a big limitation, and this target would only
download existing files, ignoring new translations.

With this, new translations and existing ones are added/updated. The
target will first do a shallow clone of the repository behind
Weblate. Then, it will check that files are not empty (contain at least
one translated string) and correct (according to scheme-format at
least). If they are correct, it will normalize them and copy them to
the po/ subdir in the repository.

Here is an example output:

copied po/doc/guix-manual.ru.po.
/tmp/tmp.dWvtATfWQA/translations/po/doc/guix-manual.si.po:6:
AVERTISSEMENT : Le champ d'en-tête « PO-Revision-Date » a encore sa
valeur initiale par défaut
WARN: po/doc/guix-manual.si.po (0 translated messages) was not
added/updated.
/tmp/tmp.dWvtATfWQA/translations/po/guix/nl.po:5135: Les spécifications
de format dans « msgid_plural » et « msgstr[1] » ne sont pas
équivalentes
msgfmt: 1 erreur fatale trouvée
WARN: po/guix/nl.po (1001 translated messages) was not added/updated.

In this output, we can see that the Russian manual was copied, the
Sinhala manual was empty, so not copied, and the Dutch translation had
an issue in one string that needs to be fixed, and was not copied.

I chose not to copy the file because an issue in msgfmt would break
guix pull. Unfortunately, this is not enough testing, as the manual and
cookbook are not checked. Checking them requires building the manual,
so before commiting changes, please run "make".

To document the new process:

make download-po
for any new file in po/guix and po/packages, add the language in a
single line in po/guix/LINGUAS or po/packages/LINGUAS.
for any new file in po/doc, add the file name to po/doc/local.mk, in
DOC_PO_FILES or DOC_COOKBOOK_PO_FILES depending on the translation
type. Add the texi file name to doc/local.mk too, in info_TEXINFOS.

[-- Attachment #2: 0001-Makefile-Reimplement-download-po-target.patch --]
[-- Type: text/x-patch, Size: 4276 bytes --]

From 5c14506d6af24b5307e03604cabf8ca10af56067 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Sun, 18 Apr 2021 23:56:48 +0200
Subject: [PATCH] Makefile: Reimplement `download-po` target.

The weblate API rate limit is very close to the number of files we need
to download.  The previous implementation did not add new translations.

* Makefile.am (download-po): Update target.
(make-download-po, make-check-po): Remove functions.
---
 Makefile.am | 87 ++++++++++++++---------------------------------------
 1 file changed, 22 insertions(+), 65 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 17ad236655..811e47dbfc 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -969,71 +969,28 @@ cuirass-jobs: $(GOBJECTS)
 
 # Downloading up-to-date PO files.
 
-# make-download-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX]
-define make-download-po-rule
-
-download-po.$(1):
-	if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then				\
-	  LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ;	\
-	else									\
-	  LINGUAS="`(cd $(top_srcdir)/$(2);					\
-	    for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ;		\
-	fi ;									\
-	for lang in $$$$LINGUAS; do						\
-	  if wget -nv -O "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp"		\
-	     "https://translate.fedoraproject.org/api/translations/guix/$(1)/$$$$lang/file/" ; \
-	  then									\
-	    msgfilter --no-wrap -i "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp"	\
-	      cat > "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp2" ;			\
-	    rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ;			\
-	    mv "$(top_srcdir)/$(2)/$(3)$$$$lang.po"{.tmp2,} ;			\
-	  else									\
-	    rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ;			\
-	  fi ;									\
-	done
-
-.PHONY: download-po.$(1)
-
-endef
-
-# Checking po files for issues.  This is useful to run after downloading new
-# po files.
-
-# make-check-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX]
-define make-check-po-rule
-
-check-po.$(1):
-	if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then				\
-	  LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ;	\
-	else									\
-	  LINGUAS="`(cd $(top_srcdir)/$(2);					\
-	    for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ;		\
-	fi ;									\
-	for lang in $$$$LINGUAS; do						\
-	  if [ -f "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ];			\
-	  then									\
-	    if ! msgfmt -c "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ;		\
-		then								\
-		  exit 1 ;							\
-	    fi ;								\
-	  fi ;									\
-	done
-
-.PHONY: check-po.$(1)
-
-endef
-
-$(eval $(call make-download-po-rule,documentation-cookbook,po/doc,guix-cookbook.))
-$(eval $(call make-download-po-rule,documentation-manual,po/doc,guix-manual.))
-$(eval $(call make-download-po-rule,guix,po/guix))
-$(eval $(call make-download-po-rule,packages,po/packages))
-
-$(eval $(call make-check-po-rule,documentation-cookbook,po/doc,guix-cookbook.))
-$(eval $(call make-check-po-rule,documentation-manual,po/doc,guix-manual.))
-$(eval $(call make-check-po-rule,guix,po/guix))
-$(eval $(call make-check-po-rule,packages,po/packages))
-
-download-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,download-po.$(domain))
+WEBLATE_REPO="https://framagit.org/tyreunom/guix-translations"
+
+# shallow clone the git repository behind weblate and copy files from it if
+# they contain at least one translation, and they are well-formed (scheme format
+# only), warn otherwise.  Copied files are converted to a canonical form.
+download-po:
+	dir=$$(mktemp -d); \
+	git clone --depth 1 "$(WEBLATE_REPO)" "$$dir/translations"; \
+	for domain in po/doc po/guix po/packages; do \
+		for po in "$$dir/translations/$$domain"/*.po; do \
+			translated=$$(LANG=en_US.UTF-8 msgfmt --statistics "$$po" 2>&1 | cut -f1 -d' '); \
+			target=$$(basename "$$po"); \
+			target="$$domain/$$target"; \
+			if msgfmt -c "$$po" && [ "$$translated" != "0" ]; then \
+				msgfilter --no-wrap -i "$$po" cat > "$$target"; \
+				echo "copied $$target."; \
+			else \
+				echo "WARN: $$target ($$translated translated messages) was not added/updated."; \
+			fi; \
+		done; \
+	done; \
+	rm -rf "$$dir"
 .PHONY: download-po
 
 check-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,check-po.$(domain))
-- 
2.31.1