From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id iCzPA4iufGAnhQAAgWs5BA (envelope-from ) for ; Mon, 19 Apr 2021 00:11:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id WGEzO4eufGAABQAA1q6Kng (envelope-from ) for ; Sun, 18 Apr 2021 22:11:19 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0C61E1BD45 for ; Mon, 19 Apr 2021 00:11:19 +0200 (CEST) Received: from localhost ([::1]:40610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lYFdN-0002qH-SJ for larch@yhetil.org; Sun, 18 Apr 2021 18:11:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35660) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYFd9-0002pz-Og for guix-patches@gnu.org; Sun, 18 Apr 2021 18:11:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:36459) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lYFd8-0007pr-Kw for guix-patches@gnu.org; Sun, 18 Apr 2021 18:11:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lYFd8-0005Ib-Fy for guix-patches@gnu.org; Sun, 18 Apr 2021 18:11:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47882] [PATCH] Makefile: Reimplement download-po target Resent-From: Julien Lepiller Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 18 Apr 2021 22:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 47882 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47882@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161878381720316 (code B ref -1); Sun, 18 Apr 2021 22:11:02 +0000 Received: (at submit) by debbugs.gnu.org; 18 Apr 2021 22:10:17 +0000 Received: from localhost ([127.0.0.1]:48005 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYFcL-0005HU-4h for submit@debbugs.gnu.org; Sun, 18 Apr 2021 18:10:17 -0400 Received: from lists.gnu.org ([209.51.188.17]:42208) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYFcJ-0005HN-W2 for submit@debbugs.gnu.org; Sun, 18 Apr 2021 18:10:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYFcJ-0002iB-RY for guix-patches@gnu.org; Sun, 18 Apr 2021 18:10:11 -0400 Received: from lepiller.eu ([2a00:5884:8208::1]:52124) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYFcG-0007GR-DC for guix-patches@gnu.org; Sun, 18 Apr 2021 18:10:10 -0400 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 4ffceb64 for ; Sun, 18 Apr 2021 22:10:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:subject:message-id:mime-version:content-type; s=dkim; bh=1HO dbailmMdwe29HPXqzU/LghhW34+gCwdzDdjVSjBM=; b=ObHtJKSx5/B8kLpx/Hm dkImKm1yK7CybjjA5hjZjMgwvM8oSQM7+/xDpHN9fK1jD876vXvQ1WC7m5nCjuqw g9O6q1DX6AWtUoTnp8oeEjZQZGqiHPkIE8PZNiRLcsEBhXVpv63666HNX4OG9wZl Q3AlZCI6LKK96zTGQd938zAgqdQ15AH4IlqJdH3ihZ/BAol83znfrSWOFXJm4zEh M7+GGGKsIXcwKQU9uSarSvWep3gJsnCcKJ3vkrDkjHiB8wp6LczvEnTJU/UFYmY6 Zi82gMNoUCa8ivGpsNAlVggJs7ETkf30Z9LrdhZCSqffxcBRPdJQSai10+l24JxW C+w== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 562f5a69 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Sun, 18 Apr 2021 22:10:03 +0000 (UTC) Date: Mon, 19 Apr 2021 00:09:58 +0200 From: Julien Lepiller Message-ID: <20210419000958.49968055@tachikoma.lepiller.eu> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/736Y_gLr=BoJc5_tZy53o=O" Received-SPF: pass client-ip=2a00:5884:8208::1; envelope-from=julien@lepiller.eu; helo=lepiller.eu X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618783879; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=eR9Ae6JD6c/8yCj2+8XRDuluf3QGLIzIaK6h+3nkeNg=; b=li2+Acq74f8ILoIaFypirX9yJzSgvN/Y0ay0bG82fXGQI0hmTABEuUbmZN19mrSE9EsKGe bHGYWIKmEWjrkmA2Kmxud1ShLuUJXO3/awGT+EqkNVSVfTfiW4GfP6kwDINHU7XKEJq37o KFnlCfxEgvOg1enDp19LIyC66qfQnTv43njR9c/L9axm13aXBN9pHJPS5n/q4w27B2U1RM grtJOKagLOY6kyXH6tV2zx7YpOda8neuolY1f28VWHaqJ6N9oqLsGcnonR0SK1/ehRvBHt KTlVjU/UqnQO83qxTaowfRRNGAUZtFdTChjRPYbcWMpKuLz/toOXWVBrTGb/TQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618783879; a=rsa-sha256; cv=none; b=YwFjlL5TKAJMra26/Ei0aCzAaOZ8nztIuanC9buE+rmRY66tw6/ll7MVR1vxvN6iBDi1Q6 ZCWTmPH9x4X0PQskTgAkM9rWU7n+Vh9HbHfPbBiBTV/RPq/9FIjXXuVzYbHoBKFUv5Tgmd iW8skRV8vrysAmRF/rgFD50ZNPiYCqwhy3UvsamgZmTuMpOlM6PPklH0/S3rncU5RMTnoy Kc6wQSck0sdRJ989/WxoQvml5viZGgMnZrGOTD6j1pMzgikqyG5L1TlqRL+aApwQRTgXoR jVsc5QL5Com9ksxSlbMOHvfpTormkbTo7ZOmcKPT3of+QrnavQhZKr7tlNYaUA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=lepiller.eu header.s=dkim header.b=ObHtJKSx; dmarc=fail reason="SPF not aligned (relaxed)" header.from=lepiller.eu (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.34 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=lepiller.eu header.s=dkim header.b=ObHtJKSx; dmarc=fail reason="SPF not aligned (relaxed)" header.from=lepiller.eu (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 0C61E1BD45 X-Spam-Score: -1.34 X-Migadu-Scanner: scn0.migadu.com X-TUID: m99fNMaBDrGU --MP_/736Y_gLr=BoJc5_tZy53o=O Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi Guix! The attached patch reimplements the download-po target in Makefile.am. The previous implementation was based on the implementation for the TP and would use the API of Weblate to get files. However, the rate-limit is very close to the number of files we need to download, which was a big limitation, and this target would only download existing files, ignoring new translations. With this, new translations and existing ones are added/updated. The target will first do a shallow clone of the repository behind Weblate. Then, it will check that files are not empty (contain at least one translated string) and correct (according to scheme-format at least). If they are correct, it will normalize them and copy them to the po/ subdir in the repository. Here is an example output: copied po/doc/guix-manual.ru.po. /tmp/tmp.dWvtATfWQA/translations/po/doc/guix-manual.si.po:6: AVERTISSEMENT=C2=A0: Le champ d'en-t=C3=AAte =C2=AB PO-Revision-Date =C2=BB= a encore sa valeur initiale par d=C3=A9faut WARN: po/doc/guix-manual.si.po (0 translated messages) was not added/updated. /tmp/tmp.dWvtATfWQA/translations/po/guix/nl.po:5135: Les sp=C3=A9cifications de format dans =C2=AB msgid_plural =C2=BB et =C2=AB=C2=A0msgstr[1]=C2=A0=C2= =BB ne sont pas =C3=A9quivalentes msgfmt: 1 erreur fatale trouv=C3=A9e WARN: po/guix/nl.po (1001 translated messages) was not added/updated. In this output, we can see that the Russian manual was copied, the Sinhala manual was empty, so not copied, and the Dutch translation had an issue in one string that needs to be fixed, and was not copied. I chose not to copy the file because an issue in msgfmt would break guix pull. Unfortunately, this is not enough testing, as the manual and cookbook are not checked. Checking them requires building the manual, so before commiting changes, please run "make". To document the new process: make download-po for any new file in po/guix and po/packages, add the language in a single line in po/guix/LINGUAS or po/packages/LINGUAS. for any new file in po/doc, add the file name to po/doc/local.mk, in DOC_PO_FILES or DOC_COOKBOOK_PO_FILES depending on the translation type. Add the texi file name to doc/local.mk too, in info_TEXINFOS. --MP_/736Y_gLr=BoJc5_tZy53o=O Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-Makefile-Reimplement-download-po-target.patch >From 5c14506d6af24b5307e03604cabf8ca10af56067 Mon Sep 17 00:00:00 2001 From: Julien Lepiller Date: Sun, 18 Apr 2021 23:56:48 +0200 Subject: [PATCH] Makefile: Reimplement `download-po` target. The weblate API rate limit is very close to the number of files we need to download. The previous implementation did not add new translations. * Makefile.am (download-po): Update target. (make-download-po, make-check-po): Remove functions. --- Makefile.am | 87 ++++++++++++++--------------------------------------- 1 file changed, 22 insertions(+), 65 deletions(-) diff --git a/Makefile.am b/Makefile.am index 17ad236655..811e47dbfc 100644 --- a/Makefile.am +++ b/Makefile.am @@ -969,71 +969,28 @@ cuirass-jobs: $(GOBJECTS) # Downloading up-to-date PO files. -# make-download-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX] -define make-download-po-rule - -download-po.$(1): - if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \ - LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \ - else \ - LINGUAS="`(cd $(top_srcdir)/$(2); \ - for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \ - fi ; \ - for lang in $$$$LINGUAS; do \ - if wget -nv -O "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \ - "https://translate.fedoraproject.org/api/translations/guix/$(1)/$$$$lang/file/" ; \ - then \ - msgfilter --no-wrap -i "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \ - cat > "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp2" ; \ - rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \ - mv "$(top_srcdir)/$(2)/$(3)$$$$lang.po"{.tmp2,} ; \ - else \ - rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \ - fi ; \ - done - -.PHONY: download-po.$(1) - -endef - -# Checking po files for issues. This is useful to run after downloading new -# po files. - -# make-check-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX] -define make-check-po-rule - -check-po.$(1): - if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \ - LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \ - else \ - LINGUAS="`(cd $(top_srcdir)/$(2); \ - for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \ - fi ; \ - for lang in $$$$LINGUAS; do \ - if [ -f "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ]; \ - then \ - if ! msgfmt -c "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ; \ - then \ - exit 1 ; \ - fi ; \ - fi ; \ - done - -.PHONY: check-po.$(1) - -endef - -$(eval $(call make-download-po-rule,documentation-cookbook,po/doc,guix-cookbook.)) -$(eval $(call make-download-po-rule,documentation-manual,po/doc,guix-manual.)) -$(eval $(call make-download-po-rule,guix,po/guix)) -$(eval $(call make-download-po-rule,packages,po/packages)) - -$(eval $(call make-check-po-rule,documentation-cookbook,po/doc,guix-cookbook.)) -$(eval $(call make-check-po-rule,documentation-manual,po/doc,guix-manual.)) -$(eval $(call make-check-po-rule,guix,po/guix)) -$(eval $(call make-check-po-rule,packages,po/packages)) - -download-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,download-po.$(domain)) +WEBLATE_REPO="https://framagit.org/tyreunom/guix-translations" + +# shallow clone the git repository behind weblate and copy files from it if +# they contain at least one translation, and they are well-formed (scheme format +# only), warn otherwise. Copied files are converted to a canonical form. +download-po: + dir=$$(mktemp -d); \ + git clone --depth 1 "$(WEBLATE_REPO)" "$$dir/translations"; \ + for domain in po/doc po/guix po/packages; do \ + for po in "$$dir/translations/$$domain"/*.po; do \ + translated=$$(LANG=en_US.UTF-8 msgfmt --statistics "$$po" 2>&1 | cut -f1 -d' '); \ + target=$$(basename "$$po"); \ + target="$$domain/$$target"; \ + if msgfmt -c "$$po" && [ "$$translated" != "0" ]; then \ + msgfilter --no-wrap -i "$$po" cat > "$$target"; \ + echo "copied $$target."; \ + else \ + echo "WARN: $$target ($$translated translated messages) was not added/updated."; \ + fi; \ + done; \ + done; \ + rm -rf "$$dir" .PHONY: download-po check-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,check-po.$(domain)) -- 2.31.1 --MP_/736Y_gLr=BoJc5_tZy53o=O--