* [bug#51122] [PATCH] maint: Factorize po xref translation. @ 2021-10-10 21:32 Julien Lepiller 2021-10-10 21:37 ` Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-10 21:32 UTC (permalink / raw) To: 51122 This ensures we use the same method in "make" as in "guix/self.scm". * Makefile.am: Build guix/build/po.scm. * build-aux/convert-xref.scm: New file. * doc/local.mk (xref_command): Use it. * guix/self.scm (translate-cross-references): Move it... * guix/build/po.scm (translate-cross-references): ...here. --- Makefile.am | 3 +- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------- guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++-- guix/self.scm | 80 ++++---------------------- 5 files changed, 155 insertions(+), 98 deletions(-) create mode 100644 build-aux/convert-xref.scm diff --git a/Makefile.am b/Makefile.am index 635147efc1..dc53b2c810 100644 --- a/Makefile.am +++ b/Makefile.am @@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \ # Modules that are not compiled but are installed nonetheless, such as # build-side modules with unusual dependencies. MODULES_NOT_COMPILED = \ - guix/build/po.scm \ guix/man-db.scm include gnu/local.mk @@ -227,6 +226,7 @@ MODULES = \ guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \ + guix/build/po.scm \ guix/build/profiles.scm \ guix/build/compile.scm \ guix/build/rpath.scm \ @@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \ + build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \ diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm new file mode 100644 index 0000000000..47c8828857 --- /dev/null +++ b/build-aux/convert-xref.scm @@ -0,0 +1,26 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +;; Translate cross-references in a translated .texi manual. + +(use-modules (guix build po) + (ice-9 match)) + +(match (command-line) + ((program texi pofile) + (translate-cross-references texi pofile))) diff --git a/doc/local.mk b/doc/local.mk index 8340b75a87..88f7e67155 100644 --- a/doc/local.mk +++ b/doc/local.mk @@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately. -# Because @pxref and @xref commands are reference to a section by name, they +# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist. -# This command loops through the translated files looking for references. For -# each of these references, it tries to find the translation and replaces the -# reference name, even in untranslated strings. -# The last sed is a multiline sed because some references span multiple lines. define xref_command -cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \ - tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \ - sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \ - if [ -n "$$e" ]; then \ - line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\ - ((line++)) ;\ - if [ "$$line" != "1" ]; then \ - translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\ - if [ "$$translation" != "" ]; then \ - sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\ - fi ;\ - fi ;\ - fi ;\ -done +$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \ + "$(top_srcdir)/build-aux/convert-xref.scm" \ + $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi +$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po +$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po +$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" diff --git a/guix/build/po.scm b/guix/build/po.scm index eb9690ad1a..a167252244 100644 --- a/guix/build/po.scm +++ b/guix/build/po.scm @@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports) - #:export (read-po-file)) + #:use-module (ice-9 vlist) + #:use-module (srfi srfi-1) + #:export (read-po-file + translate-cross-references)) ;; A small parser for po files -(define-peg-pattern po-file body (* (or comment entry whitespace))) +(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n")) +(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all - (and (ignore (* whitespace)) (ignore "msgid ") msgid - (ignore (* whitespace)) (ignore "msgstr ") msgstr)) + (and (* (or flags comment (ignore (* whitespace)))) + (ignore "msgid ") msgid (ignore (* whitespace)) + (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\") @@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree) - "Converts a po PARSE-TREE to an association list." + "Converts a po PARSE-TREE to an association list, where the key is the msgid +and the value is the msgstr. The result only contains non fuzzy strings." + (define (comments->flags comments) + (match comments + (('flags flags) + (map (lambda (flag) (string->symbol (string-trim-both flag #\space))) + (string-split flags #\,))) + ((? list? comments) + (fold + (lambda (comment res) + (match comment + ((? string? _) res) + (flags + (append (comments->flags flags) + res)))) + '() + comments)))) + (match parse-tree (() '()) ((entry . parse-tree) @@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree)) + (('entry _ ('msgid msgid) 'msgstr) + (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr) - (parse-tree->assoc parse-tree))))))) + (parse-tree->assoc parse-tree))) + (('entry ('msgid msgid) ('msgstr msgstr)) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree))) + (('entry comments ('msgid msgid) ('msgstr msgstr)) + (if (member 'fuzzy (comments->flags comments)) + (parse-tree->assoc parse-tree) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr." @@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree))) + +(define (canonicalize-whitespace str) + "Change whitespace (newlines, etc.) in STR to @code{#\\space}." + (string-map (lambda (chr) + (if (char-set-contains? char-set:whitespace chr) + #\space + chr)) + str)) + +(define xref-regexp + ;; Texinfo cross-reference regexp. + (make-regexp "@(px|x)?ref\\{([^,}]+)")) + +(define (translate-cross-references texi pofile) + "Translate the cross-references that appear in @var{texi}, the initial +translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}." + (define translations + (call-with-input-file pofile read-po-file)) + + (define content + (call-with-input-file texi get-string-all)) + + (define matches + (list-matches xref-regexp content)) + + (define translation-map + (fold (match-lambda* + (((msgid . str) result) + (vhash-cons msgid str result))) + vlist-null + translations)) + + (define translated + ;; Iterate over MATCHES and replace cross-references with their + ;; translation found in TRANSLATION-MAP. (We can't use + ;; 'substitute*' because matches can span multiple lines.) + (let loop ((matches matches) + (offset 0) + (result '())) + (match matches + (() + (string-concatenate-reverse + (cons (string-drop content offset) result))) + ((head . tail) + (let ((prefix (match:substring head 1)) + (ref (canonicalize-whitespace (match:substring head 2)))) + (define translated + (string-append "@" (or prefix "") + "ref{" + (match (vhash-assoc ref translation-map) + (#f ref) + ((_ . str) str)))) + + (loop tail + (match:end head) + (append (list translated + (string-take + (string-drop content offset) + (- (match:start head) offset))) + result))))))) + + (format (current-error-port) + "translated ~a cross-references in '~a'~%" + (length matches) texi) + + (call-with-output-file texi + (lambda (port) + (display translated port)))) diff --git a/guix/self.scm b/guix/self.scm index 61ff423086..769e0e8a7c 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -316,81 +316,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp - ;; Texinfo cross-reference regexp. - (make-regexp "@(px|x)?ref\\{([^,}]+)")) - - (define (translate-cross-references texi translations) - ;; Translate the cross-references that appear in TEXI, a Texinfo - ;; file, using the msgid/msgstr pairs from TRANSLATIONS. - (define content - (call-with-input-file texi get-string-all)) - - (define matches - (list-matches xref-regexp content)) - - (define translation-map - (fold (match-lambda* - (((msgid . str) result) - (vhash-cons msgid str result))) - vlist-null - translations)) - - (define translated - ;; Iterate over MATCHES and replace cross-references with their - ;; translation found in TRANSLATION-MAP. (We can't use - ;; 'substitute*' because matches can span multiple lines.) - (let loop ((matches matches) - (offset 0) - (result '())) - (match matches - (() - (string-concatenate-reverse - (cons (string-drop content offset) result))) - ((head . tail) - (let ((prefix (match:substring head 1)) - (ref (canonicalize-whitespace (match:substring head 2)))) - (define translated - (string-append "@" (or prefix "") - "ref{" - (match (vhash-assoc ref translation-map) - (#f ref) - ((_ . str) str)))) - - (loop tail - (match:end head) - (append (list translated - (string-take - (string-drop content offset) - (- (match:start head) offset))) - result))))))) - - (format (current-error-port) - "translated ~a cross-references in '~a'~%" - (length matches) texi) - (call-with-output-file texi - (lambda (port) - (display translated port)))) - (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")." - (let ((translations (call-with-input-file po read-po-file))) - (for-each (lambda (file) - (translate-tmp-texi po (string-append file ".texi") - (string-append file "." lang - ".texi.tmp"))) - (cons prefix extras)) + (for-each (lambda (file) + (translate-tmp-texi po (string-append file ".texi") + (string-append file "." lang + ".texi.tmp"))) + (cons prefix extras)) - (for-each (lambda (file) - (let* ((texi (string-append file "." lang ".texi")) - (tmp (string-append texi ".tmp"))) - (copy-file tmp texi) - (translate-cross-references texi - translations))) - (cons prefix extras)))) + (for-each (lambda (file) + (let* ((texi (string-append file "." lang ".texi")) + (tmp (string-append texi ".tmp"))) + (copy-file tmp texi) + (translate-cross-references texi po))) + (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for -- 2.33.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH] maint: Factorize po xref translation. 2021-10-10 21:32 [bug#51122] [PATCH] maint: Factorize po xref translation Julien Lepiller @ 2021-10-10 21:37 ` Julien Lepiller 2021-10-11 8:45 ` pelzflorian (Florian Pelz) 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-10 21:37 UTC (permalink / raw) To: 51122 For info, I built "make" and "make as-derivation" successfully with this patch. I need it because with recent changes in guix.texi, a lot of text has become fuzzy, including node names, which are translated by the current xref_command (in doc/local.mk), when they shouldn't. I improved the method we used for guix pull, and used it instead. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH] maint: Factorize po xref translation. 2021-10-10 21:37 ` Julien Lepiller @ 2021-10-11 8:45 ` pelzflorian (Florian Pelz) 2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: pelzflorian (Florian Pelz) @ 2021-10-11 8:45 UTC (permalink / raw) To: Julien Lepiller; +Cc: 51122 Hi Julien! Thank you for replacing that hard-to-read sed script. 65;6003;1c On Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote: > For info, I built "make" and "make as-derivation" successfully with > this patch. I need it because with recent changes in guix.texi, a lot > of text has become fuzzy, including node names, which are translated by > the current xref_command (in doc/local.mk), when they shouldn't. I > improved the method we used for guix pull, and used it instead. I think you should add to the commit message that your patch Fixes fuzzy translations being inserted. Your changes to guix/build/po.scm do more than just move code (which you mentioned in the commit message): you added parsing of “#, fuzzy” flags. You should update the copyright header for guix/build/po.scm. Otherwise LGTM, but I haven’t tested yet. (make now converts the xref after scheme modules are compiled, I think(?), so testing PO files takes longer for me on a clean checkout. But it is fine.) Regards, Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-11 8:45 ` pelzflorian (Florian Pelz) @ 2021-10-11 12:46 ` Julien Lepiller 2021-10-11 17:17 ` pelzflorian (Florian Pelz) 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-11 12:46 UTC (permalink / raw) To: pelzflorian (Florian Pelz); +Cc: 51122 [-- Attachment #1: Type: text/plain, Size: 1642 bytes --] Le Mon, 11 Oct 2021 10:45:45 +0200, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit : > Hi Julien! Thank you for replacing that hard-to-read sed script. > 65;6003;1c > On Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote: > > For info, I built "make" and "make as-derivation" successfully with > > this patch. I need it because with recent changes in guix.texi, a > > lot of text has become fuzzy, including node names, which are > > translated by the current xref_command (in doc/local.mk), when they > > shouldn't. I improved the method we used for guix pull, and used it > > instead. > > I think you should add to the commit message that your patch Fixes > fuzzy translations being inserted. > > Your changes to guix/build/po.scm do more than just move code (which > you mentioned in the commit message): you added parsing of “#, fuzzy” > flags. > > You should update the copyright header for guix/build/po.scm. > > Otherwise LGTM, but I haven’t tested yet. (make now converts the xref > after scheme modules are compiled, I think(?), so testing PO files > takes longer for me on a clean checkout. But it is fine.) > > Regards, > Florian Thanks for the quick review! I've updated the commit message and updated the copyright headers in guix/build/po.scm and guix/self.scm. I had to make the texi files depend on make-go because otherwise, I get messages like "guix/build/po.scm is newer than compiled .../guix/build/po.go" and it takes forever to build that file, once for each texi. For some reason, it's almost instantaneous when building in make-go. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-maint-Factorize-po-xref-translation.patch --] [-- Type: text/x-patch, Size: 17596 bytes --] From 75198f83fccf27b447e5eaaa6fb6caeed3b29408 Mon Sep 17 00:00:00 2001 Message-Id: <75198f83fccf27b447e5eaaa6fb6caeed3b29408.1633955976.git.julien@lepiller.eu> From: Julien Lepiller <julien@lepiller.eu> Date: Sun, 10 Oct 2021 22:07:51 +0200 Subject: [PATCH] maint: Factorize po xref translation. This ensures we use the same method in "make" as in "guix/self.scm". * Makefile.am: Build guix/build/po.scm. * build-aux/convert-xref.scm: New file. * doc/local.mk (xref_command): Use it. * guix/self.scm (translate-cross-references): Move it... * guix/build/po.scm: Parse comments and flags separately to find fuzzy flags. (translate-cross-references): ...here. (parse-tree->assoc): Ignore fuzzy entries. --- Makefile.am | 3 +- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------ guix/build/po.scm | 117 ++++++++++++++++++++++++++++++++++--- guix/self.scm | 81 ++++--------------------- 5 files changed, 157 insertions(+), 99 deletions(-) create mode 100644 build-aux/convert-xref.scm diff --git a/Makefile.am b/Makefile.am index 635147efc1..dc53b2c810 100644 --- a/Makefile.am +++ b/Makefile.am @@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \ # Modules that are not compiled but are installed nonetheless, such as # build-side modules with unusual dependencies. MODULES_NOT_COMPILED = \ - guix/build/po.scm \ guix/man-db.scm include gnu/local.mk @@ -227,6 +226,7 @@ MODULES = \ guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \ + guix/build/po.scm \ guix/build/profiles.scm \ guix/build/compile.scm \ guix/build/rpath.scm \ @@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \ + build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \ diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm new file mode 100644 index 0000000000..47c8828857 --- /dev/null +++ b/build-aux/convert-xref.scm @@ -0,0 +1,26 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +;; Translate cross-references in a translated .texi manual. + +(use-modules (guix build po) + (ice-9 match)) + +(match (command-line) + ((program texi pofile) + (translate-cross-references texi pofile))) diff --git a/doc/local.mk b/doc/local.mk index 8340b75a87..88f7e67155 100644 --- a/doc/local.mk +++ b/doc/local.mk @@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately. -# Because @pxref and @xref commands are reference to a section by name, they +# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist. -# This command loops through the translated files looking for references. For -# each of these references, it tries to find the translation and replaces the -# reference name, even in untranslated strings. -# The last sed is a multiline sed because some references span multiple lines. define xref_command -cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \ - tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \ - sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \ - if [ -n "$$e" ]; then \ - line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\ - ((line++)) ;\ - if [ "$$line" != "1" ]; then \ - translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\ - if [ "$$translation" != "" ]; then \ - sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\ - fi ;\ - fi ;\ - fi ;\ -done +$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \ + "$(top_srcdir)/build-aux/convert-xref.scm" \ + $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi +$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po +$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po +$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" diff --git a/guix/build/po.scm b/guix/build/po.scm index eb9690ad1a..7f88164cd8 100644 --- a/guix/build/po.scm +++ b/guix/build/po.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2019 Julien Lepiller <julien@lepiller.eu> +;;; Copyright © 2019, 2021 Julien Lepiller <julien@lepiller.eu> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix. @@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports) - #:export (read-po-file)) + #:use-module (ice-9 vlist) + #:use-module (srfi srfi-1) + #:export (read-po-file + translate-cross-references)) ;; A small parser for po files -(define-peg-pattern po-file body (* (or comment entry whitespace))) +(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n")) +(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all - (and (ignore (* whitespace)) (ignore "msgid ") msgid - (ignore (* whitespace)) (ignore "msgstr ") msgstr)) + (and (* (or flags comment (ignore (* whitespace)))) + (ignore "msgid ") msgid (ignore (* whitespace)) + (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\") @@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree) - "Converts a po PARSE-TREE to an association list." + "Converts a po PARSE-TREE to an association list, where the key is the msgid +and the value is the msgstr. The result only contains non fuzzy strings." + (define (comments->flags comments) + (match comments + (('flags flags) + (map (lambda (flag) (string->symbol (string-trim-both flag #\space))) + (string-split flags #\,))) + ((? list? comments) + (fold + (lambda (comment res) + (match comment + ((? string? _) res) + (flags + (append (comments->flags flags) + res)))) + '() + comments)))) + (match parse-tree (() '()) ((entry . parse-tree) @@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree)) + (('entry _ ('msgid msgid) 'msgstr) + (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr) - (parse-tree->assoc parse-tree))))))) + (parse-tree->assoc parse-tree))) + (('entry ('msgid msgid) ('msgstr msgstr)) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree))) + (('entry comments ('msgid msgid) ('msgstr msgstr)) + (if (member 'fuzzy (comments->flags comments)) + (parse-tree->assoc parse-tree) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr." @@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree))) + +(define (canonicalize-whitespace str) + "Change whitespace (newlines, etc.) in STR to @code{#\\space}." + (string-map (lambda (chr) + (if (char-set-contains? char-set:whitespace chr) + #\space + chr)) + str)) + +(define xref-regexp + ;; Texinfo cross-reference regexp. + (make-regexp "@(px|x)?ref\\{([^,}]+)")) + +(define (translate-cross-references texi pofile) + "Translate the cross-references that appear in @var{texi}, the initial +translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}." + (define translations + (call-with-input-file pofile read-po-file)) + + (define content + (call-with-input-file texi get-string-all)) + + (define matches + (list-matches xref-regexp content)) + + (define translation-map + (fold (match-lambda* + (((msgid . str) result) + (vhash-cons msgid str result))) + vlist-null + translations)) + + (define translated + ;; Iterate over MATCHES and replace cross-references with their + ;; translation found in TRANSLATION-MAP. (We can't use + ;; 'substitute*' because matches can span multiple lines.) + (let loop ((matches matches) + (offset 0) + (result '())) + (match matches + (() + (string-concatenate-reverse + (cons (string-drop content offset) result))) + ((head . tail) + (let ((prefix (match:substring head 1)) + (ref (canonicalize-whitespace (match:substring head 2)))) + (define translated + (string-append "@" (or prefix "") + "ref{" + (match (vhash-assoc ref translation-map) + (#f ref) + ((_ . str) str)))) + + (loop tail + (match:end head) + (append (list translated + (string-take + (string-drop content offset) + (- (match:start head) offset))) + result))))))) + + (format (current-error-port) + "translated ~a cross-references in '~a'~%" + (length matches) texi) + + (call-with-output-file texi + (lambda (port) + (display translated port)))) diff --git a/guix/self.scm b/guix/self.scm index 61ff423086..fc581becd4 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> ;;; ;;; This file is part of GNU Guix. ;;; @@ -316,81 +317,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp - ;; Texinfo cross-reference regexp. - (make-regexp "@(px|x)?ref\\{([^,}]+)")) - - (define (translate-cross-references texi translations) - ;; Translate the cross-references that appear in TEXI, a Texinfo - ;; file, using the msgid/msgstr pairs from TRANSLATIONS. - (define content - (call-with-input-file texi get-string-all)) - - (define matches - (list-matches xref-regexp content)) - - (define translation-map - (fold (match-lambda* - (((msgid . str) result) - (vhash-cons msgid str result))) - vlist-null - translations)) - - (define translated - ;; Iterate over MATCHES and replace cross-references with their - ;; translation found in TRANSLATION-MAP. (We can't use - ;; 'substitute*' because matches can span multiple lines.) - (let loop ((matches matches) - (offset 0) - (result '())) - (match matches - (() - (string-concatenate-reverse - (cons (string-drop content offset) result))) - ((head . tail) - (let ((prefix (match:substring head 1)) - (ref (canonicalize-whitespace (match:substring head 2)))) - (define translated - (string-append "@" (or prefix "") - "ref{" - (match (vhash-assoc ref translation-map) - (#f ref) - ((_ . str) str)))) - - (loop tail - (match:end head) - (append (list translated - (string-take - (string-drop content offset) - (- (match:start head) offset))) - result))))))) - - (format (current-error-port) - "translated ~a cross-references in '~a'~%" - (length matches) texi) - (call-with-output-file texi - (lambda (port) - (display translated port)))) - (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")." - (let ((translations (call-with-input-file po read-po-file))) - (for-each (lambda (file) - (translate-tmp-texi po (string-append file ".texi") - (string-append file "." lang - ".texi.tmp"))) - (cons prefix extras)) + (for-each (lambda (file) + (translate-tmp-texi po (string-append file ".texi") + (string-append file "." lang + ".texi.tmp"))) + (cons prefix extras)) - (for-each (lambda (file) - (let* ((texi (string-append file "." lang ".texi")) - (tmp (string-append texi ".tmp"))) - (copy-file tmp texi) - (translate-cross-references texi - translations))) - (cons prefix extras)))) + (for-each (lambda (file) + (let* ((texi (string-append file "." lang ".texi")) + (tmp (string-append texi ".tmp"))) + (copy-file tmp texi) + (translate-cross-references texi po))) + (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for -- 2.33.0 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller @ 2021-10-11 17:17 ` pelzflorian (Florian Pelz) 2021-10-11 22:06 ` Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: pelzflorian (Florian Pelz) @ 2021-10-11 17:17 UTC (permalink / raw) To: Julien Lepiller; +Cc: 51122 Thanks Julien! On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote: > Thanks for the quick review! I've updated the commit message and > updated the copyright headers in guix/build/po.scm and guix/self.scm. In guix/self.scm you only removed code. I don’t think a copyright header is needed there. > I had to make the texi files depend on make-go because otherwise, I get > messages like "guix/build/po.scm is newer than compiled > .../guix/build/po.go" and it takes forever to build that file, once for > each texi. For some reason, it's almost instantaneous when building in > make-go. make-core-go instead of make-go is enough in my testing, but something more fine-grained would save time. Also POXREF is run twice for each guix.??.texi file. Maybe I did something wrong, maybe it is your patch. Not sure. Testing takes long anyway. Regards, Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-11 17:17 ` pelzflorian (Florian Pelz) @ 2021-10-11 22:06 ` Julien Lepiller 2021-10-12 15:56 ` pelzflorian (Florian Pelz) 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-11 22:06 UTC (permalink / raw) To: pelzflorian (Florian Pelz); +Cc: 51122 Le Mon, 11 Oct 2021 19:17:28 +0200, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit : > Thanks Julien! > > On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote: > > Thanks for the quick review! I've updated the commit message and > > updated the copyright headers in guix/build/po.scm and > > guix/self.scm. > > In guix/self.scm you only removed code. I don’t think a copyright > header is needed there. Alright, I removed the copyright line on my side > > > I had to make the texi files depend on make-go because otherwise, I > > get messages like "guix/build/po.scm is newer than compiled > > .../guix/build/po.go" and it takes forever to build that file, once > > for each texi. For some reason, it's almost instantaneous when > > building in make-go. > > make-core-go instead of make-go is enough in my testing, but something > more fine-grained would save time. Also POXREF is run twice for each > guix.??.texi file. Maybe I did something wrong, maybe it is your > patch. Not sure. Testing takes long anyway. It only runs it once here, but there are three files to translate: guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi. It looks like depending on make-go or make-core-go means we rebuild the texi files everytime we run "make", so that's not great :/. I'll see if I can isolate guix/build/po.scm (any ideas?). > Regards, > Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-11 22:06 ` Julien Lepiller @ 2021-10-12 15:56 ` pelzflorian (Florian Pelz) 2021-10-12 18:09 ` pelzflorian (Florian Pelz) 0 siblings, 1 reply; 12+ messages in thread From: pelzflorian (Florian Pelz) @ 2021-10-12 15:56 UTC (permalink / raw) To: Julien Lepiller; +Cc: 51122 On Tue, Oct 12, 2021 at 12:06:57AM +0200, Julien Lepiller wrote: > Le Mon, 11 Oct 2021 19:17:28 +0200, > "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit : > > Also POXREF is run twice for each > > guix.??.texi file. Maybe I did something wrong, maybe it is your > > patch. Not sure. Testing takes long anyway. > It only runs it once here, but there are three files to translate: > guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi. At least when depending on make-core-go instead of make-go, it does POXREF each file twice, in between printing “Updating doc/version.texi”. Anyway, this is more important: > It looks like depending on make-go or make-core-go means we rebuild the > texi files everytime we run "make", so that's not great :/. We definitely want to compile guix/build/po.scm, I think, because it should be run each time a guix-manual.LL.po or guix-cookbook.LL.po file changes. GNU Make manual says <https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets>: > A phony target should not be a prerequisite of a real target file; if > it is, its recipe will be run every time make goes to update that > file. But Makefile.am defines guile-compilation-rule to create a Phony target. This is bad for us because likely there should not be duplicate code to compile guix/build/po.scm and to compile the rest. So I presume part of guile-compilation-rule must be split off? Regards, Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-12 15:56 ` pelzflorian (Florian Pelz) @ 2021-10-12 18:09 ` pelzflorian (Florian Pelz) 2021-10-12 19:25 ` Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: pelzflorian (Florian Pelz) @ 2021-10-12 18:09 UTC (permalink / raw) To: Julien Lepiller; +Cc: 51122 On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote: > We definitely want to compile guix/build/po.scm, I think, because it > should be run each time a guix-manual.LL.po or guix-cookbook.LL.po > file changes. Then again, is calling guix without compiling necessarily slower than the sed script? Regards, Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-12 18:09 ` pelzflorian (Florian Pelz) @ 2021-10-12 19:25 ` Julien Lepiller 2021-10-16 2:12 ` Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-12 19:25 UTC (permalink / raw) To: pelzflorian (Florian Pelz); +Cc: 51122 Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit : >On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote: >> We definitely want to compile guix/build/po.scm, I think, because it >> should be run each time a guix-manual.LL.po or guix-cookbook.LL.po >> file changes. > >Then again, is calling guix without compiling necessarily slower than >the sed script? > >Regards, >Florian Yes, each time I see the "newer than compiled" message, it takes several minutes. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-12 19:25 ` Julien Lepiller @ 2021-10-16 2:12 ` Julien Lepiller 2021-10-16 18:45 ` pelzflorian (Florian Pelz) 0 siblings, 1 reply; 12+ messages in thread From: Julien Lepiller @ 2021-10-16 2:12 UTC (permalink / raw) To: pelzflorian (Florian Pelz); +Cc: 51122 [-- Attachment #1: Type: text/plain, Size: 946 bytes --] Le Tue, 12 Oct 2021 15:25:37 -0400, Julien Lepiller <julien@lepiller.eu> a écrit : > Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)" > <pelzflorian@pelzflorian.de> a écrit : > >On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) > >wrote: > >> We definitely want to compile guix/build/po.scm, I think, because > >> it should be run each time a guix-manual.LL.po or > >> guix-cookbook.LL.po file changes. > > > >Then again, is calling guix without compiling necessarily slower than > >the sed script? > > > >Regards, > >Florian > > Yes, each time I see the "newer than compiled" message, it takes > several minutes. > > > I managed to change the dependency to only guix/po.go, and to make it not phony, so the translated manuals don't get rebuilt on every make invocation anymore. I checked again that "make" from a clean checkout and "make as-derivation" still work. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-maint-Factorize-po-xref-translation.patch --] [-- Type: text/x-patch, Size: 18092 bytes --] From 96d7096ac46f0fa8a97db76fefdf751a0584975b Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Sun, 10 Oct 2021 22:07:51 +0200 Subject: [PATCH] maint: Factorize po xref translation. This ensures we use the same method in "make" as in "guix/self.scm". * Makefile.am: Build guix/build/po.scm. * build-aux/convert-xref.scm: New file. * doc/local.mk (xref_command): Use it. * guix/self.scm (translate-cross-references): Move it... * guix/build/po.scm: Parse comments and flags separately to find fuzzy flags. (translate-cross-references): ...here. (parse-tree->assoc): Ignore fuzzy entries. --- Makefile.am | 12 +++- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------- guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++-- guix/self.scm | 81 ++++---------------------- 5 files changed, 164 insertions(+), 99 deletions(-) create mode 100644 build-aux/convert-xref.scm diff --git a/Makefile.am b/Makefile.am index 635147efc1..41ec19eb89 100644 --- a/Makefile.am +++ b/Makefile.am @@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \ + build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \ @@ -699,8 +700,6 @@ $(1): $(2) --completed $(3) \ $$(filter %.scm,$$^) -.PHONY: $(1) - endef # Split compilation in several steps, each of which building a subset of @@ -712,22 +711,31 @@ MODULES_CORE = guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULE MODULES_PACKAGES = $(filter gnu/packages/%,$(MODULES)) MODULES_SYSTEM = gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI = $(filter guix/scripts/%,$(MODULES)) +MODULES_PO = guix/build/po.scm $(eval $(call guile-compilation-rule,make-core-go, \ $(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \ 0)) +.PHONY: make-core-go $(eval $(call guile-compilation-rule,make-packages-go, \ $(MODULES_PACKAGES) make-core-go, \ $(words $(MODULES_CORE)))) +.PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ $(MODULES_SYSTEM) make-packages-go make-core-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES)))) +.PHONY: make-system-go $(eval $(call guile-compilation-rule,make-cli-go, \ $(MODULES_CLI) make-system-go make-packages-go make-core-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES) $(MODULES_SYSTEM)))) +.PHONY: make-cli-go + +$(eval $(call guile-compilation-rule,guix/build/po.go, \ + $(MODULES_PO), \ + 0)) SUFFIXES = .go diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm new file mode 100644 index 0000000000..47c8828857 --- /dev/null +++ b/build-aux/convert-xref.scm @@ -0,0 +1,26 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +;; Translate cross-references in a translated .texi manual. + +(use-modules (guix build po) + (ice-9 match)) + +(match (command-line) + ((program texi pofile) + (translate-cross-references texi pofile))) diff --git a/doc/local.mk b/doc/local.mk index 8340b75a87..fff11f8183 100644 --- a/doc/local.mk +++ b/doc/local.mk @@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately. -# Because @pxref and @xref commands are reference to a section by name, they +# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist. -# This command loops through the translated files looking for references. For -# each of these references, it tries to find the translation and replaces the -# reference name, even in untranslated strings. -# The last sed is a multiline sed because some references span multiple lines. define xref_command -cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \ - tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \ - sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \ - if [ -n "$$e" ]; then \ - line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\ - ((line++)) ;\ - if [ "$$line" != "1" ]; then \ - translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\ - if [ "$$translation" != "" ]; then \ - sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\ - fi ;\ - fi ;\ - fi ;\ -done +$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \ + "$(top_srcdir)/build-aux/convert-xref.scm" \ + $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi +$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po +$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po +$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" diff --git a/guix/build/po.scm b/guix/build/po.scm index eb9690ad1a..a167252244 100644 --- a/guix/build/po.scm +++ b/guix/build/po.scm @@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports) - #:export (read-po-file)) + #:use-module (ice-9 vlist) + #:use-module (srfi srfi-1) + #:export (read-po-file + translate-cross-references)) ;; A small parser for po files -(define-peg-pattern po-file body (* (or comment entry whitespace))) +(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n")) +(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all - (and (ignore (* whitespace)) (ignore "msgid ") msgid - (ignore (* whitespace)) (ignore "msgstr ") msgstr)) + (and (* (or flags comment (ignore (* whitespace)))) + (ignore "msgid ") msgid (ignore (* whitespace)) + (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\") @@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree) - "Converts a po PARSE-TREE to an association list." + "Converts a po PARSE-TREE to an association list, where the key is the msgid +and the value is the msgstr. The result only contains non fuzzy strings." + (define (comments->flags comments) + (match comments + (('flags flags) + (map (lambda (flag) (string->symbol (string-trim-both flag #\space))) + (string-split flags #\,))) + ((? list? comments) + (fold + (lambda (comment res) + (match comment + ((? string? _) res) + (flags + (append (comments->flags flags) + res)))) + '() + comments)))) + (match parse-tree (() '()) ((entry . parse-tree) @@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree)) + (('entry _ ('msgid msgid) 'msgstr) + (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr) - (parse-tree->assoc parse-tree))))))) + (parse-tree->assoc parse-tree))) + (('entry ('msgid msgid) ('msgstr msgstr)) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree))) + (('entry comments ('msgid msgid) ('msgstr msgstr)) + (if (member 'fuzzy (comments->flags comments)) + (parse-tree->assoc parse-tree) + (acons (interpret-newline-escape msgid) + (interpret-newline-escape msgstr) + (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr." @@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree))) + +(define (canonicalize-whitespace str) + "Change whitespace (newlines, etc.) in STR to @code{#\\space}." + (string-map (lambda (chr) + (if (char-set-contains? char-set:whitespace chr) + #\space + chr)) + str)) + +(define xref-regexp + ;; Texinfo cross-reference regexp. + (make-regexp "@(px|x)?ref\\{([^,}]+)")) + +(define (translate-cross-references texi pofile) + "Translate the cross-references that appear in @var{texi}, the initial +translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}." + (define translations + (call-with-input-file pofile read-po-file)) + + (define content + (call-with-input-file texi get-string-all)) + + (define matches + (list-matches xref-regexp content)) + + (define translation-map + (fold (match-lambda* + (((msgid . str) result) + (vhash-cons msgid str result))) + vlist-null + translations)) + + (define translated + ;; Iterate over MATCHES and replace cross-references with their + ;; translation found in TRANSLATION-MAP. (We can't use + ;; 'substitute*' because matches can span multiple lines.) + (let loop ((matches matches) + (offset 0) + (result '())) + (match matches + (() + (string-concatenate-reverse + (cons (string-drop content offset) result))) + ((head . tail) + (let ((prefix (match:substring head 1)) + (ref (canonicalize-whitespace (match:substring head 2)))) + (define translated + (string-append "@" (or prefix "") + "ref{" + (match (vhash-assoc ref translation-map) + (#f ref) + ((_ . str) str)))) + + (loop tail + (match:end head) + (append (list translated + (string-take + (string-drop content offset) + (- (match:start head) offset))) + result))))))) + + (format (current-error-port) + "translated ~a cross-references in '~a'~%" + (length matches) texi) + + (call-with-output-file texi + (lambda (port) + (display translated port)))) diff --git a/guix/self.scm b/guix/self.scm index a0d448742a..8ada1d85a4 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> ;;; ;;; This file is part of GNU Guix. ;;; @@ -316,81 +317,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp - ;; Texinfo cross-reference regexp. - (make-regexp "@(px|x)?ref\\{([^,}]+)")) - - (define (translate-cross-references texi translations) - ;; Translate the cross-references that appear in TEXI, a Texinfo - ;; file, using the msgid/msgstr pairs from TRANSLATIONS. - (define content - (call-with-input-file texi get-string-all)) - - (define matches - (list-matches xref-regexp content)) - - (define translation-map - (fold (match-lambda* - (((msgid . str) result) - (vhash-cons msgid str result))) - vlist-null - translations)) - - (define translated - ;; Iterate over MATCHES and replace cross-references with their - ;; translation found in TRANSLATION-MAP. (We can't use - ;; 'substitute*' because matches can span multiple lines.) - (let loop ((matches matches) - (offset 0) - (result '())) - (match matches - (() - (string-concatenate-reverse - (cons (string-drop content offset) result))) - ((head . tail) - (let ((prefix (match:substring head 1)) - (ref (canonicalize-whitespace (match:substring head 2)))) - (define translated - (string-append "@" (or prefix "") - "ref{" - (match (vhash-assoc ref translation-map) - (#f ref) - ((_ . str) str)))) - - (loop tail - (match:end head) - (append (list translated - (string-take - (string-drop content offset) - (- (match:start head) offset))) - result))))))) - - (format (current-error-port) - "translated ~a cross-references in '~a'~%" - (length matches) texi) - (call-with-output-file texi - (lambda (port) - (display translated port)))) - (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")." - (let ((translations (call-with-input-file po read-po-file))) - (for-each (lambda (file) - (translate-tmp-texi po (string-append file ".texi") - (string-append file "." lang - ".texi.tmp"))) - (cons prefix extras)) + (for-each (lambda (file) + (translate-tmp-texi po (string-append file ".texi") + (string-append file "." lang + ".texi.tmp"))) + (cons prefix extras)) - (for-each (lambda (file) - (let* ((texi (string-append file "." lang ".texi")) - (tmp (string-append texi ".tmp"))) - (copy-file tmp texi) - (translate-cross-references texi - translations))) - (cons prefix extras)))) + (for-each (lambda (file) + (let* ((texi (string-append file "." lang ".texi")) + (tmp (string-append texi ".tmp"))) + (copy-file tmp texi) + (translate-cross-references texi po))) + (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for -- 2.31.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation. 2021-10-16 2:12 ` Julien Lepiller @ 2021-10-16 18:45 ` pelzflorian (Florian Pelz) 2021-10-17 20:12 ` bug#51122: " Julien Lepiller 0 siblings, 1 reply; 12+ messages in thread From: pelzflorian (Florian Pelz) @ 2021-10-16 18:45 UTC (permalink / raw) To: Julien Lepiller; +Cc: 51122 On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote: > I managed to change the dependency to only guix/po.go, and to make it > not phony, so the translated manuals don't get rebuilt on every make > invocation anymore. I checked again that "make" from a clean checkout > and "make as-derivation" still work. Good, thank you! I had thought it would be more complicated than moving out the PHONY directive, but all works OK: Without patch, the fuzzy references cause make to fail for Weblate’s version of the Spanish manual. With patch, make runs through fine as you write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make update-guix-package && git commit and guix pull and still works, guix build guix failed in check but for unrelated reasons, so the patch LGTM. Except: As we discussed before, you only remove code from guix/self.scm, so you should not add a copyright line. Regards, Florian ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#51122: [PATCH v2] maint: Factorize po xref translation. 2021-10-16 18:45 ` pelzflorian (Florian Pelz) @ 2021-10-17 20:12 ` Julien Lepiller 0 siblings, 0 replies; 12+ messages in thread From: Julien Lepiller @ 2021-10-17 20:12 UTC (permalink / raw) To: pelzflorian (Florian Pelz); +Cc: 51122-done Le Sat, 16 Oct 2021 20:45:02 +0200, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit : > On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote: > > I managed to change the dependency to only guix/po.go, and to make > > it not phony, so the translated manuals don't get rebuilt on every > > make invocation anymore. I checked again that "make" from a clean > > checkout and "make as-derivation" still work. > > Good, thank you! I had thought it would be more complicated than > moving out the PHONY directive, but all works OK: > > Without patch, the fuzzy references cause make to fail for Weblate’s > version of the Spanish manual. With patch, make runs through fine as > you write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make > update-guix-package && git commit and guix pull and still works, guix > build guix failed in check but for unrelated reasons, so the patch > LGTM. Except: > > As we discussed before, you only remove code from guix/self.scm, so > you should not add a copyright line. > > Regards, > Florian Thanks, pushed as 0623138ffa5b066afc25547ffdeb97753cb0ee9a, and followed by an nls update. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-10-17 20:13 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-10-10 21:32 [bug#51122] [PATCH] maint: Factorize po xref translation Julien Lepiller 2021-10-10 21:37 ` Julien Lepiller 2021-10-11 8:45 ` pelzflorian (Florian Pelz) 2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller 2021-10-11 17:17 ` pelzflorian (Florian Pelz) 2021-10-11 22:06 ` Julien Lepiller 2021-10-12 15:56 ` pelzflorian (Florian Pelz) 2021-10-12 18:09 ` pelzflorian (Florian Pelz) 2021-10-12 19:25 ` Julien Lepiller 2021-10-16 2:12 ` Julien Lepiller 2021-10-16 18:45 ` pelzflorian (Florian Pelz) 2021-10-17 20:12 ` bug#51122: " Julien Lepiller
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).