* [bug#51122] [PATCH] maint: Factorize po xref translation.
@ 2021-10-10 21:32 Julien Lepiller
2021-10-10 21:37 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-10 21:32 UTC (permalink / raw)
To: 51122
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.
* build-aux/convert-xref.scm: New file.
* doc/local.mk (xref_command): Use it.
* guix/self.scm (translate-cross-references): Move it...
* guix/build/po.scm (translate-cross-references): ...here.
---
Makefile.am | 3 +-
build-aux/convert-xref.scm | 26 +++++++++
doc/local.mk | 29 +++-------
guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++--
guix/self.scm | 80 ++++----------------------
5 files changed, 155 insertions(+), 98 deletions(-)
create mode 100644 build-aux/convert-xref.scm
diff --git a/Makefile.am b/Makefile.am
index 635147efc1..dc53b2c810 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \
# Modules that are not compiled but are installed nonetheless, such as
# build-side modules with unusual dependencies.
MODULES_NOT_COMPILED = \
- guix/build/po.scm \
guix/man-db.scm
include gnu/local.mk
@@ -227,6 +226,7 @@ MODULES = \
guix/build/pack.scm \
guix/build/utils.scm \
guix/build/union.scm \
+ guix/build/po.scm \
guix/build/profiles.scm \
guix/build/compile.scm \
guix/build/rpath.scm \
@@ -640,6 +640,7 @@ EXTRA_DIST += \
build-aux/check-final-inputs-self-contained.scm \
build-aux/check-channel-news.scm \
build-aux/compile-as-derivation.scm \
+ build-aux/convert-xref.scm \
build-aux/generate-authors.scm \
build-aux/test-driver.scm \
build-aux/update-guix-package.scm \
diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm
new file mode 100644
index 0000000000..47c8828857
--- /dev/null
+++ b/build-aux/convert-xref.scm
@@ -0,0 +1,26 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+;; Translate cross-references in a translated .texi manual.
+
+(use-modules (guix build po)
+ (ice-9 match))
+
+(match (command-line)
+ ((program texi pofile)
+ (translate-cross-references texi pofile)))
diff --git a/doc/local.mk b/doc/local.mk
index 8340b75a87..88f7e67155 100644
--- a/doc/local.mk
+++ b/doc/local.mk
@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete
PO4A_PARAMS += -f texinfo # texinfo format
# When a change to guix.texi occurs, it is not translated immediately.
-# Because @pxref and @xref commands are reference to a section by name, they
+# Because @pxref and @xref commands are references to sections by name, they
# should be translated. If a modification adds a reference to a section, this
# reference is not translated, which means it references a section that does not
# exist.
-# This command loops through the translated files looking for references. For
-# each of these references, it tries to find the translation and replaces the
-# reference name, even in untranslated strings.
-# The last sed is a multiline sed because some references span multiple lines.
define xref_command
-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \
- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \
- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \
- if [ -n "$$e" ]; then \
- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\
- ((line++)) ;\
- if [ "$$line" != "1" ]; then \
- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\
- if [ "$$translation" != "" ]; then \
- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\
- fi ;\
- fi ;\
- fi ;\
-done
+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \
+ "$(top_srcdir)/build-aux/convert-xref.scm" \
+ $@.tmp $<
endef
-$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi
+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po
+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po
+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
diff --git a/guix/build/po.scm b/guix/build/po.scm
index eb9690ad1a..a167252244 100644
--- a/guix/build/po.scm
+++ b/guix/build/po.scm
@@ -20,17 +20,23 @@
(define-module (guix build po)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
+ #:use-module (ice-9 regex)
#:use-module (ice-9 textual-ports)
- #:export (read-po-file))
+ #:use-module (ice-9 vlist)
+ #:use-module (srfi srfi-1)
+ #:export (read-po-file
+ translate-cross-references))
;; A small parser for po files
-(define-peg-pattern po-file body (* (or comment entry whitespace)))
+(define-peg-pattern po-file body (* (or entry whitespace)))
(define-peg-pattern whitespace body (or " " "\t" "\n"))
(define-peg-pattern comment-chr body (range #\space #\頋))
(define-peg-pattern comment none (and "#" (* comment-chr) "\n"))
+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n")))
(define-peg-pattern entry all
- (and (ignore (* whitespace)) (ignore "msgid ") msgid
- (ignore (* whitespace)) (ignore "msgstr ") msgstr))
+ (and (* (or flags comment (ignore (* whitespace))))
+ (ignore "msgid ") msgid (ignore (* whitespace))
+ (ignore "msgstr ") msgstr))
(define-peg-pattern escape body (or "\\\\" "\\\"" "\\n"))
(define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"")
"\\n" (and (ignore "\\") "\\")
@@ -53,7 +59,24 @@
(append (list "\n" prefix) result)))))))
(define (parse-tree->assoc parse-tree)
- "Converts a po PARSE-TREE to an association list."
+ "Converts a po PARSE-TREE to an association list, where the key is the msgid
+and the value is the msgstr. The result only contains non fuzzy strings."
+ (define (comments->flags comments)
+ (match comments
+ (('flags flags)
+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))
+ (string-split flags #\,)))
+ ((? list? comments)
+ (fold
+ (lambda (comment res)
+ (match comment
+ ((? string? _) res)
+ (flags
+ (append (comments->flags flags)
+ res))))
+ '()
+ comments))))
+
(match parse-tree
(() '())
((entry . parse-tree)
@@ -66,10 +89,22 @@
;; empty msgstr
(('entry ('msgid msgid) 'msgstr)
(parse-tree->assoc parse-tree))
+ (('entry _ ('msgid msgid) 'msgstr)
+ (parse-tree->assoc parse-tree))
(('entry ('msgid msgid) ('msgstr msgstr))
(acons (interpret-newline-escape msgid)
(interpret-newline-escape msgstr)
- (parse-tree->assoc parse-tree)))))))
+ (parse-tree->assoc parse-tree)))
+ (('entry ('msgid msgid) ('msgstr msgstr))
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree)))
+ (('entry comments ('msgid msgid) ('msgstr msgstr))
+ (if (member 'fuzzy (comments->flags comments))
+ (parse-tree->assoc parse-tree)
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree))))))))
(define (read-po-file port)
"Read a .po file from PORT and return an alist of msgid and msgstr."
@@ -77,3 +112,71 @@
po-file
(get-string-all port)))))
(parse-tree->assoc tree)))
+
+(define (canonicalize-whitespace str)
+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."
+ (string-map (lambda (chr)
+ (if (char-set-contains? char-set:whitespace chr)
+ #\space
+ chr))
+ str))
+
+(define xref-regexp
+ ;; Texinfo cross-reference regexp.
+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))
+
+(define (translate-cross-references texi pofile)
+ "Translate the cross-references that appear in @var{texi}, the initial
+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."
+ (define translations
+ (call-with-input-file pofile read-po-file))
+
+ (define content
+ (call-with-input-file texi get-string-all))
+
+ (define matches
+ (list-matches xref-regexp content))
+
+ (define translation-map
+ (fold (match-lambda*
+ (((msgid . str) result)
+ (vhash-cons msgid str result)))
+ vlist-null
+ translations))
+
+ (define translated
+ ;; Iterate over MATCHES and replace cross-references with their
+ ;; translation found in TRANSLATION-MAP. (We can't use
+ ;; 'substitute*' because matches can span multiple lines.)
+ (let loop ((matches matches)
+ (offset 0)
+ (result '()))
+ (match matches
+ (()
+ (string-concatenate-reverse
+ (cons (string-drop content offset) result)))
+ ((head . tail)
+ (let ((prefix (match:substring head 1))
+ (ref (canonicalize-whitespace (match:substring head 2))))
+ (define translated
+ (string-append "@" (or prefix "")
+ "ref{"
+ (match (vhash-assoc ref translation-map)
+ (#f ref)
+ ((_ . str) str))))
+
+ (loop tail
+ (match:end head)
+ (append (list translated
+ (string-take
+ (string-drop content offset)
+ (- (match:start head) offset)))
+ result)))))))
+
+ (format (current-error-port)
+ "translated ~a cross-references in '~a'~%"
+ (length matches) texi)
+
+ (call-with-output-file texi
+ (lambda (port)
+ (display translated port))))
diff --git a/guix/self.scm b/guix/self.scm
index 61ff423086..769e0e8a7c 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -316,81 +316,23 @@ the result to OUTPUT."
chr))
str))
- (define xref-regexp
- ;; Texinfo cross-reference regexp.
- (make-regexp "@(px|x)?ref\\{([^,}]+)"))
-
- (define (translate-cross-references texi translations)
- ;; Translate the cross-references that appear in TEXI, a Texinfo
- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.
- (define content
- (call-with-input-file texi get-string-all))
-
- (define matches
- (list-matches xref-regexp content))
-
- (define translation-map
- (fold (match-lambda*
- (((msgid . str) result)
- (vhash-cons msgid str result)))
- vlist-null
- translations))
-
- (define translated
- ;; Iterate over MATCHES and replace cross-references with their
- ;; translation found in TRANSLATION-MAP. (We can't use
- ;; 'substitute*' because matches can span multiple lines.)
- (let loop ((matches matches)
- (offset 0)
- (result '()))
- (match matches
- (()
- (string-concatenate-reverse
- (cons (string-drop content offset) result)))
- ((head . tail)
- (let ((prefix (match:substring head 1))
- (ref (canonicalize-whitespace (match:substring head 2))))
- (define translated
- (string-append "@" (or prefix "")
- "ref{"
- (match (vhash-assoc ref translation-map)
- (#f ref)
- ((_ . str) str))))
-
- (loop tail
- (match:end head)
- (append (list translated
- (string-take
- (string-drop content offset)
- (- (match:start head) offset)))
- result)))))))
-
- (format (current-error-port)
- "translated ~a cross-references in '~a'~%"
- (length matches) texi)
- (call-with-output-file texi
- (lambda (port)
- (display translated port))))
-
(define* (translate-texi prefix po lang
#:key (extras '()))
"Translate the manual for one language LANG using the PO file.
PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is
a list of extra files, such as '(\"contributing\")."
- (let ((translations (call-with-input-file po read-po-file)))
- (for-each (lambda (file)
- (translate-tmp-texi po (string-append file ".texi")
- (string-append file "." lang
- ".texi.tmp")))
- (cons prefix extras))
+ (for-each (lambda (file)
+ (translate-tmp-texi po (string-append file ".texi")
+ (string-append file "." lang
+ ".texi.tmp")))
+ (cons prefix extras))
- (for-each (lambda (file)
- (let* ((texi (string-append file "." lang ".texi"))
- (tmp (string-append texi ".tmp")))
- (copy-file tmp texi)
- (translate-cross-references texi
- translations)))
- (cons prefix extras))))
+ (for-each (lambda (file)
+ (let* ((texi (string-append file "." lang ".texi"))
+ (tmp (string-append texi ".tmp")))
+ (copy-file tmp texi)
+ (translate-cross-references texi po)))
+ (cons prefix extras)))
(define (available-translations directory domain)
;; Return the list of available translations under DIRECTORY for
--
2.33.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH] maint: Factorize po xref translation.
2021-10-10 21:32 [bug#51122] [PATCH] maint: Factorize po xref translation Julien Lepiller
@ 2021-10-10 21:37 ` Julien Lepiller
2021-10-11 8:45 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-10 21:37 UTC (permalink / raw)
To: 51122
For info, I built "make" and "make as-derivation" successfully with
this patch. I need it because with recent changes in guix.texi, a lot
of text has become fuzzy, including node names, which are translated by
the current xref_command (in doc/local.mk), when they shouldn't. I
improved the method we used for guix pull, and used it instead.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH] maint: Factorize po xref translation.
2021-10-10 21:37 ` Julien Lepiller
@ 2021-10-11 8:45 ` pelzflorian (Florian Pelz)
2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-11 8:45 UTC (permalink / raw)
To: Julien Lepiller; +Cc: 51122
Hi Julien! Thank you for replacing that hard-to-read sed script.
65;6003;1c
On Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote:
> For info, I built "make" and "make as-derivation" successfully with
> this patch. I need it because with recent changes in guix.texi, a lot
> of text has become fuzzy, including node names, which are translated by
> the current xref_command (in doc/local.mk), when they shouldn't. I
> improved the method we used for guix pull, and used it instead.
I think you should add to the commit message that your patch Fixes
fuzzy translations being inserted.
Your changes to guix/build/po.scm do more than just move code (which
you mentioned in the commit message): you added parsing of “#, fuzzy”
flags.
You should update the copyright header for guix/build/po.scm.
Otherwise LGTM, but I haven’t tested yet. (make now converts the xref
after scheme modules are compiled, I think(?), so testing PO files
takes longer for me on a clean checkout. But it is fine.)
Regards,
Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-11 8:45 ` pelzflorian (Florian Pelz)
@ 2021-10-11 12:46 ` Julien Lepiller
2021-10-11 17:17 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-11 12:46 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 51122
[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]
Le Mon, 11 Oct 2021 10:45:45 +0200,
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
> Hi Julien! Thank you for replacing that hard-to-read sed script.
> 65;6003;1c
> On Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote:
> > For info, I built "make" and "make as-derivation" successfully with
> > this patch. I need it because with recent changes in guix.texi, a
> > lot of text has become fuzzy, including node names, which are
> > translated by the current xref_command (in doc/local.mk), when they
> > shouldn't. I improved the method we used for guix pull, and used it
> > instead.
>
> I think you should add to the commit message that your patch Fixes
> fuzzy translations being inserted.
>
> Your changes to guix/build/po.scm do more than just move code (which
> you mentioned in the commit message): you added parsing of “#, fuzzy”
> flags.
>
> You should update the copyright header for guix/build/po.scm.
>
> Otherwise LGTM, but I haven’t tested yet. (make now converts the xref
> after scheme modules are compiled, I think(?), so testing PO files
> takes longer for me on a clean checkout. But it is fine.)
>
> Regards,
> Florian
Thanks for the quick review! I've updated the commit message and
updated the copyright headers in guix/build/po.scm and guix/self.scm.
I had to make the texi files depend on make-go because otherwise, I get
messages like "guix/build/po.scm is newer than compiled
.../guix/build/po.go" and it takes forever to build that file, once for
each texi. For some reason, it's almost instantaneous when building in
make-go.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-maint-Factorize-po-xref-translation.patch --]
[-- Type: text/x-patch, Size: 17596 bytes --]
From 75198f83fccf27b447e5eaaa6fb6caeed3b29408 Mon Sep 17 00:00:00 2001
Message-Id: <75198f83fccf27b447e5eaaa6fb6caeed3b29408.1633955976.git.julien@lepiller.eu>
From: Julien Lepiller <julien@lepiller.eu>
Date: Sun, 10 Oct 2021 22:07:51 +0200
Subject: [PATCH] maint: Factorize po xref translation.
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.
* build-aux/convert-xref.scm: New file.
* doc/local.mk (xref_command): Use it.
* guix/self.scm (translate-cross-references): Move it...
* guix/build/po.scm: Parse comments and flags separately to find fuzzy
flags.
(translate-cross-references): ...here.
(parse-tree->assoc): Ignore fuzzy entries.
---
Makefile.am | 3 +-
build-aux/convert-xref.scm | 26 +++++++++
doc/local.mk | 29 +++------
guix/build/po.scm | 117 ++++++++++++++++++++++++++++++++++---
guix/self.scm | 81 ++++---------------------
5 files changed, 157 insertions(+), 99 deletions(-)
create mode 100644 build-aux/convert-xref.scm
diff --git a/Makefile.am b/Makefile.am
index 635147efc1..dc53b2c810 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \
# Modules that are not compiled but are installed nonetheless, such as
# build-side modules with unusual dependencies.
MODULES_NOT_COMPILED = \
- guix/build/po.scm \
guix/man-db.scm
include gnu/local.mk
@@ -227,6 +226,7 @@ MODULES = \
guix/build/pack.scm \
guix/build/utils.scm \
guix/build/union.scm \
+ guix/build/po.scm \
guix/build/profiles.scm \
guix/build/compile.scm \
guix/build/rpath.scm \
@@ -640,6 +640,7 @@ EXTRA_DIST += \
build-aux/check-final-inputs-self-contained.scm \
build-aux/check-channel-news.scm \
build-aux/compile-as-derivation.scm \
+ build-aux/convert-xref.scm \
build-aux/generate-authors.scm \
build-aux/test-driver.scm \
build-aux/update-guix-package.scm \
diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm
new file mode 100644
index 0000000000..47c8828857
--- /dev/null
+++ b/build-aux/convert-xref.scm
@@ -0,0 +1,26 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+;; Translate cross-references in a translated .texi manual.
+
+(use-modules (guix build po)
+ (ice-9 match))
+
+(match (command-line)
+ ((program texi pofile)
+ (translate-cross-references texi pofile)))
diff --git a/doc/local.mk b/doc/local.mk
index 8340b75a87..88f7e67155 100644
--- a/doc/local.mk
+++ b/doc/local.mk
@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete
PO4A_PARAMS += -f texinfo # texinfo format
# When a change to guix.texi occurs, it is not translated immediately.
-# Because @pxref and @xref commands are reference to a section by name, they
+# Because @pxref and @xref commands are references to sections by name, they
# should be translated. If a modification adds a reference to a section, this
# reference is not translated, which means it references a section that does not
# exist.
-# This command loops through the translated files looking for references. For
-# each of these references, it tries to find the translation and replaces the
-# reference name, even in untranslated strings.
-# The last sed is a multiline sed because some references span multiple lines.
define xref_command
-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \
- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \
- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \
- if [ -n "$$e" ]; then \
- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\
- ((line++)) ;\
- if [ "$$line" != "1" ]; then \
- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\
- if [ "$$translation" != "" ]; then \
- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\
- fi ;\
- fi ;\
- fi ;\
-done
+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \
+ "$(top_srcdir)/build-aux/convert-xref.scm" \
+ $@.tmp $<
endef
-$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi
+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po
+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po
+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
diff --git a/guix/build/po.scm b/guix/build/po.scm
index eb9690ad1a..7f88164cd8 100644
--- a/guix/build/po.scm
+++ b/guix/build/po.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019 Julien Lepiller <julien@lepiller.eu>
+;;; Copyright © 2019, 2021 Julien Lepiller <julien@lepiller.eu>
;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
@@ -20,17 +20,23 @@
(define-module (guix build po)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
+ #:use-module (ice-9 regex)
#:use-module (ice-9 textual-ports)
- #:export (read-po-file))
+ #:use-module (ice-9 vlist)
+ #:use-module (srfi srfi-1)
+ #:export (read-po-file
+ translate-cross-references))
;; A small parser for po files
-(define-peg-pattern po-file body (* (or comment entry whitespace)))
+(define-peg-pattern po-file body (* (or entry whitespace)))
(define-peg-pattern whitespace body (or " " "\t" "\n"))
(define-peg-pattern comment-chr body (range #\space #\頋))
(define-peg-pattern comment none (and "#" (* comment-chr) "\n"))
+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n")))
(define-peg-pattern entry all
- (and (ignore (* whitespace)) (ignore "msgid ") msgid
- (ignore (* whitespace)) (ignore "msgstr ") msgstr))
+ (and (* (or flags comment (ignore (* whitespace))))
+ (ignore "msgid ") msgid (ignore (* whitespace))
+ (ignore "msgstr ") msgstr))
(define-peg-pattern escape body (or "\\\\" "\\\"" "\\n"))
(define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"")
"\\n" (and (ignore "\\") "\\")
@@ -53,7 +59,24 @@
(append (list "\n" prefix) result)))))))
(define (parse-tree->assoc parse-tree)
- "Converts a po PARSE-TREE to an association list."
+ "Converts a po PARSE-TREE to an association list, where the key is the msgid
+and the value is the msgstr. The result only contains non fuzzy strings."
+ (define (comments->flags comments)
+ (match comments
+ (('flags flags)
+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))
+ (string-split flags #\,)))
+ ((? list? comments)
+ (fold
+ (lambda (comment res)
+ (match comment
+ ((? string? _) res)
+ (flags
+ (append (comments->flags flags)
+ res))))
+ '()
+ comments))))
+
(match parse-tree
(() '())
((entry . parse-tree)
@@ -66,10 +89,22 @@
;; empty msgstr
(('entry ('msgid msgid) 'msgstr)
(parse-tree->assoc parse-tree))
+ (('entry _ ('msgid msgid) 'msgstr)
+ (parse-tree->assoc parse-tree))
(('entry ('msgid msgid) ('msgstr msgstr))
(acons (interpret-newline-escape msgid)
(interpret-newline-escape msgstr)
- (parse-tree->assoc parse-tree)))))))
+ (parse-tree->assoc parse-tree)))
+ (('entry ('msgid msgid) ('msgstr msgstr))
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree)))
+ (('entry comments ('msgid msgid) ('msgstr msgstr))
+ (if (member 'fuzzy (comments->flags comments))
+ (parse-tree->assoc parse-tree)
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree))))))))
(define (read-po-file port)
"Read a .po file from PORT and return an alist of msgid and msgstr."
@@ -77,3 +112,71 @@
po-file
(get-string-all port)))))
(parse-tree->assoc tree)))
+
+(define (canonicalize-whitespace str)
+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."
+ (string-map (lambda (chr)
+ (if (char-set-contains? char-set:whitespace chr)
+ #\space
+ chr))
+ str))
+
+(define xref-regexp
+ ;; Texinfo cross-reference regexp.
+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))
+
+(define (translate-cross-references texi pofile)
+ "Translate the cross-references that appear in @var{texi}, the initial
+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."
+ (define translations
+ (call-with-input-file pofile read-po-file))
+
+ (define content
+ (call-with-input-file texi get-string-all))
+
+ (define matches
+ (list-matches xref-regexp content))
+
+ (define translation-map
+ (fold (match-lambda*
+ (((msgid . str) result)
+ (vhash-cons msgid str result)))
+ vlist-null
+ translations))
+
+ (define translated
+ ;; Iterate over MATCHES and replace cross-references with their
+ ;; translation found in TRANSLATION-MAP. (We can't use
+ ;; 'substitute*' because matches can span multiple lines.)
+ (let loop ((matches matches)
+ (offset 0)
+ (result '()))
+ (match matches
+ (()
+ (string-concatenate-reverse
+ (cons (string-drop content offset) result)))
+ ((head . tail)
+ (let ((prefix (match:substring head 1))
+ (ref (canonicalize-whitespace (match:substring head 2))))
+ (define translated
+ (string-append "@" (or prefix "")
+ "ref{"
+ (match (vhash-assoc ref translation-map)
+ (#f ref)
+ ((_ . str) str))))
+
+ (loop tail
+ (match:end head)
+ (append (list translated
+ (string-take
+ (string-drop content offset)
+ (- (match:start head) offset)))
+ result)))))))
+
+ (format (current-error-port)
+ "translated ~a cross-references in '~a'~%"
+ (length matches) texi)
+
+ (call-with-output-file texi
+ (lambda (port)
+ (display translated port))))
diff --git a/guix/self.scm b/guix/self.scm
index 61ff423086..fc581becd4 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -316,81 +317,23 @@ the result to OUTPUT."
chr))
str))
- (define xref-regexp
- ;; Texinfo cross-reference regexp.
- (make-regexp "@(px|x)?ref\\{([^,}]+)"))
-
- (define (translate-cross-references texi translations)
- ;; Translate the cross-references that appear in TEXI, a Texinfo
- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.
- (define content
- (call-with-input-file texi get-string-all))
-
- (define matches
- (list-matches xref-regexp content))
-
- (define translation-map
- (fold (match-lambda*
- (((msgid . str) result)
- (vhash-cons msgid str result)))
- vlist-null
- translations))
-
- (define translated
- ;; Iterate over MATCHES and replace cross-references with their
- ;; translation found in TRANSLATION-MAP. (We can't use
- ;; 'substitute*' because matches can span multiple lines.)
- (let loop ((matches matches)
- (offset 0)
- (result '()))
- (match matches
- (()
- (string-concatenate-reverse
- (cons (string-drop content offset) result)))
- ((head . tail)
- (let ((prefix (match:substring head 1))
- (ref (canonicalize-whitespace (match:substring head 2))))
- (define translated
- (string-append "@" (or prefix "")
- "ref{"
- (match (vhash-assoc ref translation-map)
- (#f ref)
- ((_ . str) str))))
-
- (loop tail
- (match:end head)
- (append (list translated
- (string-take
- (string-drop content offset)
- (- (match:start head) offset)))
- result)))))))
-
- (format (current-error-port)
- "translated ~a cross-references in '~a'~%"
- (length matches) texi)
- (call-with-output-file texi
- (lambda (port)
- (display translated port))))
-
(define* (translate-texi prefix po lang
#:key (extras '()))
"Translate the manual for one language LANG using the PO file.
PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is
a list of extra files, such as '(\"contributing\")."
- (let ((translations (call-with-input-file po read-po-file)))
- (for-each (lambda (file)
- (translate-tmp-texi po (string-append file ".texi")
- (string-append file "." lang
- ".texi.tmp")))
- (cons prefix extras))
+ (for-each (lambda (file)
+ (translate-tmp-texi po (string-append file ".texi")
+ (string-append file "." lang
+ ".texi.tmp")))
+ (cons prefix extras))
- (for-each (lambda (file)
- (let* ((texi (string-append file "." lang ".texi"))
- (tmp (string-append texi ".tmp")))
- (copy-file tmp texi)
- (translate-cross-references texi
- translations)))
- (cons prefix extras))))
+ (for-each (lambda (file)
+ (let* ((texi (string-append file "." lang ".texi"))
+ (tmp (string-append texi ".tmp")))
+ (copy-file tmp texi)
+ (translate-cross-references texi po)))
+ (cons prefix extras)))
(define (available-translations directory domain)
;; Return the list of available translations under DIRECTORY for
--
2.33.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller
@ 2021-10-11 17:17 ` pelzflorian (Florian Pelz)
2021-10-11 22:06 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-11 17:17 UTC (permalink / raw)
To: Julien Lepiller; +Cc: 51122
Thanks Julien!
On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote:
> Thanks for the quick review! I've updated the commit message and
> updated the copyright headers in guix/build/po.scm and guix/self.scm.
In guix/self.scm you only removed code. I don’t think a copyright
header is needed there.
> I had to make the texi files depend on make-go because otherwise, I get
> messages like "guix/build/po.scm is newer than compiled
> .../guix/build/po.go" and it takes forever to build that file, once for
> each texi. For some reason, it's almost instantaneous when building in
> make-go.
make-core-go instead of make-go is enough in my testing, but something
more fine-grained would save time. Also POXREF is run twice for each
guix.??.texi file. Maybe I did something wrong, maybe it is your
patch. Not sure. Testing takes long anyway.
Regards,
Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-11 17:17 ` pelzflorian (Florian Pelz)
@ 2021-10-11 22:06 ` Julien Lepiller
2021-10-12 15:56 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-11 22:06 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 51122
Le Mon, 11 Oct 2021 19:17:28 +0200,
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
> Thanks Julien!
>
> On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote:
> > Thanks for the quick review! I've updated the commit message and
> > updated the copyright headers in guix/build/po.scm and
> > guix/self.scm.
>
> In guix/self.scm you only removed code. I don’t think a copyright
> header is needed there.
Alright, I removed the copyright line on my side
>
> > I had to make the texi files depend on make-go because otherwise, I
> > get messages like "guix/build/po.scm is newer than compiled
> > .../guix/build/po.go" and it takes forever to build that file, once
> > for each texi. For some reason, it's almost instantaneous when
> > building in make-go.
>
> make-core-go instead of make-go is enough in my testing, but something
> more fine-grained would save time. Also POXREF is run twice for each
> guix.??.texi file. Maybe I did something wrong, maybe it is your
> patch. Not sure. Testing takes long anyway.
It only runs it once here, but there are three files to translate:
guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi.
It looks like depending on make-go or make-core-go means we rebuild the
texi files everytime we run "make", so that's not great :/. I'll see if
I can isolate guix/build/po.scm (any ideas?).
> Regards,
> Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-11 22:06 ` Julien Lepiller
@ 2021-10-12 15:56 ` pelzflorian (Florian Pelz)
2021-10-12 18:09 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 12+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-12 15:56 UTC (permalink / raw)
To: Julien Lepiller; +Cc: 51122
On Tue, Oct 12, 2021 at 12:06:57AM +0200, Julien Lepiller wrote:
> Le Mon, 11 Oct 2021 19:17:28 +0200,
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
> > Also POXREF is run twice for each
> > guix.??.texi file. Maybe I did something wrong, maybe it is your
> > patch. Not sure. Testing takes long anyway.
> It only runs it once here, but there are three files to translate:
> guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi.
At least when depending on make-core-go instead of make-go, it does
POXREF each file twice, in between printing “Updating
doc/version.texi”. Anyway, this is more important:
> It looks like depending on make-go or make-core-go means we rebuild the
> texi files everytime we run "make", so that's not great :/.
We definitely want to compile guix/build/po.scm, I think, because it
should be run each time a guix-manual.LL.po or guix-cookbook.LL.po
file changes.
GNU Make manual says
<https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets>:
> A phony target should not be a prerequisite of a real target file; if
> it is, its recipe will be run every time make goes to update that
> file.
But Makefile.am defines guile-compilation-rule to create a Phony target.
This is bad for us because likely there should not be duplicate code
to compile guix/build/po.scm and to compile the rest.
So I presume part of guile-compilation-rule must be split off?
Regards,
Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-12 15:56 ` pelzflorian (Florian Pelz)
@ 2021-10-12 18:09 ` pelzflorian (Florian Pelz)
2021-10-12 19:25 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-12 18:09 UTC (permalink / raw)
To: Julien Lepiller; +Cc: 51122
On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote:
> We definitely want to compile guix/build/po.scm, I think, because it
> should be run each time a guix-manual.LL.po or guix-cookbook.LL.po
> file changes.
Then again, is calling guix without compiling necessarily slower than
the sed script?
Regards,
Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-12 18:09 ` pelzflorian (Florian Pelz)
@ 2021-10-12 19:25 ` Julien Lepiller
2021-10-16 2:12 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-12 19:25 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 51122
Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
>On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote:
>> We definitely want to compile guix/build/po.scm, I think, because it
>> should be run each time a guix-manual.LL.po or guix-cookbook.LL.po
>> file changes.
>
>Then again, is calling guix without compiling necessarily slower than
>the sed script?
>
>Regards,
>Florian
Yes, each time I see the "newer than compiled" message, it takes several minutes.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-12 19:25 ` Julien Lepiller
@ 2021-10-16 2:12 ` Julien Lepiller
2021-10-16 18:45 ` pelzflorian (Florian Pelz)
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2021-10-16 2:12 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 51122
[-- Attachment #1: Type: text/plain, Size: 946 bytes --]
Le Tue, 12 Oct 2021 15:25:37 -0400,
Julien Lepiller <julien@lepiller.eu> a écrit :
> Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)"
> <pelzflorian@pelzflorian.de> a écrit :
> >On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz)
> >wrote:
> >> We definitely want to compile guix/build/po.scm, I think, because
> >> it should be run each time a guix-manual.LL.po or
> >> guix-cookbook.LL.po file changes.
> >
> >Then again, is calling guix without compiling necessarily slower than
> >the sed script?
> >
> >Regards,
> >Florian
>
> Yes, each time I see the "newer than compiled" message, it takes
> several minutes.
>
>
>
I managed to change the dependency to only guix/po.go, and to make it
not phony, so the translated manuals don't get rebuilt on every make
invocation anymore. I checked again that "make" from a clean checkout
and "make as-derivation" still work.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-maint-Factorize-po-xref-translation.patch --]
[-- Type: text/x-patch, Size: 18092 bytes --]
From 96d7096ac46f0fa8a97db76fefdf751a0584975b Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Sun, 10 Oct 2021 22:07:51 +0200
Subject: [PATCH] maint: Factorize po xref translation.
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.
* build-aux/convert-xref.scm: New file.
* doc/local.mk (xref_command): Use it.
* guix/self.scm (translate-cross-references): Move it...
* guix/build/po.scm: Parse comments and flags separately to find fuzzy
flags.
(translate-cross-references): ...here.
(parse-tree->assoc): Ignore fuzzy entries.
---
Makefile.am | 12 +++-
build-aux/convert-xref.scm | 26 +++++++++
doc/local.mk | 29 +++-------
guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++--
guix/self.scm | 81 ++++----------------------
5 files changed, 164 insertions(+), 99 deletions(-)
create mode 100644 build-aux/convert-xref.scm
diff --git a/Makefile.am b/Makefile.am
index 635147efc1..41ec19eb89 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -640,6 +640,7 @@ EXTRA_DIST += \
build-aux/check-final-inputs-self-contained.scm \
build-aux/check-channel-news.scm \
build-aux/compile-as-derivation.scm \
+ build-aux/convert-xref.scm \
build-aux/generate-authors.scm \
build-aux/test-driver.scm \
build-aux/update-guix-package.scm \
@@ -699,8 +700,6 @@ $(1): $(2)
--completed $(3) \
$$(filter %.scm,$$^)
-.PHONY: $(1)
-
endef
# Split compilation in several steps, each of which building a subset of
@@ -712,22 +711,31 @@ MODULES_CORE = guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULE
MODULES_PACKAGES = $(filter gnu/packages/%,$(MODULES))
MODULES_SYSTEM = gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
MODULES_CLI = $(filter guix/scripts/%,$(MODULES))
+MODULES_PO = guix/build/po.scm
$(eval $(call guile-compilation-rule,make-core-go, \
$(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \
0))
+.PHONY: make-core-go
$(eval $(call guile-compilation-rule,make-packages-go, \
$(MODULES_PACKAGES) make-core-go, \
$(words $(MODULES_CORE))))
+.PHONY: make-packages-go
$(eval $(call guile-compilation-rule,make-system-go, \
$(MODULES_SYSTEM) make-packages-go make-core-go, \
$(words $(MODULES_CORE) $(MODULES_PACKAGES))))
+.PHONY: make-system-go
$(eval $(call guile-compilation-rule,make-cli-go, \
$(MODULES_CLI) make-system-go make-packages-go make-core-go, \
$(words $(MODULES_CORE) $(MODULES_PACKAGES) $(MODULES_SYSTEM))))
+.PHONY: make-cli-go
+
+$(eval $(call guile-compilation-rule,guix/build/po.go, \
+ $(MODULES_PO), \
+ 0))
SUFFIXES = .go
diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scm
new file mode 100644
index 0000000000..47c8828857
--- /dev/null
+++ b/build-aux/convert-xref.scm
@@ -0,0 +1,26 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+;; Translate cross-references in a translated .texi manual.
+
+(use-modules (guix build po)
+ (ice-9 match))
+
+(match (command-line)
+ ((program texi pofile)
+ (translate-cross-references texi pofile)))
diff --git a/doc/local.mk b/doc/local.mk
index 8340b75a87..fff11f8183 100644
--- a/doc/local.mk
+++ b/doc/local.mk
@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete
PO4A_PARAMS += -f texinfo # texinfo format
# When a change to guix.texi occurs, it is not translated immediately.
-# Because @pxref and @xref commands are reference to a section by name, they
+# Because @pxref and @xref commands are references to sections by name, they
# should be translated. If a modification adds a reference to a section, this
# reference is not translated, which means it references a section that does not
# exist.
-# This command loops through the translated files looking for references. For
-# each of these references, it tries to find the translation and replaces the
-# reference name, even in untranslated strings.
-# The last sed is a multiline sed because some references span multiple lines.
define xref_command
-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \
- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \
- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \
- if [ -n "$$e" ]; then \
- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\
- ((line++)) ;\
- if [ "$$line" != "1" ]; then \
- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\
- if [ "$$translation" != "" ]; then \
- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\
- fi ;\
- fi ;\
- fi ;\
-done
+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \
+ "$(top_srcdir)/build-aux/convert-xref.scm" \
+ $@.tmp $<
endef
-$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi
+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi guix/build/po.go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po
+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po guix/build/po.go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp"
-sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
-$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po
+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po guix/build/po.go
-$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp"
-$(AM_V_POXREF)$(xref_command)
-mv "$@.tmp" "$@"
diff --git a/guix/build/po.scm b/guix/build/po.scm
index eb9690ad1a..a167252244 100644
--- a/guix/build/po.scm
+++ b/guix/build/po.scm
@@ -20,17 +20,23 @@
(define-module (guix build po)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
+ #:use-module (ice-9 regex)
#:use-module (ice-9 textual-ports)
- #:export (read-po-file))
+ #:use-module (ice-9 vlist)
+ #:use-module (srfi srfi-1)
+ #:export (read-po-file
+ translate-cross-references))
;; A small parser for po files
-(define-peg-pattern po-file body (* (or comment entry whitespace)))
+(define-peg-pattern po-file body (* (or entry whitespace)))
(define-peg-pattern whitespace body (or " " "\t" "\n"))
(define-peg-pattern comment-chr body (range #\space #\頋))
(define-peg-pattern comment none (and "#" (* comment-chr) "\n"))
+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n")))
(define-peg-pattern entry all
- (and (ignore (* whitespace)) (ignore "msgid ") msgid
- (ignore (* whitespace)) (ignore "msgstr ") msgstr))
+ (and (* (or flags comment (ignore (* whitespace))))
+ (ignore "msgid ") msgid (ignore (* whitespace))
+ (ignore "msgstr ") msgstr))
(define-peg-pattern escape body (or "\\\\" "\\\"" "\\n"))
(define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"")
"\\n" (and (ignore "\\") "\\")
@@ -53,7 +59,24 @@
(append (list "\n" prefix) result)))))))
(define (parse-tree->assoc parse-tree)
- "Converts a po PARSE-TREE to an association list."
+ "Converts a po PARSE-TREE to an association list, where the key is the msgid
+and the value is the msgstr. The result only contains non fuzzy strings."
+ (define (comments->flags comments)
+ (match comments
+ (('flags flags)
+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))
+ (string-split flags #\,)))
+ ((? list? comments)
+ (fold
+ (lambda (comment res)
+ (match comment
+ ((? string? _) res)
+ (flags
+ (append (comments->flags flags)
+ res))))
+ '()
+ comments))))
+
(match parse-tree
(() '())
((entry . parse-tree)
@@ -66,10 +89,22 @@
;; empty msgstr
(('entry ('msgid msgid) 'msgstr)
(parse-tree->assoc parse-tree))
+ (('entry _ ('msgid msgid) 'msgstr)
+ (parse-tree->assoc parse-tree))
(('entry ('msgid msgid) ('msgstr msgstr))
(acons (interpret-newline-escape msgid)
(interpret-newline-escape msgstr)
- (parse-tree->assoc parse-tree)))))))
+ (parse-tree->assoc parse-tree)))
+ (('entry ('msgid msgid) ('msgstr msgstr))
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree)))
+ (('entry comments ('msgid msgid) ('msgstr msgstr))
+ (if (member 'fuzzy (comments->flags comments))
+ (parse-tree->assoc parse-tree)
+ (acons (interpret-newline-escape msgid)
+ (interpret-newline-escape msgstr)
+ (parse-tree->assoc parse-tree))))))))
(define (read-po-file port)
"Read a .po file from PORT and return an alist of msgid and msgstr."
@@ -77,3 +112,71 @@
po-file
(get-string-all port)))))
(parse-tree->assoc tree)))
+
+(define (canonicalize-whitespace str)
+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."
+ (string-map (lambda (chr)
+ (if (char-set-contains? char-set:whitespace chr)
+ #\space
+ chr))
+ str))
+
+(define xref-regexp
+ ;; Texinfo cross-reference regexp.
+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))
+
+(define (translate-cross-references texi pofile)
+ "Translate the cross-references that appear in @var{texi}, the initial
+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."
+ (define translations
+ (call-with-input-file pofile read-po-file))
+
+ (define content
+ (call-with-input-file texi get-string-all))
+
+ (define matches
+ (list-matches xref-regexp content))
+
+ (define translation-map
+ (fold (match-lambda*
+ (((msgid . str) result)
+ (vhash-cons msgid str result)))
+ vlist-null
+ translations))
+
+ (define translated
+ ;; Iterate over MATCHES and replace cross-references with their
+ ;; translation found in TRANSLATION-MAP. (We can't use
+ ;; 'substitute*' because matches can span multiple lines.)
+ (let loop ((matches matches)
+ (offset 0)
+ (result '()))
+ (match matches
+ (()
+ (string-concatenate-reverse
+ (cons (string-drop content offset) result)))
+ ((head . tail)
+ (let ((prefix (match:substring head 1))
+ (ref (canonicalize-whitespace (match:substring head 2))))
+ (define translated
+ (string-append "@" (or prefix "")
+ "ref{"
+ (match (vhash-assoc ref translation-map)
+ (#f ref)
+ ((_ . str) str))))
+
+ (loop tail
+ (match:end head)
+ (append (list translated
+ (string-take
+ (string-drop content offset)
+ (- (match:start head) offset)))
+ result)))))))
+
+ (format (current-error-port)
+ "translated ~a cross-references in '~a'~%"
+ (length matches) texi)
+
+ (call-with-output-file texi
+ (lambda (port)
+ (display translated port))))
diff --git a/guix/self.scm b/guix/self.scm
index a0d448742a..8ada1d85a4 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -316,81 +317,23 @@ the result to OUTPUT."
chr))
str))
- (define xref-regexp
- ;; Texinfo cross-reference regexp.
- (make-regexp "@(px|x)?ref\\{([^,}]+)"))
-
- (define (translate-cross-references texi translations)
- ;; Translate the cross-references that appear in TEXI, a Texinfo
- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.
- (define content
- (call-with-input-file texi get-string-all))
-
- (define matches
- (list-matches xref-regexp content))
-
- (define translation-map
- (fold (match-lambda*
- (((msgid . str) result)
- (vhash-cons msgid str result)))
- vlist-null
- translations))
-
- (define translated
- ;; Iterate over MATCHES and replace cross-references with their
- ;; translation found in TRANSLATION-MAP. (We can't use
- ;; 'substitute*' because matches can span multiple lines.)
- (let loop ((matches matches)
- (offset 0)
- (result '()))
- (match matches
- (()
- (string-concatenate-reverse
- (cons (string-drop content offset) result)))
- ((head . tail)
- (let ((prefix (match:substring head 1))
- (ref (canonicalize-whitespace (match:substring head 2))))
- (define translated
- (string-append "@" (or prefix "")
- "ref{"
- (match (vhash-assoc ref translation-map)
- (#f ref)
- ((_ . str) str))))
-
- (loop tail
- (match:end head)
- (append (list translated
- (string-take
- (string-drop content offset)
- (- (match:start head) offset)))
- result)))))))
-
- (format (current-error-port)
- "translated ~a cross-references in '~a'~%"
- (length matches) texi)
- (call-with-output-file texi
- (lambda (port)
- (display translated port))))
-
(define* (translate-texi prefix po lang
#:key (extras '()))
"Translate the manual for one language LANG using the PO file.
PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is
a list of extra files, such as '(\"contributing\")."
- (let ((translations (call-with-input-file po read-po-file)))
- (for-each (lambda (file)
- (translate-tmp-texi po (string-append file ".texi")
- (string-append file "." lang
- ".texi.tmp")))
- (cons prefix extras))
+ (for-each (lambda (file)
+ (translate-tmp-texi po (string-append file ".texi")
+ (string-append file "." lang
+ ".texi.tmp")))
+ (cons prefix extras))
- (for-each (lambda (file)
- (let* ((texi (string-append file "." lang ".texi"))
- (tmp (string-append texi ".tmp")))
- (copy-file tmp texi)
- (translate-cross-references texi
- translations)))
- (cons prefix extras))))
+ (for-each (lambda (file)
+ (let* ((texi (string-append file "." lang ".texi"))
+ (tmp (string-append texi ".tmp")))
+ (copy-file tmp texi)
+ (translate-cross-references texi po)))
+ (cons prefix extras)))
(define (available-translations directory domain)
;; Return the list of available translations under DIRECTORY for
--
2.31.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bug#51122] [PATCH v2] maint: Factorize po xref translation.
2021-10-16 2:12 ` Julien Lepiller
@ 2021-10-16 18:45 ` pelzflorian (Florian Pelz)
2021-10-17 20:12 ` bug#51122: " Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: pelzflorian (Florian Pelz) @ 2021-10-16 18:45 UTC (permalink / raw)
To: Julien Lepiller; +Cc: 51122
On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote:
> I managed to change the dependency to only guix/po.go, and to make it
> not phony, so the translated manuals don't get rebuilt on every make
> invocation anymore. I checked again that "make" from a clean checkout
> and "make as-derivation" still work.
Good, thank you! I had thought it would be more complicated than
moving out the PHONY directive, but all works OK:
Without patch, the fuzzy references cause make to fail for Weblate’s
version of the Spanish manual. With patch, make runs through fine as
you write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make
update-guix-package && git commit and guix pull and still works, guix
build guix failed in check but for unrelated reasons, so the patch
LGTM. Except:
As we discussed before, you only remove code from guix/self.scm, so
you should not add a copyright line.
Regards,
Florian
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#51122: [PATCH v2] maint: Factorize po xref translation.
2021-10-16 18:45 ` pelzflorian (Florian Pelz)
@ 2021-10-17 20:12 ` Julien Lepiller
0 siblings, 0 replies; 12+ messages in thread
From: Julien Lepiller @ 2021-10-17 20:12 UTC (permalink / raw)
To: pelzflorian (Florian Pelz); +Cc: 51122-done
Le Sat, 16 Oct 2021 20:45:02 +0200,
"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
> On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote:
> > I managed to change the dependency to only guix/po.go, and to make
> > it not phony, so the translated manuals don't get rebuilt on every
> > make invocation anymore. I checked again that "make" from a clean
> > checkout and "make as-derivation" still work.
>
> Good, thank you! I had thought it would be more complicated than
> moving out the PHONY directive, but all works OK:
>
> Without patch, the fuzzy references cause make to fail for Weblate’s
> version of the Spanish manual. With patch, make runs through fine as
> you write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make
> update-guix-package && git commit and guix pull and still works, guix
> build guix failed in check but for unrelated reasons, so the patch
> LGTM. Except:
>
> As we discussed before, you only remove code from guix/self.scm, so
> you should not add a copyright line.
>
> Regards,
> Florian
Thanks, pushed as 0623138ffa5b066afc25547ffdeb97753cb0ee9a, and
followed by an nls update.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-10-17 20:13 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-10 21:32 [bug#51122] [PATCH] maint: Factorize po xref translation Julien Lepiller
2021-10-10 21:37 ` Julien Lepiller
2021-10-11 8:45 ` pelzflorian (Florian Pelz)
2021-10-11 12:46 ` [bug#51122] [PATCH v2] " Julien Lepiller
2021-10-11 17:17 ` pelzflorian (Florian Pelz)
2021-10-11 22:06 ` Julien Lepiller
2021-10-12 15:56 ` pelzflorian (Florian Pelz)
2021-10-12 18:09 ` pelzflorian (Florian Pelz)
2021-10-12 19:25 ` Julien Lepiller
2021-10-16 2:12 ` Julien Lepiller
2021-10-16 18:45 ` pelzflorian (Florian Pelz)
2021-10-17 20:12 ` bug#51122: " Julien Lepiller
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).