[bug#50359] [PATCH] import: Add 'generic-git' updater.

[Xinglu Chen <public@yoctocell.xyz>]

[-- Attachment #1: Type: text/plain, Size: 9944 bytes --]

On Sun, Sep 05 2021, Sarah Morgensen wrote:

> Hi,
>
> Xinglu Chen <public@yoctocell.xyz> writes:
>
>>> There are about 50-60 packages like this.
>>>
>>> I'm not sure how much effort should be spent including them, and for
>>> some of them I'm not sure what our ideal behavior *is*.  Even if we
>>> could reliably detect them, should "alpha" or "dev" packages be returned
>>> by the updater?
>>
>> I don’t think we usually include alpha or rc releases, so updater
>> probably shouldn’t return them either.  Not sure how we would try to
>> detect alpha/beta/rc releases, though, besides running something like
>>
>>   (string-match? "(alpha|beta|rc|dev)" TAG)
>>
>> On each tag.
>
> That heuristic is pretty good.  (It might miss a few, but I'd rather
> accidentally include some alpha/beta/rc releases than risk excluding
> real ones.)
>
> We could then safely sort tags with just the prefix removed -- this takes care
> of "1.1f" coming after "1.1", and so on.
>
> Actually, it looks like there's only a few packages with a suffix;
> instead, we can probably just only use a suffix if they provide
> 'tag-suffix.  This is all of them (and the "dev" ones probably shouldn't
> be suffixes, just part of the version):

Yeah, I think that sounds like a reasonable thing to do.

> (commit (string-append "v" version "-stable"))))
> (commit (string-append version "-stable"))))
> (commit (string-append version "-Leia"))))
> (commit (string-append "haddock-" version "-release"))))
> (commit (string-append "v" version "-8.13"))))
> (commit (string-append "v" version "-oss"))))
> (commit (string-append "v" version "-stable"))))
> (commit (string-append "ddskk-" version "_" code-name))))
> (commit (string-append version "-freebsdport"))))
> (commit (string-append version "-dev"))))
> (commit (string-append version "-release-20210531143054"))))
> (commit (string-append version "-release-20210412001032"))))
> (commit (string-append "v" version "-debian"))))
> (commit (string-append version "dev"))))
> (commit (string-append version "_Linux"))
> (commit (string-append version "R"))))
> (commit (string-append "jdk-" version "-ga"))))
> (commit (string-append "jdk-" version "-ga"))))
> (commit (string-append "jdk-" version "-ga"))))
> (commit (string-append "jdk-" version "-ga"))))
> (commit (string-append version "-opt"))))
> (commit (string-append "1.1-" version "-RELEASE"))))
>
> Additionally, these are all the weird version strings I could find that
> are actually used as the commit:
> 1.2.2.rc2
> 12-068oasis4
> 1.2-2
> 0.F-2
> 0.2.0-alpha
> 5.1.0-b2
> 1.5-11
> 1.9.14-20210407
> 0.9.3b
> 2.8-fix-2
> 2020-05-19
> 10-11.0.0
> 1.0.12-2
> 0.9.3+16.04.20160218-0ubuntu1
> 2.12.c
> 1.1+11
> 5.1+4.06.0
> 4.2-411
> 1.4.0rc1-450-g2725ef99d
> 2.0.0-alpha14
> 60.2.3-2
> 1.21c
> 1.0.0rc4
> 2.1.0b1
> 1.02r6
> 0.32-14-gcdfe14e
> 3.0.0a3
> 2.00a2.3
> 1.0beta.18
> 2.1b
> 2.7.8a
> 2.7.3a
> 1.1.alpha19
> 1.0.2-rc4
> 0.5.3+git20200502
> 1.0.3-rc3
> 4.0.0.dev8
> 3.0.0beta1-24-g024cc9fa2
> 0.16-2-ge145396
> 2.0b6
> 2.0M10
> 1.0.7+0
> 1.16.0+5
> 0.4.0+1
> 2.2.10+0
> 4.3.1+2
> 2.13.93+0
> 2.10.4+0
> 1.0.5+5
> 0.21.0+0
> 3.3.4+0
> 2.68.1+0
> 0.10.1+1
> 6.9.10-12+3
> 2.0.1+2
> 3.100.0+1
> 0.14.0+2
> 0.1.6+2
> 3.3.0+0
> 1.8.7+0
> 1.3.0+2
> 1.42.0+0
> 1.16.1+0
> 2.35.0+0
> 1.6.37+5
> 4.1.0+1
> 2.36.0+0
> 1.3.6+4
> 2.10.1+0
> 2.26.0+0
> 1.3.4+0
> 1.1.1+2
> 1.3.1+1
> 8.44.0+0
> 0.40.1+0
> 5.15.2+0
> 1.17.0+3
> 1.18.0+3
> 2020.7.14+0
> 3.0.0+1
> 0.9.1+3
> 2.9.12+0
> 0.1.0+2
> 1.6.9+2
> 1.0.9+3
> 1.13.0+2
> 1.2.0+3
> 1.1.3+3
> 1.3.4+2
> 5.0.3+3
> 1.7.10+3
> 1.1.4+3
> 1.1.0+3
> 1.5.2+3
> 0.9.10+3
> 0.4.0+1
> 0.4.0+1
> 0.4.0+1
> 0.3.9+1
> 0.4.1+1
> 1.4.2+3
> 2.27.0+3
> 1.4.0+2
> 1.1.34+0
> 1.2.12+1
> 1.5.0+0
> 2021-06-07
> 1.2.2-5-g20dc8ed
> 2.1-20201229
> 3.0-rc1
> 1.32.0-0
> 20200701.154658.b0d6223
> 3.9-0
> R63-10032.B
> 0.58.2.a
> 0.0.0+git20200527
> 3.028R
> 3.001R
> 2.57b
> 1.0.0-20201130134442-10cb98267c6c
> 0.0.0-20161123171359-e6a2ba005892
> 0.0.0-20210615171337-6886f2dfbf5b
> 2020-11-10
> 5.2.0-alpha
> 2021-01-01
> 2021-02-28
> 2020-11-01
> 4.4-git.1
> 1.217-2
> 3.3.06-1
> 0.2.0-alpha-199-g3e7a475
> 1.0.0-beta.0
> 1.20190621-4
> 1.9.0-147-g61edec1ef
> 0.0.9.4f
>
> Yep, it looks like the above would work for the majority of these.
> That's probably Good Enough^tm.

Any trick you used to find all of there weird version numbers?  :-)

>>>> +  (define no-delims
>>>> +    (if (string-match delim-regexp no-suffix)
>>>> +        (string-split* no-suffix delim*)
>>>> +        (git-tag-error 'tag-version-delimiter)))
>>>
>>> This throws an error if the version doesn't have any delimiter.
>>
>> Setting the ‘tag-version-delimiter’ prefix to an empty string would
>> solve this, right?  Or, maybe we should just get rid of the delimiter
>> thing since only a few packages use a different delimiter.
>
> IMO, just get rid of the delimiter.  If we wanted to be *that* flexible,
> we could make it so they provide a tag->version proc instead of (prefix,
> suffix, delimiter).

a ‘tag->version’ procedure would probably make things a bit too
complicated for the people writing package definitions.  For example,
having a delimiter would make it easy to match a tag like
“2021-01-01-release”

Delimiter is “.” (sorry if this hurts your eyes ;-))

scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1)
$28 = "2021"

Delimiter is “-”

scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1)
$29 = "2021-01-01-release"

And then, setting the suffix to “-release” would match just the version
part.


On Sun, Sep 05 2021, Sarah Morgensen wrote (again):

>>> And this lets us just do something like (untested):
>>>
>>> (define* (get-version tag #:key prefix suffix delim)
>>>   (define delim-rx (regexp-quote (or delim ".")))
>>>   (define prefix-rx (or prefix "[^[:digit:]]*"))
>>>   (define suffix-rx (or suffix ".*"))
>>>   (define version-char-rx
>>>    (string-append "[^" delim-rx "[:punct:]]"))
>>>
>>>   (define tag-rx
>>>    (string-append "^" prefix "(" version-char-rx "+("
>>>                   delim-rx version-char-rx ")*)" suffix-rx "$"))
>>
>> This wouldn’t match anything if the version is just a plain number,
>> e.g., 1 or 09.
>
> It does,

Oh, I missed the extra pair of parens, sorry about that.

> but I had many errors in the definition.  Again, apologies.  I
> shouldn't send emails that late, haha.  This method should read:
>
>  --8<---------------cut here---------------start------------->8---
>  (define* (get-version tag #:key prefix suffix delim)
>    (define delim-rx (regexp-quote (or delim ".")))
>    (define prefix-rx (or prefix "[^[:digit:]]*"))
>    (define suffix-rx (or suffix ".*"))
>    (define version-char-rx
>     (string-append "[^" delim-rx "[:punct:]]"))
>
>    (define tag-rx
>     (string-append "^" prefix-rx "(" version-char-rx "+("
>                    delim-rx version-char-rx "+)*)" suffix-rx "$"))
>    (and=> (string-match tag-rx tag)
>           (cut match:substring <> 1)))
> --8<---------------cut here---------------end--------------->8---
>
>>
>> With this, something like “1.4.0rc1-450-g2725ef99d” will result in
>> “1.4.0” being returned, which is incorrect.  Changing (cut
>> match:substring <> 1) to just ‘match:substring’ would solve the issue,
>> but then pre-release tags, which we usually don’t want,  would also get
>> matched.  Not sure what the best option would be in this case.
>>
>
> With the fixed method above:
>
> scheme@(emacs-guix)> (get-version "8")
> $16 = "8"
> scheme@(emacs-guix)> (get-version "1.4.0rc1-450-g2725ef99d")
> $17 = "1.4.0rc1"
>
> But, we still get:
>
> scheme@(emacs-guix)> (get-version "1.4.0-rc1")
> $18 = "1.4.0"
>
> which leads us to what you talked about in your other message.

Hmm, maybe we could check that if (string-append "1.4.0" suffix) is a
suffix on of the tag, or is this too much overhead?  This would only
work if we set the suffix to the empty string by default.

Since (string-match "1.4.0$" "1.4.0-rc1") returns #f, this tag would get
filtered out.  But then a suffix would have to be specified to match
"1.4.0rc1-450-g2725ef99d".

Not sure what the best option is here.

>>> +(define* (ls-remote-refs url #:key tags?)
>>> +  "Return the list of references advertised at Git repository URL.  If TAGS?
>>> +is true, limit to only refs/tags."
>>> +  (define (ref? ref)
>>> +    ;; Like `git ls-remote --refs', only show actual references.
>>> +    (and (string-prefix? "refs/" ref)
>>> +         (not (string-suffix? "^{}" ref))))
>>> +
>>> +  (define (tag? ref)
>>> +    (string-prefix? "refs/tags/" ref))
>>> +
>>> +  (define (include? ref)
>>> +    (and ref?
>
> This should be:
>         (and (ref? ref)

Ah, problem solved!  :-)

>>> +         (or (not tags?) (tag? ref))))
>>> +
>>> +  (with-libgit2
>>> +   (with-temporary-directory
>>> +    (lambda (cache-directory)
>>> +      (let* ((repository (repository-init cache-directory))
>>> +             ;; Create an in-memory remote so we don't touch disk.
>>> +             (remote (remote-create-anonymous repository url)))
>>> +        (remote-connect remote)
>>> +        (remote-disconnect remote)
>>> +        (repository-close! repository)
>>> +
>>> +        (filter include? (map remote-head-name (remote-ls remote))))))))
>>>
>>
>> For some reason it seems to include refs that do and don’t end with
>> “^{}”
>
> Sorry, another typo I missed.  See above.

No worries, thanks for taking the time to review this!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]