Re: Golang mudules to follow common grouping

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]


Hi,

I've pushed the split III to master.
- https://issues.guix.gnu.org/68605
- https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68605

And it's picked up by CI.
- https://ci.guix.gnu.org/eval/1082575

> Hmm, there seems to be a limit in the degree of parallelizability in
> this process unfortunately. But if there's anything you can think of
> that could help (manually) in this effort, then I'd be happy to help!

I think you may help! The identification of the group is still human
decision making process and I'm not sure it may be automated in any point.
So...
There are golang dedicated modules now and few more coming soon!

Each of the module header contains a short annotation which packages it
expects to have, feel free to improve it to make it even more clear
for others.

- golang-check
- golang-web
- golang-crypto

TBA:
- golang-compression :: Anything related to that subject, see
  python-compression, java-compression, perl-compression.

- golang-build or golang-extension :: Any low level golang add-ons not
included in core distribution see <https://pkg.go.dev/golang.org/x> or
any 0 dependencies high reference modules.

- golang-xyz :: As any other *-xyz module would absorb anything else
  left behind.

Maybe:
- golang-graphics
- golang-maths / golang-science
- ...

> 1. Put a magic comment above each package that you would like to move.
> 2. Run a simple script that makes a note of all of these into a
> to-move-list.
> 3. Then stash the change with the comments you made (in case you need
> to change things)
> 4. Run another script that takes the package list and performs the
> move in one's repository.
> 5. Sort out the use-package declarations manually and run tests.
> 6. When satisfied, stash the change and keep just the use-package
> changes.
> 7. Run a final script that loops through all the packages and commits
> each one in turn.
> 8. Rebase to suit.

We may extend handy script accelerating committing process, see
"etc/committer.scm"

> - I'm not a scheme programmer, but I did use Haskell at university so
>  I'm familiar with thinking in a functional style.
Me too =), but you still can help by just providing some review to
existing code base and available packages in golagn.scm and trying to
identify close group for each of them.

> I'm also imagining some the possibility of having a script that can
> remove redundant #:use-module's in the future, though I don't know if
> we care about a few unneeded modules being included.
The clean up task may be organasied after sort process is completed, having
not required #:use-module does not hurt too much but for keeping modules
tidy and fast to load it definitely beneficial.

Regards,
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Golang mudules to follow common grouping

[Christina O'Donnell]

[-- Attachment #1: Type: text/plain, Size: 2630 bytes --]

Hi Oleg,

> I've pushed the split III to master.

Fantastic work!

> I think you may help! The identification of the group is still human
> decision making process and I'm not sure it may be automated in any point.

I can certainly help with this then. I'll have some free time on Friday and
I can coordinate with you then.

> Each of the module header contains a  short annotation which packages it
>  expects to have, feel free to improve it to make it even more clear
> for  others.
>
> -  golang-check
> -  golang-web
> -  golang-crypto
>
> TBA:
> -  golang-compression :: Anything related to that subject, see
>  python-compression, java-compression, perl-compression.
>
> -  golang-build or golang-extension :: Any low level golang add-ons not
>  included in core distribution see <https://pkg.go.dev/golang.org/x> or
> any 0  dependencies high reference modules.
>
> -  golang-xyz :: As any other *-xyz module would absorb anything else
>   left  behind.

This all sounds sensible to me.

>> 1.  Put a magic comment above each package that you would like to move.
>> 2.  Run a simple script that makes a note of all of these into a
>>  to-move-list.
>> 3.  Then stash the change with the comments you made (in case you need
>> to  change things)
>> 4.  Run another script that takes the package list and performs the
>>  move in one's repository.
>> 5.  Sort out the use-package declarations manually and run tests.
>> 6.  When satisfied, stash the change and keep just the use-package
>>  changes.
>> 7.  Run a final script that loops through all the packages and commits
>>  each one in turn.
>> 8.  Rebase to suit.
>
> We may  extend handy script accelerating committing process, see
>  "etc/committer.scm"

Okay, cool, I'll have a look at it on Friday.

>> -  I'm not a scheme programmer, but I did use Haskell at university so
>>  I'm familiar with thinking in a functional style.
> Me too  =), but you still can help by just providing some review to
>  existing code base and available packages in golagn.scm and trying to
>  identify close group for each of them.
>
>>  I'm also imagining some the possibility of having a script that can
>>  remove redundant #:use-module's in the future, though I don't know if
>> we  care about a few unneeded modules being included.
> The  clean up task may be organasied after sort process is completed, having
> not  required #:use-module does not hurt too much but for keeping modules
> tidy  and fast to load it definitely beneficial.

This makes sense. Not a priority at present but a nice-to

Looking forward to hacking golang.scm to pieces!

Kind regards,
  - Christina


[-- Attachment #2: Type: text/html, Size: 7725 bytes --]

Re: Golang mudules to follow common grouping

[Christina O'Donnell]

Hi again Oleg,

 > > I think you may help! The identification of the group is still human
 > > decision making process and I'm not sure it may be automated in any 
point.
 >
 > I can certainly help with this then. I'll have some free time on 
Friday and
 > I can coordinate with you then.
Ah, I lost track of time..

Okay, so I wrote a grotesque shell script that extracted the name, synopsis,
and description from each package. Then I manually chunked the output into a
transformer, asking it to:

   "Given a list of packages and descriptions, please[1] suggest ways of
    categorizing them (by function or theme) in a way that keeps each 
category
    about equal size."

...followed by 5 paragraphs of minutia to make sure it does the right thing.

Then I give it 200 lines of packages ~36 packages. It gives me its output
which is inevitably missing one or more of the constraints I gave it, so I
say "Did you follow my instructions?" [2] After this, it either gives me an
output with one less thing wrong with it and an apology, or goes off in a
huff and wastes all my GPU credit.


Things that are standing out so far are:

  - golang-web.scm could become a broader golang-networking.scm
  - golang-crypto.scm could become a broader golang-security.scm
  - There could be a need for a golang-text.scm that focusses on text 
parsing and
    processing.
  - A golang-utils could exist, however this could easily become the 'other'
    file.

Anyway, point is: I've got a very poor classification of packages in
golang.scm. I want to refine that using a langchain script. However, before
I move anymore forward with this, I wanted to ask: Is it okay to use AI
tools that are non-free in the aid of the GNU project?

 > > We may extend handy script accelerating committing process, see
 > > "etc/committer.scm"
 >
 > Okay, cool, I'll have a look at it on Friday.

Great, I haven't had a chance to look at this yet, but I will early next 
week.

Here's a link to my repo: https://github.com/cdo256/guix-package-refactor

Let me know what you think.

Kind regards,
  - Christina

[1] I am always polite to AI.
[2] Welcome to modern programming, people. Feels more like primary school
children.

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

Hello,

Thank you for your research on that, I was not expected to go
far like you did :-)

My short plan was to follow existing naming model which is in
use for python-*.scm, lisp-*.scm, perl-*scm, java-*.scm.
I see that golang would need some extra modules in the future
and comparing with python-*.scm the package base is not big
yet.

Let's use  Occam's razor for now. We have few common groups,
the task is to drop use-module (gnu packages golang) for each
of them by sorting packages into recently introduced modules.

- golang-web.scm
- golang-check.scm
- golang-xyz.scm
- golang-crypto.scm

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 1108 bytes --]

Re: Golang mudules to follow common grouping

[Christina O'Donnell]

Hi,

Okay that's a better plan :)

> Let's use  Occam's razor for now. We have few common groups,
> the task is to drop use-module (gnu packages golang) for each
> of them by sorting packages into recently introduced modules.
Right, put the dependencies in of golang-web in golang-web (etc.) as 
much as sensibly possible.

How about when a library is used by many packages in more than one 
different modules?

Kind regards
- Christina

On 05/02/2024 14:00, Sharlatan Hellseher wrote:
> Hello,
>
> Thank you for your research on that, I was not expected to go
> far like you did :-)
>
> My short plan was to follow existing naming model which is in
> use for python-*.scm, lisp-*.scm, perl-*scm, java-*.scm.
> I see that golang would need some extra modules in the future
> and comparing with python-*.scm the package base is not big
> yet.
>
> Let's use  Occam's razor for now. We have few common groups,
> the task is to drop use-module (gnu packages golang) for each
> of them by sorting packages into recently introduced modules.
>
> - golang-web.scm
> - golang-check.scm
> - golang-xyz.scm
> - golang-crypto.scm
>
> Thanks,
> Oleg

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

Hi,

I stick to this flow
- pick a package from golang.scm e.g. go-package-a
- identify destination (any of golang-*.scm)
- place go-package-a to e.g. golang-web.scm in alphabetic order
- remove go-package-a from golang.scm
- by using magit find all authors contributing to go-package-a
- place found copyright headers to golang-web.scm if they are unique
- grep -l -r go-package-a gnu/packages/
- identify if the module(s) missing use-module (gnu packages golang-web),
add if so
- guix build go-package-a
- build all dependent to go-package-a
- make a commit ...
...
repeat as many times as needed.


Batch of 40-50 is a golden median amount to review.

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 1150 bytes --]

Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 112 bytes --]

Hi Guix,

Do you think it is correctly extracted to golang-web?

https://issues.guix.gnu.org/66827

Thank,
Oleg

[-- Attachment #2: Type: text/html, Size: 346 bytes --]

Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi Guix!

I've noticed the number of Golang packages start growing. I expect more packages
to be reviewed and merged by quick check of Issues.

I think it's time to split (gnu packages golang) into some logical groups, see
Python, Lisp for example.

- golang-web
- golang-check
- golang-build
- golang-compression
- golang-crypto
- golang-xyz
- golang-...

Thoughts?


-- 
… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Maxim Cournoyer]

Hi,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi Guix!
>
> I've noticed the number of Golang packages start growing. I expect more packages
> to be reviewed and merged by quick check of Issues.
>
> I think it's time to split (gnu packages golang) into some logical groups, see
> Python, Lisp for example.
>
> - golang-web
> - golang-check
> - golang-build
> - golang-compression
> - golang-crypto
> - golang-xyz
> - golang-...
>
> Thoughts?

This sounds good to me.  Beware of correctly migrating the copyright
lines though... these can be annoying to move.

-- 
Thanks,
Maxim

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi Maxim,

I identified about 17 packages related to check/assert/mock/lint
process and moved them to (gnu packages golang-check)

https://issues.guix.gnu.org/66619

Thanks,
Oleg

On Tue, 10 Oct 2023 at 04:52, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
> Hi,
>
> Sharlatan Hellseher <sharlatanus@gmail.com> writes:
>
> > Hi Guix!
> >
> > I've noticed the number of Golang packages start growing. I expect more packages
> > to be reviewed and merged by quick check of Issues.
> >
> > I think it's time to split (gnu packages golang) into some logical groups, see
> > Python, Lisp for example.
> >
> > - golang-web
> > - golang-check
> > - golang-build
> > - golang-compression
> > - golang-crypto
> > - golang-xyz
> > - golang-...
> >
> > Thoughts?
>
> This sounds good to me.  Beware of correctly migrating the copyright
> lines though... these can be annoying to move.
>
> --
> Thanks,
> Maxim



-- 

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Wilko Meyer]

Hi,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> I think it's time to split (gnu packages golang) into some logical groups, see
> Python, Lisp for example.
>
> Thoughts?

IMHO this sounds like a good idea as it would improve the
maintainability of golang packages in the long run. We have 487 package
definitions right now:

(~/devel/guix-devel/gnu/packages) λ rg -c 'define-public' golang.scm
487

which already seems quite laborous to split into logical groups (while
getting the copyright information right as well and maintaining the
gitlog history etc.); so it probably classifies as a task that should be
tackled sooner than later as it'll cause more work over time the more
golang packages exist.

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

Hi,

I'm about to create golang-check and golang-build modules, which both are
core and would be in used by others. As Maxim mentioned, the most time
consumption task would be migrating copyrights header properly.

On Sun, 15 Oct 2023, 22:25 Wilko Meyer, <w@wmeyer.eu> wrote:

>
> Hi,
>
> Sharlatan Hellseher <sharlatanus@gmail.com> writes:
>
> > I think it's time to split (gnu packages golang) into some logical
> groups, see
> > Python, Lisp for example.
> >
> > Thoughts?
>
> IMHO this sounds like a good idea as it would improve the
> maintainability of golang packages in the long run. We have 487 package
> definitions right now:
>
> (~/devel/guix-devel/gnu/packages) λ rg -c 'define-public' golang.scm
> 487
>
> which already seems quite laborous to split into logical groups (while
> getting the copyright information right as well and maintaining the
> gitlog history etc.); so it probably classifies as a task that should be
> tackled sooner than later as it'll cause more work over time the more
> golang packages exist.
>
> --
> Kind regards,
>
> Wilko Meyer
> w@wmeyer.eu
>

[-- Attachment #2: Type: text/html, Size: 1657 bytes --]

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi,

I've accidentally included other patch to this series. I've re-sent it under the
new issue number https://issues.guix.gnu.org/66627.

Thanks,
Oleg


On Mon, 16 Oct 2023 at 19:17, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:
>
> Hi,
>
> I'm about to create golang-check and golang-build modules, which both are core and would be in used by others. As Maxim mentioned, the most time consumption task would be migrating copyrights header properly.
>
> On Sun, 15 Oct 2023, 22:25 Wilko Meyer, <w@wmeyer.eu> wrote:
>>
>>
>> Hi,
>>
>> Sharlatan Hellseher <sharlatanus@gmail.com> writes:
>>
>> > I think it's time to split (gnu packages golang) into some logical groups, see
>> > Python, Lisp for example.
>> >
>> > Thoughts?
>>
>> IMHO this sounds like a good idea as it would improve the
>> maintainability of golang packages in the long run. We have 487 package
>> definitions right now:
>>
>> (~/devel/guix-devel/gnu/packages) λ rg -c 'define-public' golang.scm
>> 487
>>
>> which already seems quite laborous to split into logical groups (while
>> getting the copyright information right as well and maintaining the
>> gitlog history etc.); so it probably classifies as a task that should be
>> tackled sooner than later as it'll cause more work over time the more
>> golang packages exist.
>>
>> --
>> Kind regards,
>>
>> Wilko Meyer
>> w@wmeyer.eu



--

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi,

Part II of the split https://issues.guix.gnu.org/66827

Now we would have
- golang.scm - which would be logically keep on minimal number of packages
- golang-check.scm
- golang-web.scm - in review

Planed
- golang-crypto.scm
- golang-compression.scm
- golang-build.scm - which would include low level 0 dependencies packages.

I found a lot of golang packages in (gnu packages syncthing) which are not 100%
related to syncthing itself, possibly the were left there long ago.

Thanks,
Oleg

On Thu, 19 Oct 2023 at 10:40, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:
>
> Hi,
>
> I've accidentally included other patch to this series. I've re-sent it under the
> new issue number https://issues.guix.gnu.org/66627.
>
> Thanks,
> Oleg
>
>
> On Mon, 16 Oct 2023 at 19:17, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:
> >
> > Hi,
> >
> > I'm about to create golang-check and golang-build modules, which both are core and would be in used by others. As Maxim mentioned, the most time consumption task would be migrating copyrights header properly.
> >
> > On Sun, 15 Oct 2023, 22:25 Wilko Meyer, <w@wmeyer.eu> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Sharlatan Hellseher <sharlatanus@gmail.com> writes:
> >>
> >> > I think it's time to split (gnu packages golang) into some logical groups, see
> >> > Python, Lisp for example.
> >> >
> >> > Thoughts?
> >>
> >> IMHO this sounds like a good idea as it would improve the
> >> maintainability of golang packages in the long run. We have 487 package
> >> definitions right now:
> >>
> >> (~/devel/guix-devel/gnu/packages) λ rg -c 'define-public' golang.scm
> >> 487
> >>
> >> which already seems quite laborous to split into logical groups (while
> >> getting the copyright information right as well and maintaining the
> >> gitlog history etc.); so it probably classifies as a task that should be
> >> tackled sooner than later as it'll cause more work over time the more
> >> golang packages exist.
> >>
> >> --
> >> Kind regards,
> >>
> >> Wilko Meyer
> >> w@wmeyer.eu
>
>
>
> --
>
> … наш разум - превосходная объяснительная машина которая способна
> найти смысл почти в чем угодно, истолковать любой феномен, но
> совершенно не в состоянии принять мысль о непредсказуемости.



-- 

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 3376 bytes --]

Hi Guix,

I'm about to prepare split and aggregation of all golag packages
 related to cryptography. The process would be the same as for
 golang-check and golang-web.


In progress:
golang-cryptography

Planned:
golang-compression
golang-build

Regards,
Oleg

On Mon, 30 Oct 2023, 01:17 Sharlatan Hellseher, <sharlatanus@gmail.com>
wrote:

> Hi,
>
> Part II of the split https://issues.guix.gnu.org/66827
>
> Now we would have
> - golang.scm - which would be logically keep on minimal number of packages
> - golang-check.scm
> - golang-web.scm - in review
>
> Planed
> - golang-crypto.scm
> - golang-compression.scm
> - golang-build.scm - which would include low level 0 dependencies packages.
>
> I found a lot of golang packages in (gnu packages syncthing) which are not
> 100%
> related to syncthing itself, possibly the were left there long ago.
>
> Thanks,
> Oleg
>
> On Thu, 19 Oct 2023 at 10:40, Sharlatan Hellseher <sharlatanus@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I've accidentally included other patch to this series. I've re-sent it
> under the
> > new issue number https://issues.guix.gnu.org/66627.
> >
> > Thanks,
> > Oleg
> >
> >
> > On Mon, 16 Oct 2023 at 19:17, Sharlatan Hellseher <sharlatanus@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > I'm about to create golang-check and golang-build modules, which both
> are core and would be in used by others. As Maxim mentioned, the most time
> consumption task would be migrating copyrights header properly.
> > >
> > > On Sun, 15 Oct 2023, 22:25 Wilko Meyer, <w@wmeyer.eu> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> Sharlatan Hellseher <sharlatanus@gmail.com> writes:
> > >>
> > >> > I think it's time to split (gnu packages golang) into some logical
> groups, see
> > >> > Python, Lisp for example.
> > >> >
> > >> > Thoughts?
> > >>
> > >> IMHO this sounds like a good idea as it would improve the
> > >> maintainability of golang packages in the long run. We have 487
> package
> > >> definitions right now:
> > >>
> > >> (~/devel/guix-devel/gnu/packages) λ rg -c 'define-public' golang.scm
> > >> 487
> > >>
> > >> which already seems quite laborous to split into logical groups (while
> > >> getting the copyright information right as well and maintaining the
> > >> gitlog history etc.); so it probably classifies as a task that should
> be
> > >> tackled sooner than later as it'll cause more work over time the more
> > >> golang packages exist.
> > >>
> > >> --
> > >> Kind regards,
> > >>
> > >> Wilko Meyer
> > >> w@wmeyer.eu
> >
> >
> >
> > --
> >
> > … наш разум - превосходная объяснительная машина которая способна
> > найти смысл почти в чем угодно, истолковать любой феномен, но
> > совершенно не в состоянии принять мысль о непредсказуемости.
>
>
>
> --
>
> … наш разум - превосходная объяснительная машина которая способна
> найти смысл почти в чем угодно, истолковать любой феномен, но
> совершенно не в состоянии принять мысль о непредсказуемости.
>

[-- Attachment #2: Type: text/html, Size: 5069 bytes --]

Re: Golang mudules to follow common grouping

[Maxim Cournoyer]

Hi Oleg,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi Guix,
>
> I'm about to prepare split and aggregation of all golag packages
>  related to cryptography. The process would be the same as for
>  golang-check and golang-web.
>
>
> In progress:
> golang-cryptography
>
> Planned:
> golang-compression
> golang-build

This sounds like a good idea to me!  Thanks for volunteering to do this
laborious task.

-- 
Maxim

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi,

Part III of the split is sent for review,
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68605

Thank,
Oleg
-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

Hi Guix!

I've pushed split IV
https://issues.guix.gnu.org/69042

Now there are all base golang-* modules which I'm about to populate on demand
during patch review.

- golang-build
- golang-check
- golang-compression
- golang-crypto
- golang-web
- golang-xyz

These two may be added as well if I see enough packages related to that topic.
- golang-graphics
- golang-maths / golang-science

Thanks,
Oleg


-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang mudules to follow common grouping

[Maxim Cournoyer]

Hi Sharlatan,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi Guix!
>
> I've pushed split IV
> https://issues.guix.gnu.org/69042
>
> Now there are all base golang-* modules which I'm about to populate on demand
> during patch review.
>
> - golang-build
> - golang-check
> - golang-compression
> - golang-crypto
> - golang-web
> - golang-xyz
>
> These two may be added as well if I see enough packages related to that topic.
> - golang-graphics
> - golang-maths / golang-science

Great work!  Thank you for seeing it through!

-- 
Thanks,
Maxim

Re: Golang mudules to follow common grouping

[Christina O'Donnell]

Hi Oleg,

On 13/01/2024 21:05, Sharlatan Hellseher wrote:
 > Hi Guix,
 >
 > I'm about to prepare split and aggregation of all golag packages
 >  related to cryptography. The process would be the same as for
 >  golang-check and golang-web.
 >
 >
 > In progress:
 > golang-cryptography
 >
 > Planned:
 > golang-compression
 > golang-build

I just wanted to offer my labor to this task. I'm very new to guix 
development
(I've only packaged one package so far), but I've got a fair amount of 
free time
and I've been looking for ways to contribute to Guix. (Plus I've already 
got my
head in golang.scm trying to package gitleaks and all its dependencies.)

I know that there's a lot to do already, but I was also wondering whether
they're going in any particular order once they're moved. There's a 
comment at
the end of golang.scm that says,

 > ;;; Avoid adding new packages to the end of this file. To reduce the 
chances
 > ;;; of a merge conflict, place them above by existing packages with 
similar
 > ;;; functionality or similar names.

Would it be more organized if they was just one order: either in 
alphabetical
order or grouped by function? My suggestion would be to use the file 
split to
group by function and then sort each file alphabetically. Do you know 
how it is
arranged for other languages?

Another question I have: Is there any tooling that can help big package
migrations like this go faster? Eg. a script to split one big diff into
individual package moves with appropriate change-log entries.

If you could share any scripts that you're using then that could help speed
up similar tasks in future.

Kind regards,
  - Christina

Re: Golang mudules to follow common grouping

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Hi Christina,

> Would it be more organized if they was just one order: either
> in alphabetical order or grouped by function? My suggestion
> would be to use the file split to group by function and then sort
> each file alphabetically. Do you know how it is arranged for other
> languages?

I've added comments in commentary section in the top of the file
asking to keep packages alphabetically sorted seen in
julia-xyz.scm as well. python-*.scm ordered semi random grouped
closer to package purpose which require more thinking where to put a new
one : -)

> Another question I have: Is there any tooling that can help big package
> migrations like this go faster? Eg. a script to split one big diff into
> individual package moves with appropriate change-log entries.

Good point her, I did manual split, with Emacs keyboard macros,
magit history scan for copyright lines and manual check where
package was used to include new module name.

The split into golang-crypto is in review now and there would be 2 more
common grouping: golang-compression and golang-build (or
golang-extension). Rest packages which are hard to determine a
group wound go to generic golang-xyz sorted alphabetically.

Let me know your tooling which you familiar with I might think
about some sort of automation.

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 2478 bytes --]

Re: Golang mudules to follow common grouping

[Christina O'Donnell]

Hi Oleg,

 > I've added comments in commentary section in the top of the file 
asking to
 > keep packages alphabetically sorted seen in julia-xyz.scm as well.
 > python-*.scm ordered semi random grouped closer to package purpose which
 > require more thinking where to put a new one :-)

Ah, I suppose it isn't that important given you can grep for the right
package.

 > Good point her, I did manual split, with Emacs keyboard macros, magit 
history
 > scan for copyright lines and manual check where package was used to 
include
 > new module name.
 >
 > The split into golang-crypto is in review now and there would be 2 
more common
 > grouping: golang-compression and golang-build (or golang-extension). Rest
 > packages which are hard to determine a group wound go to generic 
golang-xyz
 > sorted alphabetically.

Hmm, there seems to be a limit in the degree of parallelizability in this
process unfortunately. But if there's anything you can think of that 
could help
(manually) in this effort, then I'd be happy to help!

 > Let me know your tooling which you familiar with I might think
 > about some sort of automation.

I've had a grep around the web and I can't see any tools or libraries 
that have
been created already to perform functions like this. I know everyone has 
their
own scripts (I've got one written in shell but it won't be of much use 
here. I
would suggest writing it in Scheme so it could be easier to add more complex
features and it's familiar to Guix developers.

  1. Put a magic comment above each package that you would like to move.
  2. Run a simple script that makes a note of all of these into a 
to-move-list.
  3. Then stash the change with the comments you made (in case you need 
to change
     things)
  4. Run another script that takes the package list and performs the move in
     one's repository.
  5. Sort out the use-package declarations manually and run tests.
  6. When satisfied, stash the change and keep just the use-package changes.
  7. Run a final script that loops through all the packages and commits 
each one
     in turn.
  8. Rebase to suit.

That's 3 scripts:
  - xyz-source-transform-read-packages scm-filename package-list (for 
step 2)
  - xyz-source-transform-move-packages package-list from-filename 
to-filename
    (for step 4)
  - xyz-source-transform-stage-package-commit package-list from-filename
    to-filename (for step 7)

(Where 'xyz' is either 'guix' or something else depending on where the 
code ends
up part of Guix or not. That's not important for now.)

Some caveats:
  - I'm not a scheme programmer, but I did use Haskell at university so I'm
    familiar with thinking in a functional style.
  - For this to work, you'd have to update most package files that 
reference (gnu
    package golang) to also reference the new package, so there will be some
    use-module redundancy (which could be resolved in a final clean-up 
commit).

I'm also imagining some the possibility of having a script that can remove
redundant #:use-module's in the future, though I don't know if we care 
about a
few unneeded modules being included.

I'd love to hear what you think before I fire up another emacs instance. 
Does it
sound like a realistic workflow?

All comments welcome!

Kind regards,
  - Christina

Re: Golang mudules to follow common grouping

[Maxim Cournoyer]

Hi Christina,

Christina O'Donnell <cdo@mutix.org> writes:

> Hi Oleg,
>
> On 13/01/2024 21:05, Sharlatan Hellseher wrote:
>> Hi Guix,
>>
>> I'm about to prepare split and aggregation of all golag packages
>>  related to cryptography. The process would be the same as for
>>  golang-check and golang-web.
>>
>>
>> In progress:
>> golang-cryptography
>>
>> Planned:
>> golang-compression
>> golang-build
>
> I just wanted to offer my labor to this task. I'm very new to guix
> development
> (I've only packaged one package so far), but I've got a fair amount of
> free time
> and I've been looking for ways to contribute to Guix. (Plus I've
> already got my
> head in golang.scm trying to package gitleaks and all its dependencies.)

Thanks for offering to help!  The most important part in doing so is to
synchronize with the other members of the Go team to avoid duplicating
the work, I think.

> I know that there's a lot to do already, but I was also wondering whether
> they're going in any particular order once they're moved. There's a
> comment at
> the end of golang.scm that says,
>
>> ;;; Avoid adding new packages to the end of this file. To reduce the
>       chances
>> ;;; of a merge conflict, place them above by existing packages with
>       similar
>> ;;; functionality or similar names.
>
> Would it be more organized if they was just one order: either in
> alphabetical
> order or grouped by function? My suggestion would be to use the file
> split to
> group by function and then sort each file alphabetically. Do you know
> how it is
> arranged for other languages?

That'd be neat, but in practice is hard to maintain as the module
becomes quite large and people easily (and understandably) miss the
intended grouping when adding new packages.

> Another question I have: Is there any tooling that can help big package
> migrations like this go faster? Eg. a script to split one big diff into
> individual package moves with appropriate change-log entries.

Not that I'm aware of.

-- 
Thanks,
Maxim