[Guix Website] A Search Page for Packages

[Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 951 bytes --]

Hello Guix,

This message is in reply to an old one (https://lists.gnu.org/archive/html/guix-devel/2020-07/msg00050.html), where Danjela shared the work done during the internship. Danjela said:

> I'd like to know if you find the page useful so that I can hopefully start working on incorporating the search functionality into this page: https://guix.gnu.org/packages/.

I'd like to have Danjela's project integrated as soon as possible. People have been asking for this feature for a long time. And we could take the oportunity to update URL paths to packages so that they work as permalinks, which is another change people have been asking for for a long time.

What was the integration plan for Danjela's work, specifically? How can I help?

I have a copy of the "guix-packages-search.scm" script (don't remember where I got it from), and tried it out locally already.


---
Luis Felipe López Acevedo
https://luis-felipe.gitlab.io/

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1615 bytes --]


Luis Felipe <luis.felipe.la@protonmail.com> writes:

> Hello Guix,
>
> This message is in reply to an old one
> (https://lists.gnu.org/archive/html/guix-devel/2020-07/msg00050.html),
> where Danjela shared the work done during the internship. Danjela
> said:
>
>> I'd like to know if you find the page useful so that I can hopefully
>> start working on incorporating the search functionality into this
>> page: https://guix.gnu.org/packages/.
>
> I'd like to have Danjela's project integrated as soon as
> possible. People have been asking for this feature for a long
> time. And we could take the oportunity to update URL paths to packages
> so that they work as permalinks, which is another change people have
> been asking for for a long time.
>
> What was the integration plan for Danjela's work, specifically? How can I help?
>
> I have a copy of the "guix-packages-search.scm" script (don't remember
> where I got it from), and tried it out locally already.

So, personally I think the simplist path forward is to create
packages.guix.gnu.org as the domain, and do stuff there. That will help
avoid getting stuck up on how to integrate with the main website.

I'm more than happy to try and help others get this done, I just don't
have enough time at the moment to commit to it myself.

What Danjela produced in terms of the search is probably fine as
something to go with, maybe the additional thing that needs writing is
pages for the individual packages that the search can link to (they can
just be very simple initially).

But yeah, are you (or anyone else who's reading) up for hacking on this?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 1009 bytes --]

------- Original Message -------
On Saturday, November 5th, 2022 at 18:53, Christopher Baines <mail@cbaines.net> wrote:

> So, personally I think the simplist path forward is to create
> packages.guix.gnu.org as the domain, and do stuff there. That will help
> avoid getting stuck up on how to integrate with the main website.

I like that.


> I'm more than happy to try and help others get this done, I just don't
> have enough time at the moment to commit to it myself.

I understand.


> What Danjela produced in terms of the search is probably fine as
> something to go with, maybe the additional thing that needs writing is
> pages for the individual packages that the search can link to (they can
> just be very simple initially).
> 

> But yeah, are you (or anyone else who's reading) up for hacking on this?

Yeah, I'm willing to work on that. I'll start by adding support for handling requests for package detail pages.

I'll report any progress (or lack of it).

Cheers,

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 1192 bytes --]

Hi,


------- Original Message -------
On Saturday, November 5th, 2022 at 19:59, Luis Felipe <luis.felipe.la@protonmail.com> wrote:

> ------- Original Message -------
> On Saturday, November 5th, 2022 at 18:53, Christopher Baines mail@cbaines.net wrote:
> 

> > What Danjela produced in terms of the search is probably fine as
> > something to go with, maybe the additional thing that needs writing is
> > pages for the individual packages that the search can link to (they can
> > just be very simple initially).
> 

> > But yeah, are you (or anyone else who's reading) up for hacking on this?
>
> Yeah, I'm willing to work on that. I'll start by adding support for handling requests for package detail pages.

I have a working package browser now and will be styling it next. I'll link to a public repository later.

However, I have some questions about the Data Service:

1. Is there any way to page through the data (e.g. https://dev.socrata.com/docs/paging.html#2.1)?
2. What would be an efficient way to get the total count of packages? (to be used as in https://guix.gnu.org/en/packages/)
3. Is there any way to get the "supported-systems" info of a package?

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2021 bytes --]


Luis Felipe <luis.felipe.la@protonmail.com> writes:

>> > What Danjela produced in terms of the search is probably fine as
>> > something to go with, maybe the additional thing that needs writing is
>> > pages for the individual packages that the search can link to (they can
>> > just be very simple initially).
>>
>> > But yeah, are you (or anyone else who's reading) up for hacking on this?
>>
>> Yeah, I'm willing to work on that. I'll start by adding support for handling requests for package detail pages.
>
> I have a working package browser now and will be styling it next. I'll link to a public repository later.
>
> However, I have some questions about the Data Service:

Hey! Sorry for the slow reply.

> 1. Is there any way to page through the data
> (e.g. https://dev.socrata.com/docs/paging.html#2.1)?

For most bits of data, yes. Although I haven't used limit/offset style
pagination as that doesn't perform particularly well.

The Guix Data Service web pages should help with how to use the API, so
you can look at how the packages page handles pagination for example:

  https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80/packages

> 2. What would be an efficient way to get the total count of packages?
> (to be used as in https://guix.gnu.org/en/packages/)

You can get a count here for example [1], that also works with the
latest-processed-revision URL [2]. I think the counting methadolody is a
bit different to what's currently used though.

1: https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80.json
2: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision.json

> 3. Is there any way to get the "supported-systems" info of a package?

The value of that specific field isn't recorded, but you could look at
the derivations of the package, then just find the systems, that should
be pretty close:

  https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80/package/0ad/0.0.26-alpha.json

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 2255 bytes --]

------- Original Message -------
On Sunday, November 13th, 2022 at 08:46, Christopher Baines <mail@cbaines.net> wrote:

> Hey! Sorry for the slow reply.

No problem :)


> > 1. Is there any way to page through the data
> > (e.g. https://dev.socrata.com/docs/paging.html#2.1)?
> 

> For most bits of data, yes. Although I haven't used limit/offset style
> pagination as that doesn't perform particularly well.
> 

> The Guix Data Service web pages should help with how to use the API, so
> you can look at how the packages page handles pagination for example:
> 

> https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80/packages

I see. Although it seems that pagination only works to page through the whole package data set and not through the results of a search. So right now the package browser can only show LIMIT, or less, number of packages after performing a search. For example:

https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?locale=en_US.UTF-8&search_query=python&field=version&field=synopsis&after_name=python-cython&limit_results=30

So one can't use "search_query" and "after_name" in the same request. I'm stuck in that part.


> > 2. What would be an efficient way to get the total count of packages?
> > (to be used as in https://guix.gnu.org/en/packages/)
> 

> You can get a count here for example [1], that also works with the
> latest-processed-revision URL [2]. I think the counting methadolody is a
> bit different to what's currently used though.
> 

> 1: https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80.json
> 2: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision.json

Ah, I'll use that.


> > 3. Is there any way to get the "supported-systems" info of a package?
> 

> The value of that specific field isn't recorded, but you could look at
> the derivations of the package, then just find the systems, that should
> be pretty close:
> 

> https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80/package/0ad/0.0.26-alpha.json

I decided to link to the package builds in the CI for now, but I'll explore that option too.

Thanks, Chris.

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]


Luis Felipe <luis.felipe.la@protonmail.com> writes:

>> > 1. Is there any way to page through the data
>> > (e.g. https://dev.socrata.com/docs/paging.html#2.1)?
>>
>
>> For most bits of data, yes. Although I haven't used limit/offset style
>> pagination as that doesn't perform particularly well.
>>
>
>> The Guix Data Service web pages should help with how to use the API, so
>> you can look at how the packages page handles pagination for example:
>>
>
>> https://data.guix.gnu.org/revision/c4025af8c11c3e9ea0c2747b6c475c916fb61d80/packages
>
> I see. Although it seems that pagination only works to page through
> the whole package data set and not through the results of a search. So
> right now the package browser can only show LIMIT, or less, number of
> packages after performing a search. For example:
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?locale=en_US.UTF-8&search_query=python&field=version&field=synopsis&after_name=python-cython&limit_results=30
>
> So one can't use "search_query" and "after_name" in the same request. I'm stuck in that part.

Ah, yeah, that's true.

I think whatever you've implemented is fine for now. The Guix Data
Service API can always be changed if that would be more helpful.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 777 bytes --]

------- Original Message -------
On Wednesday, November 16th, 2022 at 16:25, Christopher Baines <mail@cbaines.net> wrote:

> Luis Felipe luis.felipe.la@protonmail.com writes:
> 

> > So one can't use "search_query" and "after_name" in the same request. I'm stuck in that part.
> 

> Ah, yeah, that's true.
> 

> I think whatever you've implemented is fine for now. The Guix Data
> Service API can always be changed if that would be more helpful.

Okay, I'd like to have that eventually because I've been using the package browser as it is right now and I've discovered software that didn't know even existed, so I'd really like to go through the whole results of a given search.

I'll try to familiarize myself with the Data Service code and see if I can help.

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 343 bytes --]

Hi,

------- Original Message -------
On Wednesday, November 9th, 2022 at 16:43, Luis Felipe <luis.felipe.la@protonmail.com> wrote:

> I have a working package browser now and will be styling it next. I'll link to a public repository later.

Here is the source code, by the way: https://codeberg.org/luis-felipe/guix-packages-website 

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]


Luis Felipe <luis.felipe.la@protonmail.com> writes:

> ------- Original Message -------
> On Wednesday, November 9th, 2022 at 16:43, Luis Felipe <luis.felipe.la@protonmail.com> wrote:
>
>> I have a working package browser now and will be styling it
>> next. I'll link to a public repository later.
>
> Here is the source code, by the way:
> https://codeberg.org/luis-felipe/guix-packages-website

That looks great!

Maybe some kind of alpha/beta/in development banner or message would be
good, just to set expectations that this is a new thing that's being
worked on.

I can look at setting up DNS/NGinx for packages.guix.gnu.org. I'll
probably do this on bayfront out of familiarity. I think it's fine to
just run in a screen session at first.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 783 bytes --]

------- Original Message -------
On Wednesday, November 16th, 2022 at 16:32, Christopher Baines <mail@cbaines.net> wrote:

> Luis Felipe luis.felipe.la@protonmail.com writes:
> > 

> > Here is the source code, by the way:
> > https://codeberg.org/luis-felipe/guix-packages-website
> 

> That looks great!

:)


> Maybe some kind of alpha/beta/in development banner or message would be
> good, just to set expectations that this is a new thing that's being
> worked on.

I'll add that.


> I can look at setting up DNS/NGinx for packages.guix.gnu.org. I'll
> probably do this on bayfront out of familiarity. I think it's fine to
> just run in a screen session at first.

That would be great, because I'm not familiar with those things at all.

Thank you,

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 923 bytes --]

Hi Chris,

> > On Wednesday, November 9th, 2022 at 16:43, Luis Felipe luis.felipe.la@protonmail.com wrote:
> > 

> > > I have a working package browser now and will be styling it
> > > next. I'll link to a public repository later.
> > 

> > Here is the source code, by the way:
> > https://codeberg.org/luis-felipe/guix-packages-website
> 

> Maybe some kind of alpha/beta/in development banner or message would be
> good, just to set expectations that this is a new thing that's being
> worked on.

I added that and some other things that were missing and realeased version 0.1.0 of the package browser.


> I can look at setting up DNS/NGinx for packages.guix.gnu.org. I'll
> probably do this on bayfront out of familiarity. I think it's fine to
> just run in a screen session at first.

Let me know if you still can squeeze that into your schedule or if I should contact guix-sysadmin.

Cheers,

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]


Luis Felipe <luis.felipe.la@protonmail.com> writes:

>> I can look at setting up DNS/NGinx for packages.guix.gnu.org. I'll
>> probably do this on bayfront out of familiarity. I think it's fine to
>> just run in a screen session at first.
>
> Let me know if you still can squeeze that into your schedule or if I
> should contact guix-sysadmin.

I've got around to doing this now. I pretty much just copied the
configuration for qa.guix.gnu.org which I setup recently.

It's running in a screen session under my user on bayfront, which isn't
ideal, but should be OK in the short term. This is still a good step
towards deploying it properly.

It's really nice to have multiple things now (packages.guix.gnu.org,
qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
the Guix Data Service!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 1072 bytes --]

On Saturday, November 19th, 2022 at 10:51, Christopher Baines <mail@cbaines.net> wrote:


> Luis Felipe luis.felipe.la@protonmail.com writes:
> 

> > > I can look at setting up DNS/NGinx for packages.guix.gnu.org. I'll
> > > probably do this on bayfront out of familiarity. I think it's fine to
> > > just run in a screen session at first.
> > 

> > Let me know if you still can squeeze that into your schedule or if I
> > should contact guix-sysadmin.
> 

> 

> I've got around to doing this now. I pretty much just copied the
> configuration for qa.guix.gnu.org which I setup recently.
> 

> It's running in a screen session under my user on bayfront, which isn't
> ideal, but should be OK in the short term. This is still a good step
> towards deploying it properly.
> 

> It's really nice to have multiple things now (packages.guix.gnu.org,
> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
> the Guix Data Service!

Great! Thanks, Chris.

I'll send a patch to link to it from the static website's "Package" menu.

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 216 bytes --]

On Saturday, November 19th, 2022 at 12:03, Luis Felipe <luis.felipe.la@protonmail.com> wrote:

> I'll send a patch to link to it from the static website's "Package" menu.

Done: https://issues.guix.gnu.org/59385.

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi Luis,

Luis Felipe <luis.felipe.la@protonmail.com> skribis:

> On Saturday, November 19th, 2022 at 12:03, Luis Felipe <luis.felipe.la@protonmail.com> wrote:
>
>> I'll send a patch to link to it from the static website's "Package" menu.
>
> Done: https://issues.guix.gnu.org/59385.

You’re a hero, thank you!!

Ludo’.

Re: [Guix Website] A Search Page for Packages

[John Kehayias]

Hi Chris and Luis,

On Sat, Nov 19, 2022 at 10:51 AM, Christopher Baines wrote:
> I've got around to doing this now. I pretty much just copied the
> configuration for qa.guix.gnu.org which I setup recently.
>
> It's running in a screen session under my user on bayfront, which isn't
> ideal, but should be OK in the short term. This is still a good step
> towards deploying it properly.
>
> It's really nice to have multiple things now (packages.guix.gnu.org,
> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
> the Guix Data Service!
>

Very nice! Looks good and helpful.

Question: what is the search syntax that is expected? I tried searching for "stump" and it only returned a go package that ended in "stump." Searching for "stumpwm" turned up all the expected suspects though. While "stump*" was the same as "stump."

Thanks for all your work on this, very nice addition!

John

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


John Kehayias <john.kehayias@protonmail.com> writes:

> Question: what is the search syntax that is expected? I tried
> searching for "stump" and it only returned a go package that ended in
> "stump." Searching for "stumpwm" turned up all the expected suspects
> though. While "stump*" was the same as "stump."

The search functionality is being provided by the Guix Data Service,
which in turn is searching through records in a PostgreSQL database.

The relevant code is here [1], and it's the plainto_tsquery bit that
starts the transformation from a string to what PostgreSQL is using for
the search.

1: https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data-service/model/package.scm#n144

So yeah, I don't think the way it's implemented currently supports * as
a wildcard, although maybe that could be supported by changing the way
the query works.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[John Kehayias]

Hi Chris

On Sat, Nov 19, 2022 at 04:13 PM, Christopher Baines wrote:
> The search functionality is being provided by the Guix Data Service,
> which in turn is searching through records in a PostgreSQL database.
>
> The relevant code is here [1], and it's the plainto_tsquery bit that
> starts the transformation from a string to what PostgreSQL is using for
> the search.
>
> 1:
> <https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data-service/model/package.scm#n144>
>

I see. My knowledge of PostgreSQL is limited to emergency measures for when something happens with the home server :)

> So yeah, I don't think the way it's implemented currently supports * as
> a wildcard, although maybe that could be supported by changing the way
> the query works.

A low priority, but it would be nice on the list of things for future versions to improve the search. I think basic substring/partial matches would go a long way. Probably very easy for the PostgreSQL-ers. Or maybe make it match as best it can the result lists from "guix search" as a nice parallel.

Anyway, thanks again for this, great to see it live!

John

Re: [Guix Website] A Search Page for Packages

[zimoun]

Hi Chris,

Really cool!  Thank you Felipe and Chris for this new
packages.guix.gnu.org. :-)


On Sat, 19 Nov 2022 at 10:51, Christopher Baines <mail@cbaines.net> wrote:

> It's really nice to have multiple things now (packages.guix.gnu.org,
> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
> the Guix Data Service!

Chris, what do you mean by «that are made possible by the Guix Data
Service»?

Cheers,
simon

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> On Sat, 19 Nov 2022 at 10:51, Christopher Baines <mail@cbaines.net> wrote:
>
>> It's really nice to have multiple things now (packages.guix.gnu.org,
>> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
>> the Guix Data Service!
>
> Chris, what do you mean by «that are made possible by the Guix Data
> Service»?

Only that the things I mention are making use of the Guix Data Service
to provide and query data.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[zimoun]

Hi Chris,

On Mon, 21 Nov 2022 at 08:57, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> On Sat, 19 Nov 2022 at 10:51, Christopher Baines <mail@cbaines.net> wrote:
>>
>>> It's really nice to have multiple things now (packages.guix.gnu.org,
>>> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
>>> the Guix Data Service!
>>
>> Chris, what do you mean by «that are made possible by the Guix Data
>> Service»?
>
> Only that the things I mention are making use of the Guix Data Service
> to provide and query data.

Just to be sure, by Guix Data Service, do you mean the code
https://git.savannah.gnu.org/git/guix/data-service.git?  Or do you mean
a generic name for all the services as qa.guix, bordeaux.guix,
packages.guix or even issues.guix, logs.guix?


Cheers,
simon

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Mon, 21 Nov 2022 at 08:57, Christopher Baines <mail@cbaines.net> wrote:
>> zimoun <zimon.toutoune@gmail.com> writes:
>>
>>> On Sat, 19 Nov 2022 at 10:51, Christopher Baines <mail@cbaines.net> wrote:
>>>
>>>> It's really nice to have multiple things now (packages.guix.gnu.org,
>>>> qa.guix.gnu.org, bordeaux.guix.gnu.org, ...) that are made possible by
>>>> the Guix Data Service!
>>>
>>> Chris, what do you mean by «that are made possible by the Guix Data
>>> Service»?
>>
>> Only that the things I mention are making use of the Guix Data Service
>> to provide and query data.
>
> Just to be sure, by Guix Data Service, do you mean the code
> https://git.savannah.gnu.org/git/guix/data-service.git?  Or do you mean
> a generic name for all the services as qa.guix, bordeaux.guix,
> packages.guix or even issues.guix, logs.guix?

The former, the software with this source
https://git.savannah.gnu.org/git/guix/data-service.git

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi again,

Luis Felipe <luis.felipe.la@protonmail.com> skribis:

>> > On Wednesday, November 9th, 2022 at 16:43, Luis Felipe luis.felipe.la@protonmail.com wrote:
>> > 
>
>> > > I have a working package browser now and will be styling it
>> > > next. I'll link to a public repository later.
>> > 
>
>> > Here is the source code, by the way:
>> > https://codeberg.org/luis-felipe/guix-packages-website
>> 
>
>> Maybe some kind of alpha/beta/in development banner or message would be
>> good, just to set expectations that this is a new thing that's being
>> worked on.
>
> I added that and some other things that were missing and realeased version 0.1.0 of the package browser.

One thing I noticed is that i18n support seems to be missing.

It’s OK to keep that for later, but how about making sure URLs are
prepared to handle that, by adding /en in the path?

That is, from:
https://packages.guix.gnu.org/packages/ecl-cl-unicode/0.1.6/
to:
https://packages.guix.gnu.org/en/packages/ecl-cl-unicode/0.1.6/


Sysadmin-wise, we’ll need a proper service in (sysadmin web), Chris.
:-)  (I can help with that if needed.)

Thanks,
Ludo’.

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> One thing I noticed is that i18n support seems to be missing.
>
> It’s OK to keep that for later, but how about making sure URLs are
> prepared to handle that, by adding /en in the path?
>
> That is, from:
> https://packages.guix.gnu.org/packages/ecl-cl-unicode/0.1.6/
> to:
> https://packages.guix.gnu.org/en/packages/ecl-cl-unicode/0.1.6/

On a similar subject, would it be sensible to remove the packages/ bit
from the URL, e.g. this?

  https://packages.guix.gnu.org/en/ecl-cl-unicode/0.1.6/

> Sysadmin-wise, we’ll need a proper service in (sysadmin web), Chris.
> :-)  (I can help with that if needed.)

I should be able to get to it at some point as well, I need to write a
service for the qa-frontpage, so hopefully I can do some copy and
pasting.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 1100 bytes --]

On Monday, November 21st, 2022 at 13:21, Christopher Baines <mail@cbaines.net> wrote:

> Ludovic Courtès ludo@gnu.org writes:
> 

> > One thing I noticed is that i18n support seems to be missing.
> > 

> > It’s OK to keep that for later, but how about making sure URLs are
> > prepared to handle that, by adding /en in the path?
> > 

> > That is, from:
> > https://packages.guix.gnu.org/packages/ecl-cl-unicode/0.1.6/
> > to:
> > https://packages.guix.gnu.org/en/packages/ecl-cl-unicode/0.1.6/
> 

> 

> On a similar subject, would it be sensible to remove the packages/ bit
> from the URL, e.g. this?
> 

> https://packages.guix.gnu.org/en/ecl-cl-unicode/0.1.6/

About that, the "packages" part looks redundant for sure, but the reason I added it was that I started thinking that the whole Guix website could be copy-pasted to this Artanis project and have the whole website again served on guix.gnu.org at some point, where the "packages" part would not be redundant anymore, but actually necessary. But if this doesn't sound right, I can simplify the URL path then :)

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Luis Felipe]

[-- Attachment #1.1: Type: text/plain, Size: 1740 bytes --]

On Monday, November 21st, 2022 at 12:57, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi again,
> 

> Luis Felipe luis.felipe.la@protonmail.com skribis:
> 

> > > > On Wednesday, November 9th, 2022 at 16:43, Luis Felipe luis.felipe.la@protonmail.com wrote:
> > 

> > > > > I have a working package browser now and will be styling it
> > > > > next. I'll link to a public repository later.
> > 

> > > > Here is the source code, by the way:
> > > > https://codeberg.org/luis-felipe/guix-packages-website
> > 

> > > Maybe some kind of alpha/beta/in development banner or message would be
> > > good, just to set expectations that this is a new thing that's being
> > > worked on.
> > 

> > I added that and some other things that were missing and realeased version 0.1.0 of the package browser.
> 

> 

> One thing I noticed is that i18n support seems to be missing.
> 

> It’s OK to keep that for later, but how about making sure URLs are
> prepared to handle that, by adding /en in the path?
> 

> That is, from:
> https://packages.guix.gnu.org/packages/ecl-cl-unicode/0.1.6/
> to:
> https://packages.guix.gnu.org/en/packages/ecl-cl-unicode/0.1.6/

Yeah, I left i18n for later, and was actually considering getting rid of the language prefix in URL paths in favor of using HTTP Request's Accept-Language header to detect the user's preferred language and serve the localized resources in that language if available (similar to what happens with apps on the desktop). That looked like less work to me than [also] internationalizing the URLs, but maybe it is not a good practice? In any case, I'm planning to add i18n for version 0.2.0 and was hoping to ask Julien and Florian for advise about that...

[-- Attachment #1.2: publickey - luis.felipe.la@protonmail.com - 0x12DE1598.asc --]
[-- Type: application/pgp-keys, Size: 1722 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 509 bytes --]

Re: [Guix Website] A Search Page for Packages

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

I believe the website used both: an unprefixed url will serve the page fsom the prefixed directory based on accept-language, but you can also switch language easily by using the prefixed path.

Not translating URLs is fine and allows to easily switch language while staying on the same page.

Le 21 novembre 2022 20:00:29 GMT+01:00, Luis Felipe <luis.felipe.la@protonmail.com> a écrit :
>On Monday, November 21st, 2022 at 12:57, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Hi again,
>> 
>
>> Luis Felipe luis.felipe.la@protonmail.com skribis:
>> 
>
>> > > > On Wednesday, November 9th, 2022 at 16:43, Luis Felipe luis.felipe.la@protonmail.com wrote:
>> > 
>
>> > > > > I have a working package browser now and will be styling it
>> > > > > next. I'll link to a public repository later.
>> > 
>
>> > > > Here is the source code, by the way:
>> > > > https://codeberg.org/luis-felipe/guix-packages-website
>> > 
>
>> > > Maybe some kind of alpha/beta/in development banner or message would be
>> > > good, just to set expectations that this is a new thing that's being
>> > > worked on.
>> > 
>
>> > I added that and some other things that were missing and realeased version 0.1.0 of the package browser.
>> 
>
>> 
>
>> One thing I noticed is that i18n support seems to be missing.
>> 
>
>> It’s OK to keep that for later, but how about making sure URLs are
>> prepared to handle that, by adding /en in the path?
>> 
>
>> That is, from:
>> https://packages.guix.gnu.org/packages/ecl-cl-unicode/0.1.6/
>> to:
>> https://packages.guix.gnu.org/en/packages/ecl-cl-unicode/0.1.6/
>
>Yeah, I left i18n for later, and was actually considering getting rid of the language prefix in URL paths in favor of using HTTP Request's Accept-Language header to detect the user's preferred language and serve the localized resources in that language if available (similar to what happens with apps on the desktop). That looked like less work to me than [also] internationalizing the URLs, but maybe it is not a good practice? In any case, I'm planning to add i18n for version 0.2.0 and was hoping to ask Julien and Florian for advise about that...

[-- Attachment #2: Type: text/html, Size: 5739 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> So, personally I think the simplist path forward is to create
> packages.guix.gnu.org as the domain, and do stuff there. That will help
> avoid getting stuck up on how to integrate with the main website.

Agreed, let’s do that.

Luis, please email guix-sysadmin when you reach the last steps: getting
the service running on project infrastructure, and getting the
sub-domain in place.

Thanks a lot!

Ludo’.

[Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

Hello everyone,

This is Danjela, the Outreachy intern at the Guix Data Service.

Taking into consideration the suggestion made in this thread:
https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
mentor, Christopher Baines suggested me to write a script that serves a
search page for packages using the search functionality within the Guix
Data Service,
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
.

The prototype page can be accessed through a test version of the Guix
website that Chris deployed:
http://guix-website-test.cbaines.net/packages/search

I'd like to know if you find the page useful so that I can hopefully start
working on incorporating the search functionality into this page:
https://guix.gnu.org/packages/.

Best Regards,
Danjela

[-- Attachment #2: Type: text/html, Size: 1491 bytes --]

Re: [Guix Website] A Search Page for Packages

[Cook, Malcolm]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

A few quick thoughts –


  *   There should be a link documenting the search language.  Are there Booleans allowed?  Is “stemming” automatic?  Do quote characters do anything?  Is proximity searching available?  Regular expressions?  What columns are searched?  License?  Version? Author?
  *   Immediate feedback of hits as you type is “nice”.  IMO suggested completion of search times is nicer.
  *   Browsing alphabetically by picking a letter provide little to no value
  *   Full detail of hits in columnar format is preferred over “cards”

Still lurking myself but loving the improvements….

Go Guix,

~Malcolm


From: Guix-devel <guix-devel-bounces+mec=stowers.org@gnu.org> on behalf of Daniela Lura <danielaluraa@gmail.com>
Date: Wednesday, July 8, 2020 at 1:50 PM
To: "guix-devel@gnu.org" <guix-devel@gnu.org>
Subject: [Guix Website] A Search Page for Packages

CAUTION: This email was received from an External Source

Hello everyone,

This is Danjela, the Outreachy intern at the Guix Data Service.

Taking into consideration the suggestion made in this thread:
https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my mentor, Christopher Baines suggested me to write a script that serves a search page for packages using the search functionality within the Guix Data Service, https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100.

The prototype page can be accessed through a test version of the Guix website that Chris deployed: http://guix-website-test.cbaines.net/packages/search

I'd like to know if you find the page useful so that I can hopefully start working on incorporating the search functionality into this page: https://guix.gnu.org/packages/.

Best Regards,
Danjela

[-- Attachment #2: Type: text/html, Size: 9409 bytes --]

Re: [Guix Website] A Search Page for Packages

[pelzflorian (Florian Pelz)]

Yes, this kind of search is much better.  Good work!

I wonder, can bad people abuse the search for denial-of-service or is
the number of searches limited?

On Wed, Jul 08, 2020 at 08:31:37PM +0000, Cook, Malcolm wrote:
>   *   Immediate feedback of hits as you type is “nice”.  IMO suggested completion of search times is nicer.

This has two sides.  I find it great that search works without
JavaScript and think users should not normally be required to enable
privacy-killing JavaScript.  However, search-as-you-type indeed is a
nice feature for the JS-enabled users.  I do not know how to
reasonably implement it though (and in what programming language).
Also the improvement of search-as-you-type vs. the current proposal is
not big.

Regards,
Florian

Re: [Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

Hi Malcolm,

As noted in this page:
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages,
you can search by package name or synopsis.
I think Chris, whom I have cc-ed, is the most suitable person to talk about
the required functionalities of the search, because he wrote the query that
provides the search functionality within the Guix Data Service, which is
used in the prototype.
-
- > Immediate feedback of hits as you type is “nice”.  IMO suggested
completion of search times is nicer.
-
- Seems sensible, I will look into it even though I'm not yet sure how to
implement it. :)
-
- > Full detail of hits in columnar format is preferred over “cards”
- In order for the prototype page to visually fit the design here:
https://guix.gnu.org/packages/ I used cards.

Best,
Danjela

On Wed, 8 Jul 2020 at 22:31, Cook, Malcolm <MEC@stowers.org> wrote:

> A few quick thoughts –
>
>
>
>    - There should be a link documenting the search language.  Are there
>    Booleans allowed?  Is “stemming” automatic?  Do quote characters do
>    anything?  Is proximity searching available?  Regular expressions?  What
>    columns are searched?  License?  Version? Author?
>    - Immediate feedback of hits as you type is “nice”.  IMO suggested
>    completion of search times is nicer.
>    - Browsing alphabetically by picking a letter provide little to no
>    value
>    - Full detail of hits in columnar format is preferred over “cards”
>
>
>
> Still lurking myself but loving the improvements….
>
>
>
> Go Guix,
>
>
>
> ~Malcolm
>
>
>
>
>
> *From: *Guix-devel <guix-devel-bounces+mec=stowers.org@gnu.org> on behalf
> of Daniela Lura <danielaluraa@gmail.com>
> *Date: *Wednesday, July 8, 2020 at 1:50 PM
> *To: *"guix-devel@gnu.org" <guix-devel@gnu.org>
> *Subject: *[Guix Website] A Search Page for Packages
>
>
>
> CAUTION: This email was received from an External Source
>
>
>
> Hello everyone,
>
>
>
> This is Danjela, the Outreachy intern at the Guix Data Service.
>
>
>
> Taking into consideration the suggestion made in this thread:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
>
>
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>
>
>
> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.
>
>
>
> Best Regards,
>
> Danjela
>

[-- Attachment #2: Type: text/html, Size: 6987 bytes --]

Re: [Guix Website] A Search Page for Packages

[Bengt Richter]

Hi Daniela,

On +2020-07-08 20:49:35 +0200, Daniela Lura wrote:
> Hello everyone,
> 
> This is Danjela, the Outreachy intern at the Guix Data Service.
> 
> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
> 
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>

This looks useful :)

I did notice that clicking firefox-esr's[1] reader-view button (or equivalently
hitting Ctl-Alt-r [yes, lower case]) niether shows the latest search result
(I searched for -"sha" (BTW, my convention: prefixed '-' on double-quoted string means
without the quotes, ok?)).

[1] Mozilla Firefox 68.10.0esr per -"firefox --version"

It seemed to go back to a locked-in one-time-memoized value that persists
through other searches and even exiting the browser and restarting it,
and even after hitting its reload button.

All it keeps showing in reader mode is
--8<---------------cut here---------------start------------->8---
guix-website-test.cbaines.net
Packages
1 minute
ghc-cryptohash-sha1 0.11.100.1

This Haskell package provides an incremental and one-pass, pure API to the SHA-1 hash algorithm (https://en.wikipedia.org/wiki/SHA-1), including HMAC support (https://en.wikipedia.org/wiki/HMAC), with performance close to the fastest implementations available in other languages. The implementation is made…
ghc-cryptohash-sha256 0.11.101.0

This Haskell package provides an incremental and one-pass, pure API to the SHA-256 cryptographic hash algorithm (https://en.wikipedia.org/wiki/SHA-2), with performance close to the fastest implementations available in other languages. The implementation is made in C…
--8<---------------cut here---------------end--------------->8---
which I think is the result of my first search of all -- which was for
-"sha256" (sans quotes again, no more explaining that :).

Actually, on the reader page, there is visually a blank line above the
-"ghc-cryptohash-sha256 0.11.101.0" line in the snip above, though I don't think that's
relevant to not seeing a refreshed reader view of new search results -- unless
it's actually a wl-copy/paste bug somehow.

The latter has a man page, but no --version option, though debian's -"dpgk -l 'wl-*'"
found it [Version: 1.0.0-1], in case it plays any role.

On debian, -"apt show wl-clipboard" should tell the package details for you if needed.
In my case it's re-packaged for pureos, so it'd be good to test reader view  on other browsers and distributions.

Anyway, raingloom got me thinking more about reader mode, and I think he
would appreciate attention to that. I'll try to Cc: him :)

> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.
> 
> Best Regards,
> Danjela

HTH
-- 
Regards,
Bengt Richter

Re: [Guix Website] A Search Page for Packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

Great start, thanks for your work!

A few suggestions:

- We would need some sort of paging.  With JavaScript on, we could have
infinite pages.

- I would display the synopsis first and optionally the description.

- The queries seem a bit too slow, maybe there is a performance issue
that could be fixed.

Good luck!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Hi Pierre,
> We would need some sort of paging.  With JavaScript on, we could have
infinite pages.
I think pagination is available on this page: https://guix.gnu.org/packages/,
which I believe could be used if the search functionality is incorporated
there.

> I would display the synopsis first and optionally the description.
I used the same format as this page: https://guix.gnu.org/packages/, so I
guess the alteration you mentioned should be discussed with the
person/people who designed it. :)

> The queries seem a bit too slow, maybe there is a performance issue
that could be fixed.
Again, I have cc-ed Chris who could give a better response on that. :)

Thank you for the feedback,

Danjela


On Thu, 9 Jul 2020 at 08:17, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Great start, thanks for your work!
>
> A few suggestions:
>
> - We would need some sort of paging.  With JavaScript on, we could have
> infinite pages.
>
> - I would display the synopsis first and optionally the description.
>
> - The queries seem a bit too slow, maybe there is a performance issue
> that could be fixed.
>
> Good luck!
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
>

[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

Re: [Guix Website] A Search Page for Packages

[Hartmut Goebel]

Hi,

Am 08.07.20 um 20:49 schrieb Daniela Lura:
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Thanks for working on this.

I tried it briefly with a simple package name, and it took about 5
seconds to display the result. I'm curious why it takes so long, since
all relevant data should be available statically.

> I'd like to know if you find the page useful so that I can hopefully
> start working on incorporating the search functionality into this
> page: https://guix.gnu.org/packages/.

May I suggest to take the change and remove the package version from the
URL path. Thus http://guix-website-test.cbaines.net/packages/gcc-objc
would simply show all versions of "gcc-objc".


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [Guix Website] A Search Page for Packages

[zimoun]

Dear Danjela,

On Wed, 08 Jul 2020 at 20:49, Daniela Lura <danielaluraa@gmail.com> wrote:

> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Really cool!  Thank you for working on this.


> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.

For example http://issues.guix.gnu.org/ have a "hint" and "many more"
link for the query language.

It is a matter of taste, but I prefer a list:

    name version synopsis

alternating color for each line (light gray and blank for example).  And
then click to have more info as description, location, license, etc.

All the best,
simon

Re: [Guix Website] A Search Page for Packages

[YP]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Hello,

I always wondering why it was not integrated on guix.gnu.org, but this
functionality is already available at https://hpc.guix.info/browse and
seems to have many functionalities asked in comments :)

Good luck,

Regards,

On 08/07/2020 20:49, Daniela Lura wrote:
> Hello everyone, 
>
> This is Danjela, the Outreachy intern at the Guix Data Service. 
>
> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html,
> my mentor, Christopher Baines suggested me to write a script that
> serves a search page for packages using the search functionality
> within the Guix Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100. 
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>
> I'd like to know if you find the page useful so that I can hopefully
> start working on incorporating the search functionality into this
> page: https://guix.gnu.org/packages/. 
>
> Best Regards,
> Danjela

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 2497 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hello,

YP <yoann_mac_donald@hotmail.com> skribis:

> I always wondering why it was not integrated on guix.gnu.org, but this
> functionality is already available at https://hpc.guix.info/browse and
> seems to have many functionalities asked in comments :)

As discussed in the thread Danjela mentioned, the search feature at
hpc.guix.info requires JavaScript, and that’s the main reason why it’s
not available on guix.gnu.org: we wanted to have a JS-free option, or at
least “progressive enhancement” (JS-free for those who want it, and
JS-heavy for others).

Ludo’.

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Heya Danjela,

Daniela Lura <danielaluraa@gmail.com> skribis:

> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Cool!  To be clear, guix-website-test actually makes a search query on a
Data Service instance, right?  I guess that explains part of the
slowness, but maybe we could tweak nginx to keep search results in cache
for a few hours.

Florian raised an important issue regarding the possibility of denial of
service.  Ideas on how to mitigate that, either in the Data Service, or
nginx at guix.gnu.org?

> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.

I’d be happy to see that happen!  We would need the changes against
guix-artwork.git as well as (maybe) changes to the nginx config in
maintenance.git.  How should we proceed?

Thanks!

Ludo’.

Re: [Guix Website] A Search Page for Packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Heya Danjela,
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
>> Taking into consideration the suggestion made in this thread:
>> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
>> mentor, Christopher Baines suggested me to write a script that serves a
>> search page for packages using the search functionality within the Guix
>> Data Service,
>> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
>> .
>>
>> The prototype page can be accessed through a test version of the Guix
>> website that Chris deployed:
>> http://guix-website-test.cbaines.net/packages/search
>
> Cool!  To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?  I guess that explains part of the
> slowness, but maybe we could tweak nginx to keep search results in cache
> for a few hours.
>
> Florian raised an important issue regarding the possibility of denial of
> service.  Ideas on how to mitigate that, either in the Data Service, or
> nginx at guix.gnu.org?

On this point specifically, I've used the rate limiting functionality in
NGinx before, and that's probably a low cost approach to guarding
against this. It's not going to stop a DDOS, but might help avoid lots
of requests from one IP address causing load issues.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: [Guix Website] A Search Page for Packages

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Heya Danjela,
>>
>> Daniela Lura <danielaluraa@gmail.com> skribis:
>>
>>> Taking into consideration the suggestion made in this thread:
>>> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
>>> mentor, Christopher Baines suggested me to write a script that serves a
>>> search page for packages using the search functionality within the Guix
>>> Data Service,
>>> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
>>> .
>>>
>>> The prototype page can be accessed through a test version of the Guix
>>> website that Chris deployed:
>>> http://guix-website-test.cbaines.net/packages/search
>>
>> Cool!  To be clear, guix-website-test actually makes a search query on a
>> Data Service instance, right?  I guess that explains part of the
>> slowness, but maybe we could tweak nginx to keep search results in cache
>> for a few hours.
>>
>> Florian raised an important issue regarding the possibility of denial of
>> service.  Ideas on how to mitigate that, either in the Data Service, or
>> nginx at guix.gnu.org?
>
> On this point specifically, I've used the rate limiting functionality in
> NGinx before, and that's probably a low cost approach to guarding
> against this. It's not going to stop a DDOS, but might help avoid lots
> of requests from one IP address causing load issues.

We also have Varnish, a web caching proxy that excels at tasks like
this, and is used by some CDNs such as Fastly.  I'm happy to help write
a configuration if that is desired.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

>> Florian raised an important issue regarding the possibility of denial of
>> service.  Ideas on how to mitigate that, either in the Data Service, or
>> nginx at guix.gnu.org?
>
> On this point specifically, I've used the rate limiting functionality in
> NGinx before, and that's probably a low cost approach to guarding
> against this. It's not going to stop a DDOS, but might help avoid lots
> of requests from one IP address causing load issues.

Sounds good, we can take that route.

On berlin we’ll have to keep using nginx I guess (there’s too much
config in there), but perhaps Varnish is an option for the Data Service.

Ludo’.

Re: [Guix Website] A Search Page for Packages

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> On berlin we’ll have to keep using nginx I guess (there’s too much
> config in there), but perhaps Varnish is an option for the Data Service.

To be clear, Varnish is not a replacement for nginx (it does not even
speak HTTPS), the typical deployment is nginx -> Varnish -> backend.  It
introduces another "hop" though.

I'm not sure if it's worth the added complexity here, just thought I'd
mention it.  It's a programming language for caching/proxying logic.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

Hello Guix :),
Sorry for the late followup :/

Hope your summer is going well!

I'm sending this email to follow up on the search functionality for
packages.
With Chris's help I managed to make some changes that make the query for
searching packages within the Guix Data Service faster.
I don't know if the search is fast enough to be included on the Guix
website, but I thought it would be good to share.

Search packages page in the test version of the website that Chris
deployed: guix-website-test.cbaines.net/packages/search

If you'd want to have a look at the code changes:
https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
this is the commit where the changes start.

Looking forward to your feedback!

Best regards,
Danjela


On Tue, 14 Jul 2020 at 21:47, Marius Bakke <marius@gnu.org> wrote:

> Ludovic Courtès <ludo@gnu.org> writes:
>
> > On berlin we’ll have to keep using nginx I guess (there’s too much
> > config in there), but perhaps Varnish is an option for the Data Service.
>
> To be clear, Varnish is not a replacement for nginx (it does not even
> speak HTTPS), the typical deployment is nginx -> Varnish -> backend.  It
> introduces another "hop" though.
>
> I'm not sure if it's worth the added complexity here, just thought I'd
> mention it.  It's a programming language for caching/proxying logic.
>

[-- Attachment #2: Type: text/html, Size: 2273 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi Danjela!

Daniela Lura <danielaluraa@gmail.com> skribis:

> I'm sending this email to follow up on the search functionality for
> packages.
> With Chris's help I managed to make some changes that make the query for
> searching packages within the Guix Data Service faster.
> I don't know if the search is fast enough to be included on the Guix
> website, but I thought it would be good to share.
>
> Search packages page in the test version of the website that Chris
> deployed: guix-website-test.cbaines.net/packages/search

It seems to work well and it’s pretty fast!

> If you'd want to have a look at the code changes:
> https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
> this is the commit where the changes start.

Nice.

What would it take to integrate it on the web site?  That’d be really
cool.  :-)

Thanks for following up, great work!

Ludo’.

Re: [Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

Hi Ludo,

Thank you for your email!

> What would it take to integrate it on the web site?

I will start working on a patch some time soon. The beginning of school is
just around the corner so I'll try to work on the patch before all the
school related turmoil starts. :)

Best,
Danjela

On Mon, Sep 7, 2020, 12:21 Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Danjela!
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
> > I'm sending this email to follow up on the search functionality for
> > packages.
> > With Chris's help I managed to make some changes that make the query for
> > searching packages within the Guix Data Service faster.
> > I don't know if the search is fast enough to be included on the Guix
> > website, but I thought it would be good to share.
> >
> > Search packages page in the test version of the website that Chris
> > deployed: guix-website-test.cbaines.net/packages/search
>
> It seems to work well and it’s pretty fast!
>
> > If you'd want to have a look at the code changes:
> >
> https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
> > this is the commit where the changes start.
>
> Nice.
>
> What would it take to integrate it on the web site?  That’d be really
> cool.  :-)
>
> Thanks for following up, great work!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2603 bytes --]

Re: [Guix Website] A Search Page for Packages

[Hartmut Goebel]

Am 06.09.20 um 16:19 schrieb Daniela Lura:
> Search packages page in the test version of the website that Chris
> deployed: guix-website-test.cbaines.net/packages/search
> <http://guix-website-test.cbaines.net/packages/search>

Great!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [Guix Website] A Search Page for Packages

[Daniela Lura]

[-- Attachment #1: Type: text/plain, Size: 2208 bytes --]

Hi Ludo,

> To be clear, guix-website-test actually makes a search query on a
Data Service instance, right?
That's right. I'm currently learning how to read and understand query
plans:
https://www.postgresql.org/docs/10/using-explain.html#USING-EXPLAIN-ANALYZE
so that the query can hopefully be optimized.

> Ideas on how to mitigate that, either in the Data Service, or
nginx at guix.gnu.org?
To be honest, I'm not that familiar with nginx, so my opinion on that
wouldn't be that valuable.

Thank you for the feedback!

Best,
Danjela

On Mon, 13 Jul 2020 at 16:00, Ludovic Courtès <ludo@gnu.org> wrote:

> Heya Danjela,
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
> > Taking into consideration the suggestion made in this thread:
> > https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> > mentor, Christopher Baines suggested me to write a script that serves a
> > search page for packages using the search functionality within the Guix
> > Data Service,
> >
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> > .
> >
> > The prototype page can be accessed through a test version of the Guix
> > website that Chris deployed:
> > http://guix-website-test.cbaines.net/packages/search
>
> Cool!  To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?  I guess that explains part of the
> slowness, but maybe we could tweak nginx to keep search results in cache
> for a few hours.
>
> Florian raised an important issue regarding the possibility of denial of
> service.  Ideas on how to mitigate that, either in the Data Service, or
> nginx at guix.gnu.org?
>
> > I'd like to know if you find the page useful so that I can hopefully
> start
> > working on incorporating the search functionality into this page:
> > https://guix.gnu.org/packages/.
>
> I’d be happy to see that happen!  We would need the changes against
> guix-artwork.git as well as (maybe) changes to the nginx config in
> maintenance.git.  How should we proceed?
>
> Thanks!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3643 bytes --]

Re: [Guix Website] A Search Page for Packages

[Ludovic Courtès]

Hi,

Daniela Lura <danielaluraa@gmail.com> skribis:

>> To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?
> That's right. I'm currently learning how to read and understand query plans:
> https://www.postgresql.org/docs/10/using-explain.html#USING-EXPLAIN-ANALYZE so that the query can hopefully be optimized.

I see.  I guess the extra connection from guix.gnu.org to the Data
Service means queries will always be a bit expensive, but maybe that’s
OK with optimized queries and caching.

Thanks,
Ludo’.