unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* [Guix Website] A Search Page for Packages
@ 2020-07-08 18:49 Daniela Lura
  2020-07-08 20:31 ` Cook, Malcolm
                   ` (6 more replies)
  0 siblings, 7 replies; 22+ messages in thread
From: Daniela Lura @ 2020-07-08 18:49 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

Hello everyone,

This is Danjela, the Outreachy intern at the Guix Data Service.

Taking into consideration the suggestion made in this thread:
https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
mentor, Christopher Baines suggested me to write a script that serves a
search page for packages using the search functionality within the Guix
Data Service,
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
.

The prototype page can be accessed through a test version of the Guix
website that Chris deployed:
http://guix-website-test.cbaines.net/packages/search

I'd like to know if you find the page useful so that I can hopefully start
working on incorporating the search functionality into this page:
https://guix.gnu.org/packages/.

Best Regards,
Danjela

[-- Attachment #2: Type: text/html, Size: 1491 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
@ 2020-07-08 20:31 ` Cook, Malcolm
  2020-07-08 21:00   ` pelzflorian (Florian Pelz)
  2020-07-10 10:15   ` Daniela Lura
  2020-07-09  0:53 ` Bengt Richter
                   ` (5 subsequent siblings)
  6 siblings, 2 replies; 22+ messages in thread
From: Cook, Malcolm @ 2020-07-08 20:31 UTC (permalink / raw)
  To: Daniela Lura, guix-devel

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

A few quick thoughts –


  *   There should be a link documenting the search language.  Are there Booleans allowed?  Is “stemming” automatic?  Do quote characters do anything?  Is proximity searching available?  Regular expressions?  What columns are searched?  License?  Version? Author?
  *   Immediate feedback of hits as you type is “nice”.  IMO suggested completion of search times is nicer.
  *   Browsing alphabetically by picking a letter provide little to no value
  *   Full detail of hits in columnar format is preferred over “cards”

Still lurking myself but loving the improvements….

Go Guix,

~Malcolm


From: Guix-devel <guix-devel-bounces+mec=stowers.org@gnu.org> on behalf of Daniela Lura <danielaluraa@gmail.com>
Date: Wednesday, July 8, 2020 at 1:50 PM
To: "guix-devel@gnu.org" <guix-devel@gnu.org>
Subject: [Guix Website] A Search Page for Packages

CAUTION: This email was received from an External Source

Hello everyone,

This is Danjela, the Outreachy intern at the Guix Data Service.

Taking into consideration the suggestion made in this thread:
https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my mentor, Christopher Baines suggested me to write a script that serves a search page for packages using the search functionality within the Guix Data Service, https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100.

The prototype page can be accessed through a test version of the Guix website that Chris deployed: http://guix-website-test.cbaines.net/packages/search

I'd like to know if you find the page useful so that I can hopefully start working on incorporating the search functionality into this page: https://guix.gnu.org/packages/.

Best Regards,
Danjela

[-- Attachment #2: Type: text/html, Size: 9409 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 20:31 ` Cook, Malcolm
@ 2020-07-08 21:00   ` pelzflorian (Florian Pelz)
  2020-07-10 10:15   ` Daniela Lura
  1 sibling, 0 replies; 22+ messages in thread
From: pelzflorian (Florian Pelz) @ 2020-07-08 21:00 UTC (permalink / raw)
  To: Cook, Malcolm; +Cc: Daniela Lura, guix-devel

Yes, this kind of search is much better.  Good work!

I wonder, can bad people abuse the search for denial-of-service or is
the number of searches limited?

On Wed, Jul 08, 2020 at 08:31:37PM +0000, Cook, Malcolm wrote:
>   *   Immediate feedback of hits as you type is “nice”.  IMO suggested completion of search times is nicer.

This has two sides.  I find it great that search works without
JavaScript and think users should not normally be required to enable
privacy-killing JavaScript.  However, search-as-you-type indeed is a
nice feature for the JS-enabled users.  I do not know how to
reasonably implement it though (and in what programming language).
Also the improvement of search-as-you-type vs. the current proposal is
not big.

Regards,
Florian


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
  2020-07-08 20:31 ` Cook, Malcolm
@ 2020-07-09  0:53 ` Bengt Richter
  2020-07-09  6:17 ` Pierre Neidhardt
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 22+ messages in thread
From: Bengt Richter @ 2020-07-09  0:53 UTC (permalink / raw)
  To: Daniela Lura; +Cc: guix-devel

Hi Daniela,

On +2020-07-08 20:49:35 +0200, Daniela Lura wrote:
> Hello everyone,
> 
> This is Danjela, the Outreachy intern at the Guix Data Service.
> 
> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
> 
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>

This looks useful :)

I did notice that clicking firefox-esr's[1] reader-view button (or equivalently
hitting Ctl-Alt-r [yes, lower case]) niether shows the latest search result
(I searched for -"sha" (BTW, my convention: prefixed '-' on double-quoted string means
without the quotes, ok?)).

[1] Mozilla Firefox 68.10.0esr per -"firefox --version"

It seemed to go back to a locked-in one-time-memoized value that persists
through other searches and even exiting the browser and restarting it,
and even after hitting its reload button.

All it keeps showing in reader mode is
--8<---------------cut here---------------start------------->8---
guix-website-test.cbaines.net
Packages
1 minute
ghc-cryptohash-sha1 0.11.100.1

This Haskell package provides an incremental and one-pass, pure API to the SHA-1 hash algorithm (https://en.wikipedia.org/wiki/SHA-1), including HMAC support (https://en.wikipedia.org/wiki/HMAC), with performance close to the fastest implementations available in other languages. The implementation is made…
ghc-cryptohash-sha256 0.11.101.0

This Haskell package provides an incremental and one-pass, pure API to the SHA-256 cryptographic hash algorithm (https://en.wikipedia.org/wiki/SHA-2), with performance close to the fastest implementations available in other languages. The implementation is made in C…
--8<---------------cut here---------------end--------------->8---
which I think is the result of my first search of all -- which was for
-"sha256" (sans quotes again, no more explaining that :).

Actually, on the reader page, there is visually a blank line above the
-"ghc-cryptohash-sha256 0.11.101.0" line in the snip above, though I don't think that's
relevant to not seeing a refreshed reader view of new search results -- unless
it's actually a wl-copy/paste bug somehow.

The latter has a man page, but no --version option, though debian's -"dpgk -l 'wl-*'"
found it [Version: 1.0.0-1], in case it plays any role.

On debian, -"apt show wl-clipboard" should tell the package details for you if needed.
In my case it's re-packaged for pureos, so it'd be good to test reader view  on other browsers and distributions.

Anyway, raingloom got me thinking more about reader mode, and I think he
would appreciate attention to that. I'll try to Cc: him :)

> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.
> 
> Best Regards,
> Danjela

HTH
-- 
Regards,
Bengt Richter


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
  2020-07-08 20:31 ` Cook, Malcolm
  2020-07-09  0:53 ` Bengt Richter
@ 2020-07-09  6:17 ` Pierre Neidhardt
  2020-07-10 10:26   ` Daniela Lura
  2020-07-09  7:47 ` Hartmut Goebel
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 22+ messages in thread
From: Pierre Neidhardt @ 2020-07-09  6:17 UTC (permalink / raw)
  To: Daniela Lura, guix-devel

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

Great start, thanks for your work!

A few suggestions:

- We would need some sort of paging.  With JavaScript on, we could have
infinite pages.

- I would display the synopsis first and optionally the description.

- The queries seem a bit too slow, maybe there is a performance issue
that could be fixed.

Good luck!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
                   ` (2 preceding siblings ...)
  2020-07-09  6:17 ` Pierre Neidhardt
@ 2020-07-09  7:47 ` Hartmut Goebel
  2020-07-09  8:50 ` zimoun
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 22+ messages in thread
From: Hartmut Goebel @ 2020-07-09  7:47 UTC (permalink / raw)
  To: guix-devel

Hi,

Am 08.07.20 um 20:49 schrieb Daniela Lura:
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Thanks for working on this.

I tried it briefly with a simple package name, and it took about 5
seconds to display the result. I'm curious why it takes so long, since
all relevant data should be available statically.

> I'd like to know if you find the page useful so that I can hopefully
> start working on incorporating the search functionality into this
> page: https://guix.gnu.org/packages/.

May I suggest to take the change and remove the package version from the
URL path. Thus http://guix-website-test.cbaines.net/packages/gcc-objc
would simply show all versions of "gcc-objc".


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
                   ` (3 preceding siblings ...)
  2020-07-09  7:47 ` Hartmut Goebel
@ 2020-07-09  8:50 ` zimoun
  2020-07-09  9:32 ` YP
  2020-07-13 14:00 ` Ludovic Courtès
  6 siblings, 0 replies; 22+ messages in thread
From: zimoun @ 2020-07-09  8:50 UTC (permalink / raw)
  To: Daniela Lura, guix-devel

Dear Danjela,

On Wed, 08 Jul 2020 at 20:49, Daniela Lura <danielaluraa@gmail.com> wrote:

> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Really cool!  Thank you for working on this.


> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.

For example http://issues.guix.gnu.org/ have a "hint" and "many more"
link for the query language.

It is a matter of taste, but I prefer a list:

    name version synopsis

alternating color for each line (light gray and blank for example).  And
then click to have more info as description, location, license, etc.

All the best,
simon




^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
                   ` (4 preceding siblings ...)
  2020-07-09  8:50 ` zimoun
@ 2020-07-09  9:32 ` YP
  2020-07-13 13:56   ` Ludovic Courtès
  2020-07-13 14:00 ` Ludovic Courtès
  6 siblings, 1 reply; 22+ messages in thread
From: YP @ 2020-07-09  9:32 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Hello,

I always wondering why it was not integrated on guix.gnu.org, but this
functionality is already available at https://hpc.guix.info/browse and
seems to have many functionalities asked in comments :)

Good luck,

Regards,

On 08/07/2020 20:49, Daniela Lura wrote:
> Hello everyone, 
>
> This is Danjela, the Outreachy intern at the Guix Data Service. 
>
> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html,
> my mentor, Christopher Baines suggested me to write a script that
> serves a search page for packages using the search functionality
> within the Guix Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100. 
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>
> I'd like to know if you find the page useful so that I can hopefully
> start working on incorporating the search functionality into this
> page: https://guix.gnu.org/packages/. 
>
> Best Regards,
> Danjela

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 2497 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 20:31 ` Cook, Malcolm
  2020-07-08 21:00   ` pelzflorian (Florian Pelz)
@ 2020-07-10 10:15   ` Daniela Lura
  1 sibling, 0 replies; 22+ messages in thread
From: Daniela Lura @ 2020-07-10 10:15 UTC (permalink / raw)
  To: Cook, Malcolm, guix-devel

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

Hi Malcolm,

As noted in this page:
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages,
you can search by package name or synopsis.
I think Chris, whom I have cc-ed, is the most suitable person to talk about
the required functionalities of the search, because he wrote the query that
provides the search functionality within the Guix Data Service, which is
used in the prototype.
-
- > Immediate feedback of hits as you type is “nice”.  IMO suggested
completion of search times is nicer.
-
- Seems sensible, I will look into it even though I'm not yet sure how to
implement it. :)
-
- > Full detail of hits in columnar format is preferred over “cards”
- In order for the prototype page to visually fit the design here:
https://guix.gnu.org/packages/ I used cards.

Best,
Danjela

On Wed, 8 Jul 2020 at 22:31, Cook, Malcolm <MEC@stowers.org> wrote:

> A few quick thoughts –
>
>
>
>    - There should be a link documenting the search language.  Are there
>    Booleans allowed?  Is “stemming” automatic?  Do quote characters do
>    anything?  Is proximity searching available?  Regular expressions?  What
>    columns are searched?  License?  Version? Author?
>    - Immediate feedback of hits as you type is “nice”.  IMO suggested
>    completion of search times is nicer.
>    - Browsing alphabetically by picking a letter provide little to no
>    value
>    - Full detail of hits in columnar format is preferred over “cards”
>
>
>
> Still lurking myself but loving the improvements….
>
>
>
> Go Guix,
>
>
>
> ~Malcolm
>
>
>
>
>
> *From: *Guix-devel <guix-devel-bounces+mec=stowers.org@gnu.org> on behalf
> of Daniela Lura <danielaluraa@gmail.com>
> *Date: *Wednesday, July 8, 2020 at 1:50 PM
> *To: *"guix-devel@gnu.org" <guix-devel@gnu.org>
> *Subject: *[Guix Website] A Search Page for Packages
>
>
>
> CAUTION: This email was received from an External Source
>
>
>
> Hello everyone,
>
>
>
> This is Danjela, the Outreachy intern at the Guix Data Service.
>
>
>
> Taking into consideration the suggestion made in this thread:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
>
>
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search
>
>
>
> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.
>
>
>
> Best Regards,
>
> Danjela
>

[-- Attachment #2: Type: text/html, Size: 6987 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-09  6:17 ` Pierre Neidhardt
@ 2020-07-10 10:26   ` Daniela Lura
  0 siblings, 0 replies; 22+ messages in thread
From: Daniela Lura @ 2020-07-10 10:26 UTC (permalink / raw)
  To: Pierre Neidhardt, guix-devel

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Hi Pierre,
> We would need some sort of paging.  With JavaScript on, we could have
infinite pages.
I think pagination is available on this page: https://guix.gnu.org/packages/,
which I believe could be used if the search functionality is incorporated
there.

> I would display the synopsis first and optionally the description.
I used the same format as this page: https://guix.gnu.org/packages/, so I
guess the alteration you mentioned should be discussed with the
person/people who designed it. :)

> The queries seem a bit too slow, maybe there is a performance issue
that could be fixed.
Again, I have cc-ed Chris who could give a better response on that. :)

Thank you for the feedback,

Danjela


On Thu, 9 Jul 2020 at 08:17, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Great start, thanks for your work!
>
> A few suggestions:
>
> - We would need some sort of paging.  With JavaScript on, we could have
> infinite pages.
>
> - I would display the synopsis first and optionally the description.
>
> - The queries seem a bit too slow, maybe there is a performance issue
> that could be fixed.
>
> Good luck!
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
>

[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-09  9:32 ` YP
@ 2020-07-13 13:56   ` Ludovic Courtès
  0 siblings, 0 replies; 22+ messages in thread
From: Ludovic Courtès @ 2020-07-13 13:56 UTC (permalink / raw)
  To: YP; +Cc: guix-devel

Hello,

YP <yoann_mac_donald@hotmail.com> skribis:

> I always wondering why it was not integrated on guix.gnu.org, but this
> functionality is already available at https://hpc.guix.info/browse and
> seems to have many functionalities asked in comments :)

As discussed in the thread Danjela mentioned, the search feature at
hpc.guix.info requires JavaScript, and that’s the main reason why it’s
not available on guix.gnu.org: we wanted to have a JS-free option, or at
least “progressive enhancement” (JS-free for those who want it, and
JS-heavy for others).

Ludo’.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
                   ` (5 preceding siblings ...)
  2020-07-09  9:32 ` YP
@ 2020-07-13 14:00 ` Ludovic Courtès
  2020-07-13 20:10   ` Christopher Baines
  2020-07-14  9:19   ` Daniela Lura
  6 siblings, 2 replies; 22+ messages in thread
From: Ludovic Courtès @ 2020-07-13 14:00 UTC (permalink / raw)
  To: Daniela Lura; +Cc: guix-devel

Heya Danjela,

Daniela Lura <danielaluraa@gmail.com> skribis:

> Taking into consideration the suggestion made in this thread:
> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> mentor, Christopher Baines suggested me to write a script that serves a
> search page for packages using the search functionality within the Guix
> Data Service,
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> .
>
> The prototype page can be accessed through a test version of the Guix
> website that Chris deployed:
> http://guix-website-test.cbaines.net/packages/search

Cool!  To be clear, guix-website-test actually makes a search query on a
Data Service instance, right?  I guess that explains part of the
slowness, but maybe we could tweak nginx to keep search results in cache
for a few hours.

Florian raised an important issue regarding the possibility of denial of
service.  Ideas on how to mitigate that, either in the Data Service, or
nginx at guix.gnu.org?

> I'd like to know if you find the page useful so that I can hopefully start
> working on incorporating the search functionality into this page:
> https://guix.gnu.org/packages/.

I’d be happy to see that happen!  We would need the changes against
guix-artwork.git as well as (maybe) changes to the nginx config in
maintenance.git.  How should we proceed?

Thanks!

Ludo’.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-13 14:00 ` Ludovic Courtès
@ 2020-07-13 20:10   ` Christopher Baines
  2020-07-13 22:04     ` Marius Bakke
  2020-07-14 13:24     ` Ludovic Courtès
  2020-07-14  9:19   ` Daniela Lura
  1 sibling, 2 replies; 22+ messages in thread
From: Christopher Baines @ 2020-07-13 20:10 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Daniela Lura, guix-devel

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Heya Danjela,
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
>> Taking into consideration the suggestion made in this thread:
>> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
>> mentor, Christopher Baines suggested me to write a script that serves a
>> search page for packages using the search functionality within the Guix
>> Data Service,
>> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
>> .
>>
>> The prototype page can be accessed through a test version of the Guix
>> website that Chris deployed:
>> http://guix-website-test.cbaines.net/packages/search
>
> Cool!  To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?  I guess that explains part of the
> slowness, but maybe we could tweak nginx to keep search results in cache
> for a few hours.
>
> Florian raised an important issue regarding the possibility of denial of
> service.  Ideas on how to mitigate that, either in the Data Service, or
> nginx at guix.gnu.org?

On this point specifically, I've used the rate limiting functionality in
NGinx before, and that's probably a low cost approach to guarding
against this. It's not going to stop a DDOS, but might help avoid lots
of requests from one IP address causing load issues.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-13 20:10   ` Christopher Baines
@ 2020-07-13 22:04     ` Marius Bakke
  2020-07-14 13:24     ` Ludovic Courtès
  1 sibling, 0 replies; 22+ messages in thread
From: Marius Bakke @ 2020-07-13 22:04 UTC (permalink / raw)
  To: Christopher Baines, Ludovic Courtès; +Cc: Daniela Lura, guix-devel

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Heya Danjela,
>>
>> Daniela Lura <danielaluraa@gmail.com> skribis:
>>
>>> Taking into consideration the suggestion made in this thread:
>>> https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
>>> mentor, Christopher Baines suggested me to write a script that serves a
>>> search page for packages using the search functionality within the Guix
>>> Data Service,
>>> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
>>> .
>>>
>>> The prototype page can be accessed through a test version of the Guix
>>> website that Chris deployed:
>>> http://guix-website-test.cbaines.net/packages/search
>>
>> Cool!  To be clear, guix-website-test actually makes a search query on a
>> Data Service instance, right?  I guess that explains part of the
>> slowness, but maybe we could tweak nginx to keep search results in cache
>> for a few hours.
>>
>> Florian raised an important issue regarding the possibility of denial of
>> service.  Ideas on how to mitigate that, either in the Data Service, or
>> nginx at guix.gnu.org?
>
> On this point specifically, I've used the rate limiting functionality in
> NGinx before, and that's probably a low cost approach to guarding
> against this. It's not going to stop a DDOS, but might help avoid lots
> of requests from one IP address causing load issues.

We also have Varnish, a web caching proxy that excels at tasks like
this, and is used by some CDNs such as Fastly.  I'm happy to help write
a configuration if that is desired.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-13 14:00 ` Ludovic Courtès
  2020-07-13 20:10   ` Christopher Baines
@ 2020-07-14  9:19   ` Daniela Lura
  2020-07-14 13:23     ` Ludovic Courtès
  1 sibling, 1 reply; 22+ messages in thread
From: Daniela Lura @ 2020-07-14  9:19 UTC (permalink / raw)
  To: Ludovic Courtès, guix-devel

[-- Attachment #1: Type: text/plain, Size: 2208 bytes --]

Hi Ludo,

> To be clear, guix-website-test actually makes a search query on a
Data Service instance, right?
That's right. I'm currently learning how to read and understand query
plans:
https://www.postgresql.org/docs/10/using-explain.html#USING-EXPLAIN-ANALYZE
so that the query can hopefully be optimized.

> Ideas on how to mitigate that, either in the Data Service, or
nginx at guix.gnu.org?
To be honest, I'm not that familiar with nginx, so my opinion on that
wouldn't be that valuable.

Thank you for the feedback!

Best,
Danjela

On Mon, 13 Jul 2020 at 16:00, Ludovic Courtès <ludo@gnu.org> wrote:

> Heya Danjela,
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
> > Taking into consideration the suggestion made in this thread:
> > https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00096.html, my
> > mentor, Christopher Baines suggested me to write a script that serves a
> > search page for packages using the search functionality within the Guix
> > Data Service,
> >
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages?search_query=git&field=version&field=synopsis&after_name=&limit_results=100
> > .
> >
> > The prototype page can be accessed through a test version of the Guix
> > website that Chris deployed:
> > http://guix-website-test.cbaines.net/packages/search
>
> Cool!  To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?  I guess that explains part of the
> slowness, but maybe we could tweak nginx to keep search results in cache
> for a few hours.
>
> Florian raised an important issue regarding the possibility of denial of
> service.  Ideas on how to mitigate that, either in the Data Service, or
> nginx at guix.gnu.org?
>
> > I'd like to know if you find the page useful so that I can hopefully
> start
> > working on incorporating the search functionality into this page:
> > https://guix.gnu.org/packages/.
>
> I’d be happy to see that happen!  We would need the changes against
> guix-artwork.git as well as (maybe) changes to the nginx config in
> maintenance.git.  How should we proceed?
>
> Thanks!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3643 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-14  9:19   ` Daniela Lura
@ 2020-07-14 13:23     ` Ludovic Courtès
  0 siblings, 0 replies; 22+ messages in thread
From: Ludovic Courtès @ 2020-07-14 13:23 UTC (permalink / raw)
  To: Daniela Lura; +Cc: guix-devel

Hi,

Daniela Lura <danielaluraa@gmail.com> skribis:

>> To be clear, guix-website-test actually makes a search query on a
> Data Service instance, right?
> That's right. I'm currently learning how to read and understand query plans:
> https://www.postgresql.org/docs/10/using-explain.html#USING-EXPLAIN-ANALYZE so that the query can hopefully be optimized.

I see.  I guess the extra connection from guix.gnu.org to the Data
Service means queries will always be a bit expensive, but maybe that’s
OK with optimized queries and caching.

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-13 20:10   ` Christopher Baines
  2020-07-13 22:04     ` Marius Bakke
@ 2020-07-14 13:24     ` Ludovic Courtès
  2020-07-14 19:47       ` Marius Bakke
  1 sibling, 1 reply; 22+ messages in thread
From: Ludovic Courtès @ 2020-07-14 13:24 UTC (permalink / raw)
  To: Christopher Baines; +Cc: Daniela Lura, guix-devel

Hi,

Christopher Baines <mail@cbaines.net> skribis:

>> Florian raised an important issue regarding the possibility of denial of
>> service.  Ideas on how to mitigate that, either in the Data Service, or
>> nginx at guix.gnu.org?
>
> On this point specifically, I've used the rate limiting functionality in
> NGinx before, and that's probably a low cost approach to guarding
> against this. It's not going to stop a DDOS, but might help avoid lots
> of requests from one IP address causing load issues.

Sounds good, we can take that route.

On berlin we’ll have to keep using nginx I guess (there’s too much
config in there), but perhaps Varnish is an option for the Data Service.

Ludo’.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-14 13:24     ` Ludovic Courtès
@ 2020-07-14 19:47       ` Marius Bakke
  2020-09-06 14:19         ` Daniela Lura
  0 siblings, 1 reply; 22+ messages in thread
From: Marius Bakke @ 2020-07-14 19:47 UTC (permalink / raw)
  To: Ludovic Courtès, Christopher Baines; +Cc: Daniela Lura, guix-devel

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> On berlin we’ll have to keep using nginx I guess (there’s too much
> config in there), but perhaps Varnish is an option for the Data Service.

To be clear, Varnish is not a replacement for nginx (it does not even
speak HTTPS), the typical deployment is nginx -> Varnish -> backend.  It
introduces another "hop" though.

I'm not sure if it's worth the added complexity here, just thought I'd
mention it.  It's a programming language for caching/proxying logic.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-07-14 19:47       ` Marius Bakke
@ 2020-09-06 14:19         ` Daniela Lura
  2020-09-07 10:20           ` Ludovic Courtès
  2020-09-08 16:38           ` Hartmut Goebel
  0 siblings, 2 replies; 22+ messages in thread
From: Daniela Lura @ 2020-09-06 14:19 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

Hello Guix :),
Sorry for the late followup :/

Hope your summer is going well!

I'm sending this email to follow up on the search functionality for
packages.
With Chris's help I managed to make some changes that make the query for
searching packages within the Guix Data Service faster.
I don't know if the search is fast enough to be included on the Guix
website, but I thought it would be good to share.

Search packages page in the test version of the website that Chris
deployed: guix-website-test.cbaines.net/packages/search

If you'd want to have a look at the code changes:
https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
this is the commit where the changes start.

Looking forward to your feedback!

Best regards,
Danjela


On Tue, 14 Jul 2020 at 21:47, Marius Bakke <marius@gnu.org> wrote:

> Ludovic Courtès <ludo@gnu.org> writes:
>
> > On berlin we’ll have to keep using nginx I guess (there’s too much
> > config in there), but perhaps Varnish is an option for the Data Service.
>
> To be clear, Varnish is not a replacement for nginx (it does not even
> speak HTTPS), the typical deployment is nginx -> Varnish -> backend.  It
> introduces another "hop" though.
>
> I'm not sure if it's worth the added complexity here, just thought I'd
> mention it.  It's a programming language for caching/proxying logic.
>

[-- Attachment #2: Type: text/html, Size: 2273 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-09-06 14:19         ` Daniela Lura
@ 2020-09-07 10:20           ` Ludovic Courtès
  2020-09-08 16:50             ` Daniela Lura
  2020-09-08 16:38           ` Hartmut Goebel
  1 sibling, 1 reply; 22+ messages in thread
From: Ludovic Courtès @ 2020-09-07 10:20 UTC (permalink / raw)
  To: Daniela Lura; +Cc: guix-devel

Hi Danjela!

Daniela Lura <danielaluraa@gmail.com> skribis:

> I'm sending this email to follow up on the search functionality for
> packages.
> With Chris's help I managed to make some changes that make the query for
> searching packages within the Guix Data Service faster.
> I don't know if the search is fast enough to be included on the Guix
> website, but I thought it would be good to share.
>
> Search packages page in the test version of the website that Chris
> deployed: guix-website-test.cbaines.net/packages/search

It seems to work well and it’s pretty fast!

> If you'd want to have a look at the code changes:
> https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
> this is the commit where the changes start.

Nice.

What would it take to integrate it on the web site?  That’d be really
cool.  :-)

Thanks for following up, great work!

Ludo’.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-09-06 14:19         ` Daniela Lura
  2020-09-07 10:20           ` Ludovic Courtès
@ 2020-09-08 16:38           ` Hartmut Goebel
  1 sibling, 0 replies; 22+ messages in thread
From: Hartmut Goebel @ 2020-09-08 16:38 UTC (permalink / raw)
  To: guix-devel

Am 06.09.20 um 16:19 schrieb Daniela Lura:
> Search packages page in the test version of the website that Chris
> deployed: guix-website-test.cbaines.net/packages/search
> <http://guix-website-test.cbaines.net/packages/search>

Great!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [Guix Website] A Search Page for Packages
  2020-09-07 10:20           ` Ludovic Courtès
@ 2020-09-08 16:50             ` Daniela Lura
  0 siblings, 0 replies; 22+ messages in thread
From: Daniela Lura @ 2020-09-08 16:50 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

Hi Ludo,

Thank you for your email!

> What would it take to integrate it on the web site?

I will start working on a patch some time soon. The beginning of school is
just around the corner so I'll try to work on the patch before all the
school related turmoil starts. :)

Best,
Danjela

On Mon, Sep 7, 2020, 12:21 Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Danjela!
>
> Daniela Lura <danielaluraa@gmail.com> skribis:
>
> > I'm sending this email to follow up on the search functionality for
> > packages.
> > With Chris's help I managed to make some changes that make the query for
> > searching packages within the Guix Data Service faster.
> > I don't know if the search is fast enough to be included on the Guix
> > website, but I thought it would be good to share.
> >
> > Search packages page in the test version of the website that Chris
> > deployed: guix-website-test.cbaines.net/packages/search
>
> It seems to work well and it’s pretty fast!
>
> > If you'd want to have a look at the code changes:
> >
> https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=ee613cdb305cc1e443135d311aead6c799902b8a
> > this is the commit where the changes start.
>
> Nice.
>
> What would it take to integrate it on the web site?  That’d be really
> cool.  :-)
>
> Thanks for following up, great work!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2603 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2020-09-08 16:50 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-08 18:49 [Guix Website] A Search Page for Packages Daniela Lura
2020-07-08 20:31 ` Cook, Malcolm
2020-07-08 21:00   ` pelzflorian (Florian Pelz)
2020-07-10 10:15   ` Daniela Lura
2020-07-09  0:53 ` Bengt Richter
2020-07-09  6:17 ` Pierre Neidhardt
2020-07-10 10:26   ` Daniela Lura
2020-07-09  7:47 ` Hartmut Goebel
2020-07-09  8:50 ` zimoun
2020-07-09  9:32 ` YP
2020-07-13 13:56   ` Ludovic Courtès
2020-07-13 14:00 ` Ludovic Courtès
2020-07-13 20:10   ` Christopher Baines
2020-07-13 22:04     ` Marius Bakke
2020-07-14 13:24     ` Ludovic Courtès
2020-07-14 19:47       ` Marius Bakke
2020-09-06 14:19         ` Daniela Lura
2020-09-07 10:20           ` Ludovic Courtès
2020-09-08 16:50             ` Daniela Lura
2020-09-08 16:38           ` Hartmut Goebel
2020-07-14  9:19   ` Daniela Lura
2020-07-14 13:23     ` Ludovic Courtès

unofficial mirror of guix-devel@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git