Re: [Outreachy] [Guix Data Service]: Identify the slow parts of process

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 2394 bytes --]


Canan Talayhan <canan.t.talayhan@gmail.com> writes:

>>From this I'm guessing the temp_package_metadata table has only one
>>row. My understanding is that this table would normally have as many
>>rows as packages in the revision of Guix being processed. It might not
>>be possible to reproduce the slowness of the query without more rows.
>
> I've inserted one row just as an example. As you've already said,
> the temp_package_metadata table should have as many rows
> as package_metadata.

"as many rows as packages in the revision of Guix being processed" is
only going to be similar to the number of rows in the package_metadata
table if there's only been one or a number of similar revisions
processed, since the package_metadata table has entires covering all
processed revisions.

> After populated the temp_package_metadata with 500 rows of
> package_metadata, the query takes a long time as we expected.

Great, being able to reproduce the problem in a way that makes trying
things out easy is a good step forward.

I'd pull on this thread further, now you've got a slow query, how can
you make it faster?

> I'm using Flame Graph to visualize the slow paths on the revision part.
> At first, I choose the slow one that I already know.
> However, I can't successfully trigger the slow query following the below step:
>
> * Run the **guix-data-service-process-job** under guix-data-service/scripts
> folder as standalone providing an existing revision on my local db.
>
> Am I on the right path for adding new jobs log to my local db?
>
> In addition, I've successfully generated simple Flame Graph using Linux perf.
> It visualizes only the data that was captured while I'm browsing on the
> Guix Data Service Page. Please find the svg file attached.

If this relates to the query involving the temp_package_metadata table,
I'd focus on analyzing the slow query you're able to execute manually,
rather than processing an entire revision.

If you do however want to add more unprocessed jobs to your local
database, then you can use the
guix-data-service-process-branch-updated-mbox script to do this. It
takes one argument, an mbox file (file containing a bunch of
emails). You can download files by month from here [1], and you'll
probably want the month or next month on from the latest revision your
local database knows about.

1: https://lists.gnu.org/archive/mbox/guix-commits/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]