From: Tassilo Horn <tsdh@gnu.org>
To: Eli Zaretskii <eliz@gnu.org>
Cc: toon@iotcl.com, emacs-devel@gnu.org,
Stefan Monnier <monnier@iro.umontreal.ca>,
agrambot@gmail.com, dgutov@yandex.ru
Subject: Re: [OFFTOPIC] size of issue tracker
Date: Mon, 13 May 2019 20:59:51 +0200 [thread overview]
Message-ID: <8736liuqmw.fsf@gnu.org> (raw)
In-Reply-To: <83tvdy9rpm.fsf@gnu.org> (Eli Zaretskii's message of "Mon, 13 May 2019 20:42:13 +0300")
Eli Zaretskii <eliz@gnu.org> writes:
>> > Why does Forge insist downloading the entire DB to the local
>> > machine? That sounds like something that wouldn't scale well.
>>
>> I don't know why it does so, but it sounds like a good idea to me, so
>> you can later use them when you're offline.
>
> If the DB is small enough, sure. It sounds like somehow it isn't the
> case with Forge and GitLab.
My database (~/.emacs.d/forge-database.sqlite) is currently 38 MB, and
it contains the contents of 4 different projects from which gtk with its
1900 issues and 900 MRs is by far the largest one. There's a repository
table from which you can join your way to the issues, merge/pull
requests, their comments, labels, and assignees.
That's the current schema if you are interested in details.
--8<---------------cut here---------------start------------->8---
sqlite> .schema
CREATE TABLE repository (class NOT NULL, id NOT NULL PRIMARY KEY, forge_id , forge , owner , name , apihost , githost , remote , sparse_p , created , updated , pushed , parent , description , homepage , default_branch , archived_p , fork_p , locked_p , mirror_p , private_p , issues_p , wiki_p , stars , watchers , assignees DEFAULT eieio_unbound, forks DEFAULT eieio_unbound, issues DEFAULT eieio_unbound, labels DEFAULT eieio_unbound, revnotes DEFAULT eieio_unbound, pullreqs DEFAULT eieio_unbound);
CREATE TABLE assignee (repository NOT NULL, id NOT NULL PRIMARY KEY, login , name , forge_id , FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE fork (parent NOT NULL, id NOT NULL PRIMARY KEY, owner , name , FOREIGN KEY (parent) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE issue (class NOT NULL, id NOT NULL PRIMARY KEY, repository , number , state , author , title , created , updated , closed , unread_p , locked_p , milestone , body , assignees DEFAULT eieio_unbound, cards DEFAULT eieio_unbound, edits DEFAULT eieio_unbound, labels DEFAULT eieio_unbound, participants DEFAULT eieio_unbound, posts DEFAULT eieio_unbound, reactions DEFAULT eieio_unbound, timeline DEFAULT eieio_unbound, FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE issue_assignee (issue NOT NULL, id NOT NULL, FOREIGN KEY (issue) REFERENCES issue (id) ON DELETE CASCADE);
CREATE TABLE issue_label (issue NOT NULL, id NOT NULL, FOREIGN KEY (issue) REFERENCES issue (id) ON DELETE CASCADE, FOREIGN KEY (id) REFERENCES label (id) ON DELETE CASCADE);
CREATE TABLE issue_post (class NOT NULL, id NOT NULL PRIMARY KEY, issue , number , author , created , updated , body , edits DEFAULT eieio_unbound, reactions DEFAULT eieio_unbound, FOREIGN KEY (issue) REFERENCES issue (id) ON DELETE CASCADE);
CREATE TABLE label (repository NOT NULL, id NOT NULL PRIMARY KEY, name , color , description , FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE notification (class NOT NULL, id NOT NULL PRIMARY KEY, repository , forge , reason , unread_p , last_read , updated , title , type , topic , url , FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE pullreq (class NOT NULL, id NOT NULL PRIMARY KEY, repository , number , state , author , title , created , updated , closed , merged , unread_p , locked_p , editable_p , cross_repo_p , base_ref , base_repo , head_ref , head_user , head_repo , milestone , body , assignees DEFAULT eieio_unbound, cards DEFAULT eieio_unbound, commits DEFAULT eieio_unbound, edits DEFAULT eieio_unbound, labels DEFAULT eieio_unbound, participants DEFAULT eieio_unbound, posts DEFAULT eieio_unbound, reactions DEFAULT eieio_unbound, review_requests DEFAULT eieio_unbound, reviews DEFAULT eieio_unbound, timeline DEFAULT eieio_unbound, FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
CREATE TABLE pullreq_assignee (pullreq NOT NULL, id NOT NULL, FOREIGN KEY (pullreq) REFERENCES pullreq (id) ON DELETE CASCADE);
CREATE TABLE pullreq_label (pullreq NOT NULL, id NOT NULL, FOREIGN KEY (pullreq) REFERENCES pullreq (id) ON DELETE CASCADE, FOREIGN KEY (id) REFERENCES label (id) ON DELETE CASCADE);
CREATE TABLE pullreq_post (class NOT NULL, id NOT NULL PRIMARY KEY, pullreq , number , author , created , updated , body , edits DEFAULT eieio_unbound, reactions DEFAULT eieio_unbound, FOREIGN KEY (pullreq) REFERENCES pullreq (id) ON DELETE CASCADE);
CREATE TABLE revnote (class NOT NULL, id NOT NULL PRIMARY KEY, repository , "commit" , file , line , author , body , FOREIGN KEY (repository) REFERENCES repository (id) ON DELETE CASCADE);
--8<---------------cut here---------------end--------------->8---
Fetching all new 19 issues & 10 MRs since my initial 2-hour fetch
yesterday took about 3 minutes.
I think the bottleneck here is the current GitLab API. If I find some
time, I can re-run the procedure with an even bigger GitHub project.
I've read GitHub has a more efficient GraphQL API for this stuff, and
AFAIK, GitLab is or will implement something similar, so this slowness
might be resolved in the mid-term future.
If you have a preference for some project hosted on GitHub (e.g.,
because it has about the same numbers of issues as we have), drop me a
mail.
Bye,
Tassilo
next prev parent reply other threads:[~2019-05-13 18:59 UTC|newest]
Thread overview: 284+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-17 2:17 bug#34889: [RFE] Migration to gitlab Konstantin Kharlamov
2019-03-17 3:01 ` Konstantin Kharlamov
2019-03-17 3:34 ` bug#34889: " Konstantin Kharlamov
2019-03-17 8:20 ` Tim Cross
2019-03-17 9:51 ` Michael Albinus
2019-03-17 11:16 ` Konstantin Kharlamov
2019-03-17 18:05 ` Eli Zaretskii
2019-03-19 0:52 ` Dmitry Gutov
2019-03-19 1:43 ` Glenn Morris
2019-03-19 1:50 ` Glenn Morris
2019-03-20 2:28 ` Richard Stallman
2019-03-19 7:57 ` Eli Zaretskii
2019-03-19 7:27 ` Philippe Vaucher
2019-03-19 8:47 ` Tadeus Prastowo
2019-03-19 12:31 ` Philippe Vaucher
2019-03-19 12:46 ` Tadeus Prastowo
2019-03-19 11:03 ` Ergus
2019-03-19 7:45 ` Eli Zaretskii
2019-03-19 14:13 ` Dmitry Gutov
[not found] ` <message from DmitryGutovon Tue>
2019-03-19 18:15 ` Eli Zaretskii
2019-03-19 21:59 ` Konstantin Kharlamov
2019-03-20 6:13 ` Eli Zaretskii
2019-03-20 6:56 ` Konstantin Kharlamov
2019-03-20 7:23 ` Eli Zaretskii
2019-03-21 8:28 ` Philippe Vaucher
2019-03-21 9:02 ` Tadeus Prastowo
2019-03-21 9:48 ` Philippe Vaucher
2019-03-21 9:59 ` Tadeus Prastowo
2019-03-21 17:54 ` Philippe Vaucher
2019-03-21 19:03 ` Eli Zaretskii
2019-04-23 21:19 ` Toon Claes
2019-04-24 7:06 ` Eli Zaretskii
2019-04-25 7:52 ` Toon Claes
2019-03-22 10:37 ` Marcin Borkowski
2019-03-22 10:56 ` Jean-Christophe Helary
2019-03-22 18:52 ` Marcin Borkowski
2019-03-23 0:37 ` Jean-Christophe Helary
2019-03-22 11:24 ` Konstantin Kharlamov
2019-03-22 12:38 ` Philippe Vaucher
2019-03-22 13:27 ` Konstantin Kharlamov
2019-03-22 13:57 ` Stefan Monnier
2019-03-22 13:17 ` Eli Zaretskii
2019-03-22 13:50 ` Stefan Monnier
2019-03-22 14:05 ` Konstantin Kharlamov
2019-03-22 14:20 ` Teemu Likonen
2019-03-22 14:29 ` Stefan Monnier
2019-03-22 14:54 ` Eli Zaretskii
2019-03-22 15:19 ` Stefan Monnier
2019-03-22 15:38 ` Eli Zaretskii
2019-03-22 15:58 ` Stefan Monnier
2019-03-23 21:58 ` Juri Linkov
2019-03-22 14:41 ` Eli Zaretskii
2019-03-23 2:33 ` Richard Stallman
2019-03-23 7:18 ` Eli Zaretskii
2019-03-23 14:04 ` Konstantin Kharlamov
2019-03-23 14:28 ` Eli Zaretskii
2019-03-24 6:29 ` Van L
2019-03-24 11:22 ` Eli Zaretskii
2019-03-24 1:44 ` Richard Stallman
2019-03-22 10:01 ` Konstantin Kharlamov
2019-03-22 10:16 ` Eli Zaretskii
2019-03-22 10:34 ` Konstantin Kharlamov
2019-03-22 13:44 ` Eli Zaretskii
2019-03-22 14:36 ` Dmitry Gutov
2019-03-22 14:57 ` Stefan Monnier
2019-03-22 17:01 ` Dmitry Gutov
2019-03-22 15:28 ` Eli Zaretskii
2019-03-22 12:43 ` Basil L. Contovounesios
2019-03-22 13:05 ` Stefan Monnier
2019-03-22 13:30 ` Konstantin Kharlamov
2019-03-22 13:44 ` Stefan Monnier
2019-03-22 16:46 ` Glenn Morris
2019-03-22 18:56 ` Stefan Monnier
2019-03-22 13:32 ` Eli Zaretskii
2019-03-22 13:48 ` Stefan Monnier
2019-03-22 14:37 ` Eli Zaretskii
2019-03-22 14:50 ` Dmitry Gutov
2019-03-22 15:31 ` Eli Zaretskii
2019-03-22 16:46 ` Dmitry Gutov
2019-03-22 16:23 ` Michael Albinus
2019-03-22 16:37 ` Eli Zaretskii
2019-03-22 16:48 ` Michael Albinus
2019-03-22 17:22 ` Eli Zaretskii
2019-03-22 16:52 ` Glenn Morris
2019-03-22 16:57 ` Michael Albinus
2019-03-22 17:24 ` Eli Zaretskii
2019-03-24 13:53 ` Michael Albinus
2019-03-24 15:52 ` Eli Zaretskii
2019-03-25 16:29 ` Michael Albinus
2019-03-25 17:09 ` Eli Zaretskii
2019-03-25 17:52 ` Tadeus Prastowo
2019-03-25 17:56 ` Michael Albinus
2019-03-25 17:54 ` Michael Albinus
2019-03-22 18:50 ` Glenn Morris
2019-03-22 19:00 ` Dmitry Gutov
2019-03-22 17:23 ` Eli Zaretskii
2019-04-20 23:26 ` Dmitry Gutov
2019-04-21 5:43 ` Eli Zaretskii
2019-04-21 7:58 ` Michael Albinus
2019-04-25 1:17 ` Dmitry Gutov
2019-04-25 8:17 ` Michael Albinus
2019-04-25 1:06 ` Dmitry Gutov
2019-04-25 9:22 ` Eli Zaretskii
2019-04-25 10:35 ` Dmitry Gutov
2019-04-25 10:55 ` Eli Zaretskii
2019-04-25 15:01 ` Dmitry Gutov
2019-04-25 19:54 ` Eli Zaretskii
2019-04-25 23:16 ` Dmitry Gutov
2019-04-26 7:52 ` Michael Albinus
2019-04-26 12:49 ` Dmitry Gutov
2019-04-26 13:03 ` Michael Albinus
2019-04-26 8:05 ` Eli Zaretskii
2019-04-27 1:40 ` Dmitry Gutov
2019-04-27 9:43 ` Eli Zaretskii
2019-05-15 2:04 ` Dmitry Gutov
2019-05-15 2:30 ` Lars Ingebrigtsen
2019-05-15 5:42 ` Lars Ingebrigtsen
2019-05-15 13:45 ` Dmitry Gutov
2019-05-15 14:34 ` Eli Zaretskii
2019-05-16 3:57 ` Lars Ingebrigtsen
2019-05-16 13:41 ` Eli Zaretskii
2019-05-16 13:48 ` Lars Ingebrigtsen
2019-05-16 14:09 ` Eli Zaretskii
2019-05-16 14:34 ` debbugs extensions (was: [RFE] Migration to gitlab) Michael Albinus
2019-05-16 23:40 ` Noam Postavsky
2019-05-17 7:30 ` debbugs extensions Michael Albinus
2019-05-17 8:40 ` Eli Zaretskii
2019-05-17 9:25 ` Michael Albinus
2019-05-17 10:45 ` Noam Postavsky
2019-05-15 13:37 ` [RFE] Migration to gitlab Dmitry Gutov
2019-05-16 3:54 ` Lars Ingebrigtsen
2019-04-26 8:42 ` Ricardo Wurmus
2019-04-26 19:41 ` Dmitry Gutov
2019-03-20 1:02 ` Dmitry Gutov
2019-03-18 1:48 ` Richard Stallman
2019-03-18 2:41 ` Tim Cross
2019-03-18 13:19 ` Van L
2019-03-19 2:15 ` Richard Stallman
2019-03-19 14:24 ` Dmitry Gutov
2019-03-20 2:33 ` Richard Stallman
2019-03-18 16:14 ` Karl Fogel
2019-03-17 16:48 ` Eric Abrahamsen
2019-03-17 18:05 ` Amin Bandali
2019-03-17 3:40 ` Eli Zaretskii
2019-03-17 12:37 ` Philippe Vaucher
2019-03-17 13:14 ` Tadeus Prastowo
2019-03-17 13:23 ` Konstantin Kharlamov
2019-03-17 13:49 ` Tadeus Prastowo
2019-03-17 14:06 ` Konstantin Kharlamov
2019-03-17 14:26 ` Tadeus Prastowo
2019-03-17 15:06 ` Stefan Monnier
2019-03-17 16:55 ` Eli Zaretskii
2019-03-17 17:45 ` Stefan Monnier
2019-03-17 17:29 ` Alex
2019-04-18 8:27 ` Toon Claes
2019-04-20 21:12 ` Dmitry Gutov
2019-04-23 21:08 ` Toon Claes
2019-04-24 15:26 ` Alex Gramiak
2019-04-25 8:24 ` Toon Claes
2019-04-25 13:45 ` Alex Gramiak
2019-04-25 0:42 ` Dmitry Gutov
2019-04-25 8:32 ` Eli Zaretskii
2019-05-10 9:16 ` Toon Claes
2019-05-10 9:49 ` Eli Zaretskii
2019-05-10 10:37 ` 조성빈
2019-05-10 12:21 ` Eli Zaretskii
2019-05-10 13:09 ` 조성빈
2019-05-10 22:23 ` Alex Gramiak
2019-05-11 2:12 ` Alan Mackenzie
2019-05-11 3:47 ` 조성빈
2019-05-11 7:01 ` Eli Zaretskii
2019-05-11 7:38 ` 조성빈
2019-05-11 10:02 ` Eli Zaretskii
2019-05-11 13:13 ` Dmitry Gutov
2019-05-11 13:49 ` Eli Zaretskii
2019-05-11 13:57 ` Dmitry Gutov
2019-05-11 14:04 ` Eli Zaretskii
2019-05-11 19:25 ` Basil L. Contovounesios
2019-05-11 19:25 ` Basil L. Contovounesios
2019-05-11 19:24 ` Basil L. Contovounesios
2019-05-11 19:22 ` Basil L. Contovounesios
2019-05-12 15:50 ` Alan Mackenzie
2019-05-12 20:51 ` Basil L. Contovounesios
2019-06-18 15:36 ` Simon Leinen
2019-06-25 22:38 ` Basil L. Contovounesios
2019-06-26 18:01 ` Simon Leinen
2019-06-26 18:21 ` Basil L. Contovounesios
2019-05-12 0:58 ` Alex Gramiak
2019-05-11 6:32 ` Eli Zaretskii
2019-05-12 0:23 ` Alex Gramiak
2019-05-12 5:31 ` Eli Zaretskii
2019-05-12 7:04 ` Tassilo Horn
2019-05-12 13:56 ` Eli Zaretskii
2019-05-13 4:32 ` Tassilo Horn
2019-05-13 14:51 ` Eli Zaretskii
2019-05-13 18:24 ` Clément Pit-Claudel
2019-05-13 16:41 ` [OFFTOPIC] size of issue tracker (was: [RFE] Migration to gitlab) Stefan Monnier
2019-05-13 17:42 ` Eli Zaretskii
2019-05-13 18:55 ` [OFFTOPIC] size of issue tracker Stefan Monnier
2019-05-13 18:59 ` Óscar Fuentes
2019-05-13 19:16 ` Stefan Monnier
2019-05-13 18:59 ` Tassilo Horn [this message]
2019-05-13 20:02 ` Tassilo Horn
2019-05-13 20:11 ` Tassilo Horn
2019-05-13 20:56 ` Stefan Monnier
2019-05-14 8:43 ` Toon Claes
2019-05-14 19:58 ` Stefan Monnier
2019-05-15 7:45 ` Toon Claes
2019-05-15 14:04 ` Stefan Monnier
2019-05-15 14:41 ` Eli Zaretskii
2019-05-16 17:54 ` Clemens Radermacher
2019-05-16 19:58 ` Stefan Monnier
2019-05-16 23:19 ` Jean-Christophe Helary
2019-05-16 23:31 ` Stefan Monnier
2019-05-19 19:34 ` Juri Linkov
2019-05-19 20:12 ` Stefan Monnier
2019-05-19 20:46 ` Juri Linkov
2019-05-20 11:57 ` Toon Claes
2019-05-20 12:29 ` Basil L. Contovounesios
2019-05-10 11:16 ` [RFE] Migration to gitlab Dmitry Gutov
2019-05-10 12:54 ` Eli Zaretskii
2019-05-10 13:56 ` Dmitry Gutov
2019-05-10 14:19 ` Eli Zaretskii
2019-05-10 14:32 ` Tadeus Prastowo
2019-05-10 14:56 ` Óscar Fuentes
2019-05-10 15:16 ` Tadeus Prastowo
2019-05-10 15:00 ` 조성빈
2019-05-10 15:26 ` Clément Pit-Claudel
2019-05-11 12:13 ` Eli Zaretskii
2019-05-11 15:37 ` Clément Pit-Claudel
2019-05-11 15:51 ` Eli Zaretskii
2019-05-11 15:57 ` Clément Pit-Claudel
2019-05-13 8:47 ` Toon Claes
2019-05-10 16:33 ` Dmitry Gutov
2019-05-10 20:43 ` Eli Zaretskii
2019-05-10 21:12 ` Óscar Fuentes
2019-05-11 6:13 ` Eli Zaretskii
2019-05-11 6:16 ` 조성빈
2019-05-11 12:16 ` Eli Zaretskii
2019-05-11 12:34 ` Dmitry Gutov
2019-05-11 12:40 ` Eli Zaretskii
2019-05-11 13:29 ` Amin Bandali
2019-05-11 13:58 ` Eli Zaretskii
2019-05-11 14:06 ` Eli Zaretskii
2019-05-11 14:42 ` Amin Bandali
2019-05-11 14:57 ` Eli Zaretskii
2019-05-11 16:09 ` Amin Bandali
2019-05-11 14:11 ` Amin Bandali
2019-05-11 15:41 ` 조성빈
2019-05-13 9:23 ` Toon Claes
2019-05-10 21:32 ` Stefan Monnier
2019-05-10 21:56 ` Alex Gramiak
2019-05-11 6:22 ` Eli Zaretskii
2019-05-11 19:19 ` Basil L. Contovounesios
2019-05-13 1:43 ` Dmitry Gutov
2019-05-13 1:45 ` Dmitry Gutov
2019-05-13 14:48 ` Eli Zaretskii
2019-05-13 18:14 ` Dmitry Gutov
2019-05-13 9:03 ` Toon Claes
2019-05-13 18:22 ` Dmitry Gutov
2019-05-14 10:23 ` EMBA enable Reply by Email (was: [RFE] Migration to gitlab) Toon Claes
2019-05-10 14:02 ` [RFE] Migration to gitlab Óscar Fuentes
2019-05-10 14:28 ` Eli Zaretskii
2019-05-10 14:54 ` Óscar Fuentes
2019-05-10 15:34 ` Eli Zaretskii
2019-05-10 16:23 ` Alan Mackenzie
2019-05-12 19:09 ` Juri Linkov
2019-05-12 22:24 ` Óscar Fuentes
2019-05-14 13:13 ` Stefan Monnier
2019-05-10 14:02 ` Debbugs problems (was: [RFE] Migration to gitlab) Stefan Monnier
2019-05-10 14:24 ` Debbugs problems Michael Albinus
2019-05-10 15:16 ` Eli Zaretskii
2019-05-13 1:09 ` Dmitry Gutov
2019-05-13 14:27 ` Eli Zaretskii
2019-05-13 17:56 ` Dmitry Gutov
2019-05-13 18:03 ` Eli Zaretskii
2019-05-13 18:57 ` Óscar Fuentes
2019-05-14 14:36 ` Eli Zaretskii
2019-05-13 20:20 ` Dmitry Gutov
2019-05-14 14:36 ` Eli Zaretskii
2019-05-10 10:41 ` [RFE] Migration to gitlab Dmitry Gutov
2019-05-10 15:23 ` Toon Claes
2019-05-10 13:48 ` Stefan Monnier
2019-03-17 17:49 ` Eli Zaretskii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8736liuqmw.fsf@gnu.org \
--to=tsdh@gnu.org \
--cc=agrambot@gmail.com \
--cc=dgutov@yandex.ru \
--cc=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
--cc=monnier@iro.umontreal.ca \
--cc=toon@iotcl.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).