unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* Setup pointers for extindex and /all
@ 2021-04-23 20:13 Konstantin Ryabitsev
  2021-04-23 20:29 ` Eric Wong
  0 siblings, 1 reply; 2+ messages in thread
From: Konstantin Ryabitsev @ 2021-04-23 20:13 UTC (permalink / raw)
  To: meta

Eric:

I'm working on the new incarnation of lore.kernel.org (that will run on
multiple frontends as opposed to the centralized version we have now) -- I
hope to have everything ready to go by the time 1.7.x rolls out. I wonder if
you can give some pointers for extindex and /all, specifically:

- what needs to go into the config file to enable the feature? I poked at the
  source, and it appears that this will do it?:
  [extindex "all"]
  topdir = /path/to/where/xap15/will/live

- when "public-inbox-index" runs in grok-post script, will it automatically
  update the "all" extindex without any additional flags, or do we need to
  pass something special to it?

- when handling message-id based lookups, will the "all" extindex be used
  automatically, if found? 

Thanks in advance!

-K

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Setup pointers for extindex and /all
  2021-04-23 20:13 Setup pointers for extindex and /all Konstantin Ryabitsev
@ 2021-04-23 20:29 ` Eric Wong
  0 siblings, 0 replies; 2+ messages in thread
From: Eric Wong @ 2021-04-23 20:29 UTC (permalink / raw)
  To: meta

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Eric:
> 
> I'm working on the new incarnation of lore.kernel.org (that will run on
> multiple frontends as opposed to the centralized version we have now) -- I
> hope to have everything ready to go by the time 1.7.x rolls out. I wonder if
> you can give some pointers for extindex and /all, specifically:
> 
> - what needs to go into the config file to enable the feature? I poked at the
>   source, and it appears that this will do it?:
>   [extindex "all"]
>   topdir = /path/to/where/xap15/will/live

Yes, I would run `public-inbox-extindex' to create the extindex
before altering the config file.

It's probably safe to use --no-fsync by default and some larger
value of --batch-size= since you have beefy machines.  The
initial index of lore took over 30 hours for me IIRC; but
that's on an ancient machine with a SATA2 SSD.

I'm not 100% sure how configuration should work since there's no
"public-inbox-extindex-init" command (and I'm not sure if it's
necessary).

> - when "public-inbox-index" runs in grok-post script, will it automatically
>   update the "all" extindex without any additional flags, or do we need to
>   pass something special to it?

Yes.  However I've been favoring using --no-update-extindex with
plain -index, and then doing "public-inbox-extindex --all /path/to/extindex"
after all -index are done to reduce transactions.

> - when handling message-id based lookups, will the "all" extindex be used
>   automatically, if found?

Yes, it should be.

> Thanks in advance!

No problem.  Please let us know if anything doesn't work as it
should.  The configuration could be smoother, I think...

I think the on-disk format is sufficiently stable at this point,
but not yet 100%...

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2021-04-23 20:29 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-23 20:13 Setup pointers for extindex and /all Konstantin Ryabitsev
2021-04-23 20:29 ` Eric Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).