Setup pointers for extindex and /all

Setup pointers for extindex and /all

[Konstantin Ryabitsev]

Eric:

I'm working on the new incarnation of lore.kernel.org (that will run on
multiple frontends as opposed to the centralized version we have now) -- I
hope to have everything ready to go by the time 1.7.x rolls out. I wonder if
you can give some pointers for extindex and /all, specifically:

- what needs to go into the config file to enable the feature? I poked at the
  source, and it appears that this will do it?:
  [extindex "all"]
  topdir = /path/to/where/xap15/will/live

- when "public-inbox-index" runs in grok-post script, will it automatically
  update the "all" extindex without any additional flags, or do we need to
  pass something special to it?

- when handling message-id based lookups, will the "all" extindex be used
  automatically, if found? 

Thanks in advance!

-K

Re: Setup pointers for extindex and /all

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Eric:
> 
> I'm working on the new incarnation of lore.kernel.org (that will run on
> multiple frontends as opposed to the centralized version we have now) -- I
> hope to have everything ready to go by the time 1.7.x rolls out. I wonder if
> you can give some pointers for extindex and /all, specifically:
> 
> - what needs to go into the config file to enable the feature? I poked at the
>   source, and it appears that this will do it?:
>   [extindex "all"]
>   topdir = /path/to/where/xap15/will/live

Yes, I would run `public-inbox-extindex' to create the extindex
before altering the config file.

It's probably safe to use --no-fsync by default and some larger
value of --batch-size= since you have beefy machines.  The
initial index of lore took over 30 hours for me IIRC; but
that's on an ancient machine with a SATA2 SSD.

I'm not 100% sure how configuration should work since there's no
"public-inbox-extindex-init" command (and I'm not sure if it's
necessary).

> - when "public-inbox-index" runs in grok-post script, will it automatically
>   update the "all" extindex without any additional flags, or do we need to
>   pass something special to it?

Yes.  However I've been favoring using --no-update-extindex with
plain -index, and then doing "public-inbox-extindex --all /path/to/extindex"
after all -index are done to reduce transactions.

> - when handling message-id based lookups, will the "all" extindex be used
>   automatically, if found?

Yes, it should be.

> Thanks in advance!

No problem.  Please let us know if anything doesn't work as it
should.  The configuration could be smoother, I think...

I think the on-disk format is sufficiently stable at this point,
but not yet 100%...