Re: msgmap serial number regeneration [was: Q: V2 format]

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> ebiederm@xmission.com (Eric W. Biederman) writes:
>> > Eric Wong <e@80x24.org> writes:
>> >> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> >>> 
>> >>> Because of the parallelism in V2 I have noticed messages in numbered
>> >>> in an order that does not correspond to their commit order.  So the
>> >>> SQLite database isn't as recoverable as it might be.  Especially as the
>> >>> parallelism introduces an element of non-determinancy.
>> >>
>> >> *puzzled* were you able to reproduce that?  The serial number
>> >> generation + threading happens in the main process and the
>> >> parallelism is limited to Xapian text indexing.  -index
>> >> generates serial numbers by walking backwards with v2, and
>> >> complains on unexpected results.
>> 
>> Digging into this I have found consistenly non-reproducible numbering,
>> because of deleted files.  Apparently in both V1 and V2 an a worst-case
>> estimate is made of the total numbers that are going to be needed and
>> numbers are assigned backwards from there.
>> 
>> A fresh indexing of the git mailling list archive on v1 gives me numbers
>> starting with 360 and on v2 numbers starting with 355.  Which
>> corresponds with the number of deleted messages.
>> 
>> I am still looking to see if there are any other weird things here.
>
> Ah, yes, you're correct deletes don't get accounted for when
> regenerating.  Oh well.  I guess it was correct to document msgmap
> as something important to backup and not break for instances of
> particular servers.  (emphasis on "particular servers")
>
> So I think you'd need to walk revision history twice to account
> for deleted messages...
>
> Across different machines, it should not  matter to preserve
> serials.

I believe we can modify the msg number assignment to assign numbers to
deletes as well as adds.   Short of the same Message-ID coming up twice
that should be enough for the current backwards loop to assign message
ids reliably.  And even Message-IDs comming up twice is handle-able.

>> I definitely do not like not being able to reconstruct message numbers
>> from a backup.
>
> For v2, I see serial numbers are an internal optimization which
> happens to map to NNTP.
>
> If the git repo is cloned and the cloner sets up a different
> server, it'll have a different address and clients won't know to
> deduplicate them anyways.  I suppose it makes the load-balanced
> case a little more complex to sync(*)

But if the server hardware fails.  The case I am dealing with at the
moment I can stand up a new server with the same ip address.

Further if we can make everything but the git repository non-essential
it yields more flexibility for changing and optimizing things in the
future.

> (*) But optimizing for load-balanced instances isn't ideal,
>     I'd rather see more independently-run servers than giant
>     load-balanced instances which everybody relies on.

True.

At this point I am just optimizing for my own operational simplicity
of my own indpendentyly-run server.

Eric