Getting a copy of the Cuirass (berlin.guixsd.org) database

Getting a copy of the Cuirass (berlin.guixsd.org) database

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

Hey,

Would someone be able to make a copy of the Cuirass database available?
It might be useful when tinkering with Cuirass, as it contains some
realistic data.

I don't know much about SQLite, but it looks like there's a ".backup"
command [1] which might be useful.

1: https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Sun, 10 Feb 2019 13:21:57 +0000
Christopher Baines <mail@cbaines.net> wrote:

> Would someone be able to make a copy of the Cuirass database
> available? It might be useful when tinkering with Cuirass, as it
> contains some realistic data.

Hi Chris,

I'm not sure if anybody else responded to you, but I have a database
around. I got it last year from Clement Lassieur. It is 1.6 GB in size,
I made a tar.xz an that is only 86 MB! I wonder if there is anything
wrong or it is just thaaat compressible! Anyway, you can try it out
here:

http://bjoernhoefling.de/berlin.tar.xz

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]


Björn Höfling <bjoern.hoefling@bjoernhoefling.de> writes:

> On Sun, 10 Feb 2019 13:21:57 +0000
> Christopher Baines <mail@cbaines.net> wrote:
>
>> Would someone be able to make a copy of the Cuirass database
>> available? It might be useful when tinkering with Cuirass, as it
>> contains some realistic data.
>
> Hi Chris,
>
> I'm not sure if anybody else responded to you, but I have a database
> around. I got it last year from Clement Lassieur. It is 1.6 GB in size,
> I made a tar.xz an that is only 86 MB! I wonder if there is anything
> wrong or it is just thaaat compressible! Anyway, you can try it out
> here:

Awesome, thanks Björn. I've downloaded it now :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Hi Björn,

On Wed, 13 Feb 2019 22:00:06 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> I'm not sure if anybody else responded to you, but I have a database
> around. I got it last year from Clement Lassieur. It is 1.6 GB in size,
> I made a tar.xz an that is only 86 MB! I wonder if there is anything
> wrong or it is just thaaat compressible!

I think we didn't set up the infrastructure to VACUUM the database.

The default in sqlite is not to VACUUM ever, so we probably should
either stop the entire thing and vacuum it once a week or we could
set up autovacuum to incremental or full (the latter would vacuum
at each commit) if we ever ran out of space.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

On Thu, 21 Feb 2019 21:30:39 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> Hi Björn,
> 
> On Wed, 13 Feb 2019 22:00:06 +0100
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:
> 
> > I'm not sure if anybody else responded to you, but I have a database
> > around. I got it last year from Clement Lassieur. It is 1.6 GB in
> > size, I made a tar.xz an that is only 86 MB! I wonder if there is
> > anything wrong or it is just thaaat compressible!  
> 
> I think we didn't set up the infrastructure to VACUUM the database.
> 
> The default in sqlite is not to VACUUM ever, so we probably should
> either stop the entire thing and vacuum it once a week or we could
> set up autovacuum to incremental or full (the latter would vacuum
> at each commit) if we ever ran out of space.

That would at least tell why it is so good compressable.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Wed, 13 Feb 2019 22:00:06 +0100
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:
>
>> I'm not sure if anybody else responded to you, but I have a database
>> around. I got it last year from Clement Lassieur. It is 1.6 GB in size,
>> I made a tar.xz an that is only 86 MB! I wonder if there is anything
>> wrong or it is just thaaat compressible!
>
> I think we didn't set up the infrastructure to VACUUM the database.
>
> The default in sqlite is not to VACUUM ever, so we probably should
> either stop the entire thing and vacuum it once a week or we could
> set up autovacuum to incremental or full (the latter would vacuum
> at each commit) if we ever ran out of space.

What would it take to enable autovacuum?  Sounds like something we
should do.  :-)

Ludo’.

Re: Getting a copy of the Cuirass (berlin.guixsd.org) database

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi Danny,
>
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
>> On Wed, 13 Feb 2019 22:00:06 +0100
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:
>>
>>> I'm not sure if anybody else responded to you, but I have a database
>>> around. I got it last year from Clement Lassieur. It is 1.6 GB in size,
>>> I made a tar.xz an that is only 86 MB! I wonder if there is anything
>>> wrong or it is just thaaat compressible!
>>
>> I think we didn't set up the infrastructure to VACUUM the database.
>>
>> The default in sqlite is not to VACUUM ever, so we probably should
>> either stop the entire thing and vacuum it once a week or we could
>> set up autovacuum to incremental or full (the latter would vacuum
>> at each commit) if we ever ran out of space.
>
> What would it take to enable autovacuum?  Sounds like something we
> should do.  :-)

I think vacuum can result in the rowid values changing [1], and in at
least one place (the build ids), the rowid values are exposed to users.

1: https://www.sqlite.org/lang_createtable.html#rowid

I think by creating a explicit column for the id, it's possible to avoid
them changing during the vacuum, so it might be good to do that before
vacuuming.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]