Can general compute and packaging be more formally merged into a single case?

Can general compute and packaging be more formally merged into a single case?

[Josh Marshall]

At the airport, thinking on the fundamental differences between gwl and
guix.  It seems like these can be articulated as the same case when
considering a tracked and linked compute history.

How I see this, when packaging you take checksums off of inputs not for
your own assurance that they are correct (though you could) but to ensure
that under different circumstances another user can be sure that they have
the right starting points.  Then as a matter of storing results and
ensuring the integrity of our results for later we take more checksums.
What we can do is to create a unit computational step of sorts whereby a
user enters a monitored shell whereby they install packages, perform their
work, and produce changes which can be taken to be outputs.  All downloads,
uploads, and files changes tracked.  Then perform a basic minimization
algorithm to reduce the inputs so long as the outputs do not differ.  This
optimized unit computational step can then be tracked with the input
checksums and outputs.  This merges general compute and packaging, then
adding compute power only needs to scale here.

From these, computational chains may also be produced to know a full graph
of what is happening.  Thoughts?

Re: Can general compute and packaging be more formally merged into a single case?

[zimoun]

Hi Josh

On Tue, 3 Dec 2019 at 18:34, Josh Marshall
<joshua.r.marshall.1991@gmail.com> wrote:
>
> At the airport, thinking on the fundamental differences between gwl and
> guix.  It seems like these can be articulated as the same case when
> considering a tracked and linked compute history.

On gwl-devel@gnu.org, from my understanding, we are discussing that
and it seems related to the Content Addressable Store (CAS).

Otherwise, about the differences between GWL and Guix, you can dig in
some archeology; especially read the initial proposal by Roel and the
comments by Ludo. (I think I already pointed to you where the related
messages live.)

> How I see this, when packaging you take checksums off of inputs not for
> your own assurance that they are correct (though you could) but to ensure
> that under different circumstances another user can be sure that they have
> the right starting points.  Then as a matter of storing results and
> ensuring the integrity of our results for later we take more checksums.
> What we can do is to create a unit computational step of sorts whereby a
> user enters a monitored shell whereby they install packages, perform their
> work, and produce changes which can be taken to be outputs.

This already works in GWL. :-)

>All downloads,
> uploads, and files changes tracked.

To me, it is not clear how GWL should track this because they can be
really huge.

> Then perform a basic minimization
> algorithm to reduce the inputs so long as the outputs do not differ.

Which kind of minimization algorithm do you have in mind?

> This
> optimized unit computational step can then be tracked with the input
> checksums and outputs.  This merges general compute and packaging, then
> adding compute power only needs to scale here.
>
> From these, computational chains may also be produced to know a full graph
> of what is happening.  Thoughts?

It is already the case. If I understand well.


All the best,
simon

Re: Can general compute and packaging be more formally merged into a single case?

[Josh Marshall]

Neat!  I'll go digging and see what I can find.

On Tue, Dec 3, 2019, 16:22 zimoun <zimon.toutoune@gmail.com> wrote:

> Hi Josh
>
> On Tue, 3 Dec 2019 at 18:34, Josh Marshall
> <joshua.r.marshall.1991@gmail.com> wrote:
> >
> > At the airport, thinking on the fundamental differences between gwl and
> > guix.  It seems like these can be articulated as the same case when
> > considering a tracked and linked compute history.
>
> On gwl-devel@gnu.org, from my understanding, we are discussing that
> and it seems related to the Content Addressable Store (CAS).
>
> Otherwise, about the differences between GWL and Guix, you can dig in
> some archeology; especially read the initial proposal by Roel and the
> comments by Ludo. (I think I already pointed to you where the related
> messages live.)
>
> > How I see this, when packaging you take checksums off of inputs not for
> > your own assurance that they are correct (though you could) but to ensure
> > that under different circumstances another user can be sure that they
> have
> > the right starting points.  Then as a matter of storing results and
> > ensuring the integrity of our results for later we take more checksums.
> > What we can do is to create a unit computational step of sorts whereby a
> > user enters a monitored shell whereby they install packages, perform
> their
> > work, and produce changes which can be taken to be outputs.
>
> This already works in GWL. :-)
>
> >All downloads,
> > uploads, and files changes tracked.
>
> To me, it is not clear how GWL should track this because they can be
> really huge.
>
> > Then perform a basic minimization
> > algorithm to reduce the inputs so long as the outputs do not differ.
>
> Which kind of minimization algorithm do you have in mind?
>
> > This
> > optimized unit computational step can then be tracked with the input
> > checksums and outputs.  This merges general compute and packaging, then
> > adding compute power only needs to scale here.
> >
> > From these, computational chains may also be produced to know a full
> graph
> > of what is happening.  Thoughts?
>
> It is already the case. If I understand well.
>
>
> All the best,
> simon
>

Re: Can general compute and packaging be more formally merged into a single case?

[Josh Marshall]

For the minimization step, I believe that a practical approach would
be to omit individual commands run on the terminal while monitoring
and ignoring terminal prints and bash history if at all possible.
This is so that common commands used for checking what is going on
like `ls` can be omitted.

Still trying to get to the reading and packaging here.  Lots to do and
I don't get much time at work.

On Tue, Dec 3, 2019 at 7:16 PM Josh Marshall
<joshua.r.marshall.1991@gmail.com> wrote:
>
> Neat!  I'll go digging and see what I can find.
>
> On Tue, Dec 3, 2019, 16:22 zimoun <zimon.toutoune@gmail.com> wrote:
>>
>> Hi Josh
>>
>> On Tue, 3 Dec 2019 at 18:34, Josh Marshall
>> <joshua.r.marshall.1991@gmail.com> wrote:
>> >
>> > At the airport, thinking on the fundamental differences between gwl and
>> > guix.  It seems like these can be articulated as the same case when
>> > considering a tracked and linked compute history.
>>
>> On gwl-devel@gnu.org, from my understanding, we are discussing that
>> and it seems related to the Content Addressable Store (CAS).
>>
>> Otherwise, about the differences between GWL and Guix, you can dig in
>> some archeology; especially read the initial proposal by Roel and the
>> comments by Ludo. (I think I already pointed to you where the related
>> messages live.)
>>
>> > How I see this, when packaging you take checksums off of inputs not for
>> > your own assurance that they are correct (though you could) but to ensure
>> > that under different circumstances another user can be sure that they have
>> > the right starting points.  Then as a matter of storing results and
>> > ensuring the integrity of our results for later we take more checksums.
>> > What we can do is to create a unit computational step of sorts whereby a
>> > user enters a monitored shell whereby they install packages, perform their
>> > work, and produce changes which can be taken to be outputs.
>>
>> This already works in GWL. :-)
>>
>> >All downloads,
>> > uploads, and files changes tracked.
>>
>> To me, it is not clear how GWL should track this because they can be
>> really huge.
>>
>> > Then perform a basic minimization
>> > algorithm to reduce the inputs so long as the outputs do not differ.
>>
>> Which kind of minimization algorithm do you have in mind?
>>
>> > This
>> > optimized unit computational step can then be tracked with the input
>> > checksums and outputs.  This merges general compute and packaging, then
>> > adding compute power only needs to scale here.
>> >
>> > From these, computational chains may also be produced to know a full graph
>> > of what is happening.  Thoughts?
>>
>> It is already the case. If I understand well.
>>
>>
>> All the best,
>> simon