Hello GUIX

Hello GUIX

[Shivam Madlani]

[-- Attachment #1.1: Type: text/plain, Size: 927 bytes --]

Hey everyone!
I am Shivam Madlani, an undergraduate at DAIICT, India. I want to
participate in GSoC'23 for The GNU organization. The project "Decentralized
substitute distribution" in GUIX amused me and I want to apply for the
same. I have been in touch with the project mentor -pukkamustard for over a
month now and started learning more about the project. I'm very excited to
work on this project as it lies in my interests.

I've drafted a proposal on this project and I aim to add a transport
protocol-Sneakernet. This will allow the substitutes to be air-gappable. My
approach is to use SQLite to store the ERIS encoded substitute in the
external drive using a custom schema. This is just a brief overview of the
project and more details are included in the attached proposal.

I would love to hear your thoughts and reviews on my proposal and hopefully
improve it by filling in any missing key points.

Thank you,
Shivam

[-- Attachment #1.2: Type: text/html, Size: 1069 bytes --]

[-- Attachment #2: GSoC'23 GUIX proposal.pdf --]
[-- Type: application/pdf, Size: 300200 bytes --]

Re: Hello GUIX

[pukkamustard]

Hi Shivam,

Welcome to Guix! I think that Sneakernet-esque distribution of
substitutes is very interesting and something that was also talked about
at the Guix Days before FOSDEM 23 (Guix with no internet and air-gapped
machines).

Some comments on the proposal:

- You might want to refer to the document "SQLite for ERIS block
  storage" (https://eris.codeberg.page/eer/sqlite.xml).

- I think the user experience as described in the proposal could be
  improved/might not quite work as expected:

  - Reading from a disk maybe should happen automagically if it can be
    detected (and is enabled in some configuration). This might also
    require some integration with udisks. The complexity of this should
    not be underestimated.

  - Writing packages to a USB stick seems to be closer to a `guix
    publish` or `guix deploy`. Maybe you can read up on those commands
    and think of a nice way to publish or deploy a set of packages to a
    USB stick. This is also relevant for publishing to other p2p
    networks (IPFS, GNUnet, etc.).

- Maybe the Narinfos also need to packed onto the USB drive? How would
  an air-gapped computer otherwise know which substitute to get? Also
  have a look at the nar-herder tool
  (https://git.cbaines.net/guix/nar-herder) by Christopher Baines (in
  CC).

-pukkamustard

Shivam Madlani <shivammadlani5@gmail.com> writes:

> Hey everyone!
> I am Shivam Madlani, an undergraduate at DAIICT, India. I want to participate in GSoC'23 for The GNU
> organization. The project "Decentralized substitute distribution" in GUIX amused me and I want to
> apply for the same. I have been in touch with the project mentor -pukkamustard for over a month
> now and started learning more about the project. I'm very excited to work on this project as it lies in
> my interests.
>
> I've drafted a proposal on this project and I aim to add a transport protocol-Sneakernet. This will allow
> the substitutes to be air-gappable. My approach is to use SQLite to store the ERIS encoded substitute
> in the external drive using a custom schema. This is just a brief overview of the project and more
> details are included in the attached proposal.
>
> I would love to hear your thoughts and reviews on my proposal and hopefully improve it by filling in
> any missing key points.
>
> Thank you,
> Shivam
>
> [4. application/pdf; GSoC'23 GUIX proposal.pdf]...

Re: Hello GUIX

[Shivam Madlani]

[-- Attachment #1: Type: text/plain, Size: 4697 bytes --]

Thanks a lot for the feedback!!

>  Reading from a disk maybe should happen automagically if it can be
>  detected (and is enabled in some configuration). This might also
>  require some integration with udisks. The complexity of this should
>  not be underestimated.

We can set it up in such a way that on running `guix install {xyz}` it will
first check for mounted drives in /media directory and install {xyz} if
it's present in the drive.

> Writing packages to a USB stick seems to be closer to a `guix
>  publish` or `guix deploy`. Maybe you can read up on those commands
>  and think of a nice way to publish or deploy a set of packages to a
>  USB stick. This is also relevant for publishing to other p2p
>  networks (IPFS, GNUnet, etc.).

Perhaps, We can introduce a new utility? (maybe something like `guix
sneak`). This will scan for external drives and on detecting a drive it
will encode and store the nars into it.
The size of the drive can cause an issue here in case the content to be
encoded is very large. In this case an appropriate error message will be
displayed. Although this is rare as flash drives/hard drives have a lot of
storage capacity these days but it's still a thing to consider.

Another way would be to use `guix publish` itself to do all of this. We can
introduce a flag `--sneak` which instead of spawning an HTTP server,
encodes and stored into the drive. But i don't think we should modify an
already existing utility for this. I would like to hear your thoughts on
this.

> Maybe the Narinfos also need to packed onto the USB drive? How would
> an air-gapped computer otherwise know which substitute to get? Also
> have a look at the nar-herder tool
> (https://git.cbaines.net/guix/nar-herder) by Christopher Baines (in
> CC).

Yes, we will be introducing a manifest file in the drive while writing to
it. It would contain the narinfos of all the packages in the drive.
Apologies for not mentioning in the proposal.


I have made the above mentioned changes to my proposal as per the
suggestions. If i can further improve it in any way please do let me know.
:)

-Shivam

On Wed, 29 Mar 2023 at 15:42, pukkamustard <pukkamustard@posteo.net> wrote:

>
> Hi Shivam,
>
> Welcome to Guix! I think that Sneakernet-esque distribution of
> substitutes is very interesting and something that was also talked about
> at the Guix Days before FOSDEM 23 (Guix with no internet and air-gapped
> machines).
>
> Some comments on the proposal:
>
> - You might want to refer to the document "SQLite for ERIS block
>   storage" (https://eris.codeberg.page/eer/sqlite.xml).
>
> - I think the user experience as described in the proposal could be
>   improved/might not quite work as expected:
>
>   - Reading from a disk maybe should happen automagically if it can be
>     detected (and is enabled in some configuration). This might also
>     require some integration with udisks. The complexity of this should
>     not be underestimated.
>
>   - Writing packages to a USB stick seems to be closer to a `guix
>     publish` or `guix deploy`. Maybe you can read up on those commands
>     and think of a nice way to publish or deploy a set of packages to a
>     USB stick. This is also relevant for publishing to other p2p
>     networks (IPFS, GNUnet, etc.).
>
> - Maybe the Narinfos also need to packed onto the USB drive? How would
>   an air-gapped computer otherwise know which substitute to get? Also
>   have a look at the nar-herder tool
>   (https://git.cbaines.net/guix/nar-herder) by Christopher Baines (in
>   CC).
>
> -pukkamustard
>
> Shivam Madlani <shivammadlani5@gmail.com> writes:
>
> > Hey everyone!
> > I am Shivam Madlani, an undergraduate at DAIICT, India. I want to
> participate in GSoC'23 for The GNU
> > organization. The project "Decentralized substitute distribution" in
> GUIX amused me and I want to
> > apply for the same. I have been in touch with the project mentor
> -pukkamustard for over a month
> > now and started learning more about the project. I'm very excited to
> work on this project as it lies in
> > my interests.
> >
> > I've drafted a proposal on this project and I aim to add a transport
> protocol-Sneakernet. This will allow
> > the substitutes to be air-gappable. My approach is to use SQLite to
> store the ERIS encoded substitute
> > in the external drive using a custom schema. This is just a brief
> overview of the project and more
> > details are included in the attached proposal.
> >
> > I would love to hear your thoughts and reviews on my proposal and
> hopefully improve it by filling in
> > any missing key points.
> >
> > Thank you,
> > Shivam
> >
> > [4. application/pdf; GSoC'23 GUIX proposal.pdf]...
>
>

[-- Attachment #2: Type: text/html, Size: 5917 bytes --]

Re: Hello GUIX

[Csepp]

Shivam Madlani <shivammadlani5@gmail.com> writes:

> Thanks a lot for the feedback!!
>
>>  Reading from a disk maybe should happen automagically if it can be
>>  detected (and is enabled in some configuration). This might also
>>  require some integration with udisks. The complexity of this should
>>  not be underestimated.
>
> We can set it up in such a way that on running `guix install {xyz}` it will first check for mounted drives in /media
> directory and install {xyz} if it's present in the drive. 
>
>> Writing packages to a USB stick seems to be closer to a `guix
>>  publish` or `guix deploy`. Maybe you can read up on those commands
>>  and think of a nice way to publish or deploy a set of packages to a
>>  USB stick. This is also relevant for publishing to other p2p
>>  networks (IPFS, GNUnet, etc.).
>
> Perhaps, We can introduce a new utility? (maybe something like `guix sneak`). This will scan for external drives
> and on detecting a drive it will encode and store the nars into it.
> The size of the drive can cause an issue here in case the content to be encoded is very large. In this case an
> appropriate error message will be displayed. Although this is rare as flash drives/hard drives have a lot of storage
> capacity these days but it's still a thing to consider.
>
> Another way would be to use `guix publish` itself to do all of this. We can introduce a flag `--sneak` which instead
> of spawning an HTTP server, encodes and stored into the drive. But i don't think we should modify an already
> existing utility for this. I would like to hear your thoughts on this.

The NNCP utility might be worth taking a look at.  There is a way to set
it up alongside udev to automatically transfer files from connected USB
storage.

Re: Hello GUIX

[Shivam Madlani]

[-- Attachment #1: Type: text/plain, Size: 2887 bytes --]

> The NNCP utility might be worth taking a look at.  There is a way to set
> it up alongside udev to automatically transfer files from connected USB
> storage.

I did a bit of research at the NNCP util and yes, this can be used for
automatic file transfers.
We can create a new udev rule that would trigger nncp when a USB device is
connected. The directories where the files need to be transferred can be
done by creating a config file for NNCP and setting the 'nncpdir' variable
as the desired path. Further customization can be done by altering the nncp
config and udev rules.

I also took a look at the rsync utility which is more widely used but does
not provide features like encryption and compression. Although, it does
have a feature that transfers only the changes within the file instead of
the whole file (better transfer speeds).

Personally, I think nncp is a better choice here as it provides more
features and the tradeoff for using rsync is not that much.

Regards,
Shivam

On Thu, 6 Apr 2023 at 06:55, Csepp <raingloom@riseup.net> wrote:

>
> Shivam Madlani <shivammadlani5@gmail.com> writes:
>
> > Thanks a lot for the feedback!!
> >
> >>  Reading from a disk maybe should happen automagically if it can be
> >>  detected (and is enabled in some configuration). This might also
> >>  require some integration with udisks. The complexity of this should
> >>  not be underestimated.
> >
> > We can set it up in such a way that on running `guix install {xyz}` it
> will first check for mounted drives in /media
> > directory and install {xyz} if it's present in the drive.
> >
> >> Writing packages to a USB stick seems to be closer to a `guix
> >>  publish` or `guix deploy`. Maybe you can read up on those commands
> >>  and think of a nice way to publish or deploy a set of packages to a
> >>  USB stick. This is also relevant for publishing to other p2p
> >>  networks (IPFS, GNUnet, etc.).
> >
> > Perhaps, We can introduce a new utility? (maybe something like `guix
> sneak`). This will scan for external drives
> > and on detecting a drive it will encode and store the nars into it.
> > The size of the drive can cause an issue here in case the content to be
> encoded is very large. In this case an
> > appropriate error message will be displayed. Although this is rare as
> flash drives/hard drives have a lot of storage
> > capacity these days but it's still a thing to consider.
> >
> > Another way would be to use `guix publish` itself to do all of this. We
> can introduce a flag `--sneak` which instead
> > of spawning an HTTP server, encodes and stored into the drive. But i
> don't think we should modify an already
> > existing utility for this. I would like to hear your thoughts on this.
>
> The NNCP utility might be worth taking a look at.  There is a way to set
> it up alongside udev to automatically transfer files from connected USB
> storage.
>

[-- Attachment #2: Type: text/html, Size: 3589 bytes --]