GSoC final update

GSoC final update

[Caleb Ristvedt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

As the official work time for GSoC draws to a close, it seems fitting to
summarize where the project is at right now. *deep breath*

What works: building hello. I tried building SBCL as well, but building
one of the dependencies caused hanging (but the same thing happened when
I tried building it with the C++ daemon). In short, for at least simple
derivations, building derivations works.

What doesn't: Way too much. RPCs aren't implemented (for testing
building I've been manually running builds from a REPL within test-env),
scheduling of multiple concurrent derivations isn't implemented, garbage
collection isn't implemented, and a lot of features even of
derivation-building aren't implemented (substitutes, offloading,
etc). call-with-container is also in a semi-broken state, though the big
issues could be fixed by going back to the way it was and bind-mounting
a store directory to collect output in (bind-mounts mess with my head
for some reason... thanks for mentioning that, Ricardo). The list goes
on.

Maybe I should describe my experience a bit. Reading the C++ source was
extremely tedious. The only way to know what a value is at any point is
to follow the entire execution of the program up to that point. A
slightly faster heuristic is to rgrep for anything that might change
that value, and consider only those statements. This, however, is
confused by C++'s scoping rules. For example, there are 3 "settings"
variables visible in globals.cc: a global variable from globals.hh, a
private variable from globals.hh, and a namespace-local (I think? The
scoping rules still aren't completely clear to me) variable from
globals.cc. As a side-note, in the future I will consider any code that
repeats four of the same names in a row when read aloud ("Settings
settings; Settings::Settings()") to be something to run far away from.

The scheme side of things was nice. The most frustrating part was not
knowing what was already implemented, especially what was implemented
but used an RPC for a low-level task. I also didn't know that a scanner
already existed (in hindsight it should have been obvious - how else
would grafts work?) and so ended up writing my own inefficient version
(though I intuitively think for some reason that it could be made fast).

<Digression> I found myself checking the guile reference quite
frequently. One time I happened upon a part describing the ECMAScript
implementation (curiosity and all that), and noticed that several times
it was mentioned how irresponsible the implementor was. That scared me
probably more than it should have. By all objective criteria, I have
failed in this project, and it frightens me to think that, nice as the
community is, I could screw up badly enough to end up a footnote
somewhere as an example of what not to be. That fear pretty much sums up
my mental state during the second half of the summer.</Digression>

In short, I've failed to achieve the goals I set out in my
proposal. Worst of all is that I failed to regularly communicate. I
specifically said that I would regularly communicate, and then I failed
to do so, especially during the second half of the summer. It seems that
the less I communicated, the more afraid of communicating I
became.

Currently nothing has been merged - I don't think it's in a fit state to
be merged presently. I intend to continue working on this as time
allows.  School started last week, though, and it's kept me quite busy
so far.

The code so far can be found at the guile-daemon branch on Savannah.

tl;dr - I did not achieve the goals I set out to achieve. That makes me
sad. I plan on working on it more.

- - reepca

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAlmlGwUACgkQwWaqSV9/
GJybxwf/ZYtcANxxfaeS2YQFO3IQOlz+eI5cESvlzjyeQUZ6OgBm7sRTkSg67b2S
W4i3TvPKeEC/H1skFnYeh+ltlpQQo06ifgrh9ZUGNWbMeMTMq1Aqop6ZP97WQEtx
xihPCkjDve45zR4BwdqR6EB162c9Q16SJ3297g4SpcMKfoAKjnIv7T49tWCd2wCm
qp8Ia9D29DDYb8WAvD7ZyDo5g0fvB+qsHXpkNm5Lgic2m7XB0eRoWx/PzvtDNPzb
4z3OWN4vVvj1NZStyEgOkifuciY+Pn+W7RjcaQS0vrfkIcGFM7yZBm19nmBZAyrD
sz/vNcOfauVsMx3cI1MEZ7QAHIZfpg==
=GYqh
-----END PGP SIGNATURE-----

Re: GSoC final update

[Andy Wingo]

On Tue 29 Aug 2017 09:44, Caleb Ristvedt <caleb.ristvedt@cune.org> writes:

> <Digression> I found myself checking the guile reference quite
> frequently. One time I happened upon a part describing the ECMAScript
> implementation (curiosity and all that), and noticed that several times
> it was mentioned how irresponsible the implementor was. That scared me
> probably more than it should have. By all objective criteria, I have
> failed in this project, and it frightens me to think that, nice as the
> community is, I could screw up badly enough to end up a footnote
> somewhere as an example of what not to be. That fear pretty much sums up
> my mental state during the second half of the summer.</Digression>

LOL I wrote that about myself :-) As in, the ECMAScript implementation
was mostly mine, and its faults were mine, and I documented that and
moved on.  It would be nice to fix some day but it takes time; in the
meantime I just wanted to moderate peoples' expectations.  Apologies if
that language made you feel intimidated, the intention was precisely the
opposite.

Happy hacking! :-)

Andy

Re: GSoC final update

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On Tue, Aug 29, 2017 at 02:44:56AM -0500, Caleb Ristvedt wrote:
> 
> As the official work time for GSoC draws to a close, it seems fitting to
> summarize where the project is at right now. *deep breath*
> 
> What works: building hello. I tried building SBCL as well, but building
> one of the dependencies caused hanging (but the same thing happened when
> I tried building it with the C++ daemon). In short, for at least simple
> derivations, building derivations works.
> 
<snip>
> 
> Currently nothing has been merged - I don't think it's in a fit state to
> be merged presently. I intend to continue working on this as time
> allows.  School started last week, though, and it's kept me quite busy
> so far.
> 
> The code so far can be found at the guile-daemon branch on Savannah.
> 
> tl;dr - I did not achieve the goals I set out to achieve. That makes me
> sad. I plan on working on it more.
> 
> - - reepca
> 

What can we do to help it along? Is there any sort of user testing that
would be helpful feedback?


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GSoC final update

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

(off-list)

Hey Caleb,

On Tue, Aug 29, 2017 at 02:44:56AM -0500, Caleb Ristvedt wrote:
> <Digression> I found myself checking the guile reference quite
> frequently. One time I happened upon a part describing the ECMAScript
> implementation (curiosity and all that), and noticed that several times
> it was mentioned how irresponsible the implementor was. That scared me
> probably more than it should have. By all objective criteria, I have
> failed in this project, and it frightens me to think that, nice as the
> community is, I could screw up badly enough to end up a footnote
> somewhere as an example of what not to be. That fear pretty much sums up
> my mental state during the second half of the summer.</Digression>
> 
> In short, I've failed to achieve the goals I set out in my
> proposal. Worst of all is that I failed to regularly communicate. I
> specifically said that I would regularly communicate, and then I failed
> to do so, especially during the second half of the summer. It seems that
> the less I communicated, the more afraid of communicating I
> became.

I've had similar feelings at different parts of my life. Sometimes it's
been related to work, sometimes to relationships.

I want to say that I can relate to what you experienced, and I hope you
can let go of those negative feelings.

For me, I've since realized that I was *way* too hard on myself each
time it happened.

From time to time, we all have the experience of not reaching some goal
that we have set for ourselves. This seems not to affect some people at
all; I tend to feel very bad about it.

But, as I've grown older I've learned to see it happening and tell
myself to let it go. Nobody else wants me to feel bad, and they won't be
upset that I didn't meet my goal. The harsh taskmaster is in my head, so
it's completely safe for me to tell him to get lost ;)

I hope you won't feel bad if you don't pick this work back up right
away, or ever.

Peace,
Leo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GSoC final update

[Ludovic Courtès]

Hi reepca!

Caleb Ristvedt <caleb.ristvedt@cune.org> skribis:

> As the official work time for GSoC draws to a close, it seems fitting to
> summarize where the project is at right now. *deep breath*
>
> What works: building hello. I tried building SBCL as well, but building
> one of the dependencies caused hanging (but the same thing happened when
> I tried building it with the C++ daemon). In short, for at least simple
> derivations, building derivations works.
>
> What doesn't: Way too much. RPCs aren't implemented (for testing
> building I've been manually running builds from a REPL within test-env),
> scheduling of multiple concurrent derivations isn't implemented, garbage
> collection isn't implemented, and a lot of features even of
> derivation-building aren't implemented (substitutes, offloading,
> etc). call-with-container is also in a semi-broken state, though the big
> issues could be fixed by going back to the way it was and bind-mounting
> a store directory to collect output in (bind-mounts mess with my head
> for some reason... thanks for mentioning that, Ricardo). The list goes
> on.

Thanks for the status update!  I think you don’t have to be so negative
about what you did, though.  The things that you implemented look
well-thought-out to me, which to me suggests that maybe you worked
“depth-first” rather than “breadth-first”.  It does mean that some of
the things initially planned for are missing, but it also means that
existing bits are more polished than one would expect for a GSoC.

> Maybe I should describe my experience a bit. Reading the C++ source was
> extremely tedious.

I can sympathize with this.  This code base is rather good and
well-commented, but yeah, C++ scoping rules and the fully imperative
style make it hard to follow.

> The scheme side of things was nice. The most frustrating part was not
> knowing what was already implemented, especially what was implemented
> but used an RPC for a low-level task. I also didn't know that a scanner
> already existed (in hindsight it should have been obvious - how else
> would grafts work?) and so ended up writing my own inefficient version
> (though I intuitively think for some reason that it could be made fast).

[...]

> In short, I've failed to achieve the goals I set out in my
> proposal. Worst of all is that I failed to regularly communicate. I
> specifically said that I would regularly communicate, and then I failed
> to do so, especially during the second half of the summer. It seems that
> the less I communicated, the more afraid of communicating I
> became.

Communication is hard, but I think it’s crucial on short-term projects
like this one, as I think I said when you got started.  For example,
asking “do we have a reference scanner?” on IRC could have saved you
quite a bit of time.  In general, it’s often better to ask than to
remain stuck on something for ages—we’re here to help.

It seems to me that GSoC’s mentor-student relationship is sometimes seen
as boss-subordinate.  That’s really not the way I see it; I think the
student is just another person learning about the project and willing to
help, and they shouldn’t be any more afraid of asking for help than
anyone else.  I hope I didn’t give you the wrong impression.

> Currently nothing has been merged - I don't think it's in a fit state to
> be merged presently. I intend to continue working on this as time
> allows.  School started last week, though, and it's kept me quite busy
> so far.

It can’t be merged as-is, but most of the bits look solid enough as a
foundation.  I hope we can keep you on board in the coming months to
pursue this endeavor!

Thanks for the update, thanks for the code!

Ludo’.

Re: GSoC final update

[Ricardo Wurmus]

Hi reepca,

thanks for your report and for your work!

> tl;dr - I did not achieve the goals I set out to achieve. That makes me
> sad. I plan on working on it more.

Sometimes projects are like that.  They can get more confusing as you go
on and try to rebuild the execution model in your head, especially when
dealing with a language like C++.  These things cannot always be
anticipated at the beginning.  Like Ludo said, communicating early and
often can be a real time saver in situations like this.

On the plus side, you did manage to get a deeper understanding of the
daemon code than most of us who rarely venture onto the C++ side.  Your
understanding of what has been implemented already and what still needs
work is very valuable in driving the project to completion in the
following months.  I hope school won’t erode your internal model too
quickly.  It would be great to see more progress for which you’ve
already laid the foundation.

Thank you!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: GSoC final update

[Andreas Enge]

Hello Caleb,

On Tue, Aug 29, 2017 at 02:44:56AM -0500, Caleb Ristvedt wrote:
> Currently nothing has been merged - I don't think it's in a fit state to
> be merged presently. I intend to continue working on this as time
> allows.  School started last week, though, and it's kept me quite busy
> so far.
> 
> The code so far can be found at the guile-daemon branch on Savannah.

hopefully you will find some time to continue working on the project!
It would definitely be great to end up with a daemon written in Guile
at some point in time.

I have a probably naïve question, since I did not follow the project closely:
Instead of an all-or-nothing outcome, success or failure, merge or bit rot,
would it be possible to set up an incremental update path?
Use some of your scheme code to replace part of the C++ daemon?
Maybe write a high-level scheme engine that shells out to C++ functions in
the current daemon, that could gradually be replaced by chunks of your code?

If this is nonsense, please do not hesitate, you or other specialists of
the topic, to call it so ;-)

Cheers,

Andreas

Re: GSoC final update

[Ludovic Courtès]

Heya,

Andreas Enge <andreas@enge.fr> skribis:

> I have a probably naïve question, since I did not follow the project closely:
> Instead of an all-or-nothing outcome, success or failure, merge or bit rot,
> would it be possible to set up an incremental update path?
> Use some of your scheme code to replace part of the C++ daemon?
> Maybe write a high-level scheme engine that shells out to C++ functions in
> the current daemon, that could gradually be replaced by chunks of your code?

I like this idea.  One of the first things reepca worked on was writing
a replacement for the ‘guix-register’ C++ program in Scheme, which also
provides the foundation for everything else on that branch.

I think we should be able to merge this bit independently of the rest,
as a first step.

It requires care, but that’s definitely something we should look into.

Thanks,
Ludo’.