the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[ng0]

tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
Danny, could you list what's left for completion? Is it just circular
dependencies? I'm about to publish my project page within the next few
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
If you do mind, please let me know. I will not publish your email
address, I'll rather point to a git commit.

----- Forwarded message from Sebastian Hahn -----

From: Sebastian Hahn
To: tor-dev@lists.torproject.org
Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam

Hi there tor-dev,

as an update to those who didn't have the chance to meet with us in
Amsterdam or those who haven't followed the efforts to rely on C less,
here's what happened at the "let's not fight about Go versus Rust, but
talk about how to migrate Tor to a safer language" session and what
happened after.

Notes from session:

We didn't fight about Rust or Go or modern C++. Instead, we focused on
identifying goals for migrating Tor to a memory-safe language, and how
to get there. With that frame of reference, Rust emerged as a extremely
strong candidate for the incremental improvement style that we
considered necessary. We were strongly advised to not use cgo, by people
who have used it extensively.

As there are clearly a lot of unknowns with this endeavor, and a lot
that we will learn/come up against along the way, we feel that Rust is a
compelling option to start with,  with the caveat that we will first
experiment, learn from the experience, and then build on what we learn.

You can also check out the session notes on the wiki (submitted, but not
posted yet).[1]

The real fun part started after the session. We got together to actually
make a plan for an experiment and to give Rust a serious chance. We
quickly got a few trivial things working like statically linking Rust
into Tor, integrating with the build system to call out to cargo for the
Rust build, and using Tor's allocator from Rust.

We're planning to write up a blog post summarizing our experiences so
far while hopefully poking the Rust developers to prioritize the missing
features so we can stop using nightly Rust soon (~months, instead of
years).

We want to have a patch merged into tor soon so you can all play with
your dev setup to help identify any challenges. We want to stress that
this is an optional experiment for now, we would love feedback but
nobody is paid to work on this and nobody is expected to spend more
time than they have sitting around.

We have committed to reviewing any patch that includes any Rust code to
provide feedback, get experience to develop a style, and actually make
use of this experiment. This means we're not ready to take on big
patches that add lots of tricky stuff quite now, we want to take it slow
and learn from this.

We would like to do a session at the next dev meeting to give updates on
this effort, but in the meantime, if team members would like to start
learning Rust and helping us identify/implement small and well-isolated
areas to begin migration, or new pieces of functionality that we can
build  immediately in Rust, that would be really great.

So, for a TLDR:

What has already been done:
- Rust in Tor build
- Putting together environment setup instructions and a (very small)
 initial draft for coding standards
- Initial work to identify good candidates for migration (not tightly
 interdependent)

What we think are next steps:
- Define conventions for the API boundary between Rust and C
- Add a non-trivial Rust API and deploy with a flag to optionally use
 (to test support with a safe fallback)
- Learn from similar projects
- Add automated tooling for Rust, such as linting and testing


Cheers
Alex, Chelsea, Sebastian

[1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
_______________________________________________
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

----- End forwarded message -----

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[Leo Famulari]

On Sat, Apr 01, 2017 at 07:58:41AM +0000, ng0 wrote:
> tor is seriously considering to switch to rust. I have my own open
> bugs / roadmap points which depend on our rust-build-system working.
> tor will not switch immediately, but it shows the importance of having a
> working rust-build-system. Firefox will switch at some point.

It's immediately important because the latest version of librsvg (used
to create the GuixSD GRUB background image) is written in Rust. We could
even update that package "the hard way", without a full
rust-build-system.

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Apr 01, 2017 at 07:58:41AM +0000, ng0 wrote:
>> tor is seriously considering to switch to rust. I have my own open
>> bugs / roadmap points which depend on our rust-build-system working.
>> tor will not switch immediately, but it shows the importance of having a
>> working rust-build-system. Firefox will switch at some point.
>
> It's immediately important because the latest version of librsvg (used
> to create the GuixSD GRUB background image) is written in Rust. We could
> even update that package "the hard way", without a full
> rust-build-system.

For librsvg and Tor, a switch to Rust is both good news and bad news:
it’s good news because using a memory-safe language is indeed a wise
decision, but it’s bad news because it introduces a single point of
trust (opaque Rust binaries built by Mozilla) in our dependency graph.

Ludo’.

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

Ludovic Courtès transcribed 0.9K bytes:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Sat, Apr 01, 2017 at 07:58:41AM +0000, ng0 wrote:
> >> tor is seriously considering to switch to rust. I have my own open
> >> bugs / roadmap points which depend on our rust-build-system working.
> >> tor will not switch immediately, but it shows the importance of having a
> >> working rust-build-system. Firefox will switch at some point.
> >
> > It's immediately important because the latest version of librsvg (used
> > to create the GuixSD GRUB background image) is written in Rust. We could
> > even update that package "the hard way", without a full
> > rust-build-system.
> 
> For librsvg and Tor, a switch to Rust is both good news and bad news:
> it’s good news because using a memory-safe language is indeed a wise
> decision, but it’s bad news because it introduces a single point of
> trust (opaque Rust binaries built by Mozilla) in our dependency graph.
> 
> Ludo’.
> 

It gets even worse: https://www.mercurial-scm.org/wiki/OxidationPlan
So being able to handle rust will be necessary not only for Guix, but
for GNU (iirc we had some Mercurial repos somewhere in GNU, at least
GNU Octave and GNU Health)!
-- 
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
  WWW: https://n0.is

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[ng0]

Oh sweet damn.... My friends and myself so ignorant of this April Fools
annoyance that I would say: Please double check this message if
torproject
does not have the same opinion on aprils fools as I do.

But: the message of tor was merely an opener to my question. I'm still
curious in what needs to be done so that I can get people to join on the
rust parts of my project ;)

ng0 transcribed 4.2K bytes:
> tor is seriously considering to switch to rust. I have my own open
> bugs / roadmap points which depend on our rust-build-system working.
> tor will not switch immediately, but it shows the importance of having a
> working rust-build-system. Firefox will switch at some point.
> Danny, could you list what's left for completion? Is it just circular
> dependencies? I'm about to publish my project page within the next few
> days. I hope you don't mind if I list you as a go-to person for getting
> involved in upstream (Guix) to fix up the rust-build-system.
> If you do mind, please let me know. I will not publish your email
> address, I'll rather point to a git commit.
> 
> ----- Forwarded message from Sebastian Hahn -----
> 
> From: Sebastian Hahn
> To: tor-dev@lists.torproject.org
> Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam
> 
> Hi there tor-dev,
> 
> as an update to those who didn't have the chance to meet with us in
> Amsterdam or those who haven't followed the efforts to rely on C less,
> here's what happened at the "let's not fight about Go versus Rust, but
> talk about how to migrate Tor to a safer language" session and what
> happened after.
> 
> Notes from session:
> 
> We didn't fight about Rust or Go or modern C++. Instead, we focused on
> identifying goals for migrating Tor to a memory-safe language, and how
> to get there. With that frame of reference, Rust emerged as a extremely
> strong candidate for the incremental improvement style that we
> considered necessary. We were strongly advised to not use cgo, by people
> who have used it extensively.
> 
> As there are clearly a lot of unknowns with this endeavor, and a lot
> that we will learn/come up against along the way, we feel that Rust is a
> compelling option to start with,  with the caveat that we will first
> experiment, learn from the experience, and then build on what we learn.
> 
> You can also check out the session notes on the wiki (submitted, but not
> posted yet).[1]
> 
> The real fun part started after the session. We got together to actually
> make a plan for an experiment and to give Rust a serious chance. We
> quickly got a few trivial things working like statically linking Rust
> into Tor, integrating with the build system to call out to cargo for the
> Rust build, and using Tor's allocator from Rust.
> 
> We're planning to write up a blog post summarizing our experiences so
> far while hopefully poking the Rust developers to prioritize the missing
> features so we can stop using nightly Rust soon (~months, instead of
> years).
> 
> We want to have a patch merged into tor soon so you can all play with
> your dev setup to help identify any challenges. We want to stress that
> this is an optional experiment for now, we would love feedback but
> nobody is paid to work on this and nobody is expected to spend more
> time than they have sitting around.
> 
> We have committed to reviewing any patch that includes any Rust code to
> provide feedback, get experience to develop a style, and actually make
> use of this experiment. This means we're not ready to take on big
> patches that add lots of tricky stuff quite now, we want to take it slow
> and learn from this.
> 
> We would like to do a session at the next dev meeting to give updates on
> this effort, but in the meantime, if team members would like to start
> learning Rust and helping us identify/implement small and well-isolated
> areas to begin migration, or new pieces of functionality that we can
> build  immediately in Rust, that would be really great.
> 
> So, for a TLDR:
> 
> What has already been done:
> - Rust in Tor build
> - Putting together environment setup instructions and a (very small)
>  initial draft for coding standards
> - Initial work to identify good candidates for migration (not tightly
>  interdependent)
> 
> What we think are next steps:
> - Define conventions for the API boundary between Rust and C
> - Add a non-trivial Rust API and deploy with a flag to optionally use
>  (to test support with a safe fallback)
> - Learn from similar projects
> - Add automated tooling for Rust, such as linting and testing
> 
> 
> Cheers
> Alex, Chelsea, Sebastian
> 
> [1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
> _______________________________________________
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
> 
> ----- End forwarded message -----
> 

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[ng0]

ng0 transcribed 4.8K bytes:
> Oh sweet damn.... My friends and myself so ignorant of this April Fools
> annoyance that I would say: Please double check this message if
> torproject
> does not have the same opinion on aprils fools as I do.

https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
and
https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes/MemorySafeLanguagesandTor

suggest however that it really is *not* a joke as it happened long
before April 1st.
 
> But: the message of tor was merely an opener to my question. I'm still
> curious in what needs to be done so that I can get people to join on the
> rust parts of my project ;)
> 
> ng0 transcribed 4.2K bytes:
> > tor is seriously considering to switch to rust. I have my own open
> > bugs / roadmap points which depend on our rust-build-system working.
> > tor will not switch immediately, but it shows the importance of having a
> > working rust-build-system. Firefox will switch at some point.
> > Danny, could you list what's left for completion? Is it just circular
> > dependencies? I'm about to publish my project page within the next few
> > days. I hope you don't mind if I list you as a go-to person for getting
> > involved in upstream (Guix) to fix up the rust-build-system.
> > If you do mind, please let me know. I will not publish your email
> > address, I'll rather point to a git commit.
> > 
> > ----- Forwarded message from Sebastian Hahn -----
> > 
> > From: Sebastian Hahn
> > To: tor-dev@lists.torproject.org
> > Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam
> > 
> > Hi there tor-dev,
> > 
> > as an update to those who didn't have the chance to meet with us in
> > Amsterdam or those who haven't followed the efforts to rely on C less,
> > here's what happened at the "let's not fight about Go versus Rust, but
> > talk about how to migrate Tor to a safer language" session and what
> > happened after.
> > 
> > Notes from session:
> > 
> > We didn't fight about Rust or Go or modern C++. Instead, we focused on
> > identifying goals for migrating Tor to a memory-safe language, and how
> > to get there. With that frame of reference, Rust emerged as a extremely
> > strong candidate for the incremental improvement style that we
> > considered necessary. We were strongly advised to not use cgo, by people
> > who have used it extensively.
> > 
> > As there are clearly a lot of unknowns with this endeavor, and a lot
> > that we will learn/come up against along the way, we feel that Rust is a
> > compelling option to start with,  with the caveat that we will first
> > experiment, learn from the experience, and then build on what we learn.
> > 
> > You can also check out the session notes on the wiki (submitted, but not
> > posted yet).[1]
> > 
> > The real fun part started after the session. We got together to actually
> > make a plan for an experiment and to give Rust a serious chance. We
> > quickly got a few trivial things working like statically linking Rust
> > into Tor, integrating with the build system to call out to cargo for the
> > Rust build, and using Tor's allocator from Rust.
> > 
> > We're planning to write up a blog post summarizing our experiences so
> > far while hopefully poking the Rust developers to prioritize the missing
> > features so we can stop using nightly Rust soon (~months, instead of
> > years).
> > 
> > We want to have a patch merged into tor soon so you can all play with
> > your dev setup to help identify any challenges. We want to stress that
> > this is an optional experiment for now, we would love feedback but
> > nobody is paid to work on this and nobody is expected to spend more
> > time than they have sitting around.
> > 
> > We have committed to reviewing any patch that includes any Rust code to
> > provide feedback, get experience to develop a style, and actually make
> > use of this experiment. This means we're not ready to take on big
> > patches that add lots of tricky stuff quite now, we want to take it slow
> > and learn from this.
> > 
> > We would like to do a session at the next dev meeting to give updates on
> > this effort, but in the meantime, if team members would like to start
> > learning Rust and helping us identify/implement small and well-isolated
> > areas to begin migration, or new pieces of functionality that we can
> > build  immediately in Rust, that would be really great.
> > 
> > So, for a TLDR:
> > 
> > What has already been done:
> > - Rust in Tor build
> > - Putting together environment setup instructions and a (very small)
> >  initial draft for coding standards
> > - Initial work to identify good candidates for migration (not tightly
> >  interdependent)
> > 
> > What we think are next steps:
> > - Define conventions for the API boundary between Rust and C
> > - Add a non-trivial Rust API and deploy with a flag to optionally use
> >  (to test support with a safe fallback)
> > - Learn from similar projects
> > - Add automated tooling for Rust, such as linting and testing
> > 
> > 
> > Cheers
> > Alex, Chelsea, Sebastian
> > 
> > [1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
> > _______________________________________________
> > tor-dev mailing list
> > tor-dev@lists.torproject.org
> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
> > 
> > ----- End forwarded message -----
> > 
> 

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[Danny Milosavljevic]

Hi ng0,

On Sat, 1 Apr 2017 07:58:41 +0000
ng0 <contact.ng0@cryptolab.net> wrote:

> Danny, could you list what's left for completion? Is it just circular
> dependencies? 

Very little is missing:
- Rustc and cargo should be disentangled.  Right now they have to be updated in lockstep.
- Rust has a new optional maker (instead of makefiles) called rustbuild; for some reason I didn't get it to work.  Future versions will drop the makefiles, so we have to get it to work eventually.
- Eventually we could try to bootstrap a Rust compiler from the original ocaml sources.  I'm trying to do that now but it's still broken. It would be fine to make it memory-safe by just not freeing anything ever - since it would only be used to compile the Rust compiler.
- Earlier we had an heuristic in that if there's a Cargo.lock file present we assume that it's an application, and if there isn't one we assume that it's a library.  Not sure how safe that heuristic is.  What's the official way to find out what it is?
- There's a design limitation in that our rust build system doesn't API support version ranges but cargo does.  If multiple libraries depend on the same library with differing API version ranges, cargo uses an version range intersection algorithm in order to find out which implementation version to use.  We don't do that - although we do use cargo, so it will fail in that case (and not do something invalid silently).

Note that Rust itself makes a distinction between stable features that are guaranteed to stay and stay the same in the future and unstable features that don't.  Many Rust crates use unstable features, among them very basic crates.  That makes us unable to use them, and rightfully so.

Other than that I've got a big number of (unpolished) Rust packages that do work.

> days. I hope you don't mind if I list you as a go-to person for getting
> involved in upstream (Guix) to fix up the rust-build-system.

Okay.

Re: the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]

[ng0]

[-- Attachment #1: Type: text/plain, Size: 3563 bytes --]

Hi,

my apologies. Occasionally I have a tendency for
very late replies in some threads, as you might have noticed.

Danny Milosavljevic transcribed 1.9K bytes:
> Hi ng0,
> 
> On Sat, 1 Apr 2017 07:58:41 +0000
> ng0 <contact.ng0@cryptolab.net> wrote:
> 
> > Danny, could you list what's left for completion? Is it just circular
> > dependencies? 
> 
> Very little is missing:
> - Rustc and cargo should be disentangled.  Right now they have to be updated in lockstep.

That's not really a problem for producing functional rust crate binaries.
I consider this optional. Or am I wrong?

> - Rust has a new optional maker (instead of makefiles) called rustbuild;
> for some reason I didn't get it to work.  Future versions will drop
> the makefiles, so we have to get it to work eventually.

Can you share the issues you had with this? How far did you come?

> - Eventually we could try to bootstrap a Rust compiler from the
> original ocaml sources.  I'm trying to do that now but it's still
> broken. It would be fine to make it memory-safe by just not
> freeing anything ever - since it would only be used to compile the
> Rust compiler.

Wouldn't it be easier to help the mrustc project, which is aimed at just that (bootstrapping rustc) in the long run?
Is it even possible at this point to bootstrap rustc using OCaml without wasting too much resources?
They've come a long way and many releases since they've gone selfhosted.

Can you share your findings and work on this?

> - Earlier we had an heuristic in that if there's a Cargo.lock file present we assume
> that it's an application, and if there isn't one we assume that it's a library.
> Not sure how safe that heuristic is.  What's the official way to find out what it is?

> - There's a design limitation in that our rust build system doesn't API support
> version ranges but cargo does.  If multiple libraries depend on the same library
> with differing API version ranges, cargo uses an version range intersection
> algorithm in order to find out which implementation version to use.
> We don't do that - although we do use cargo, so it will fail in that case (and
> not do something invalid silently).

Can you explain this a bit more in detail? You seem to have put more time to
work/think about these issues than I have done so far.
I need to understand most of the issues, so that I can communicate them to
people who'd work on them. At least to get them started on the issues.

> Note that Rust itself makes a distinction between stable features that
> are guaranteed to stay and stay
> the same in the future and unstable features that don't.
> Many Rust crates use unstable features, among them very basic crates.
> That makes us unable to use them, and rightfully so.

You mean fundamental basic crates required by many applications?
Do you have some insight into the implications of this?
I would provide a rustc-nightly outside of Guix master/official,
that's not an issue for me.

> 
> Other than that I've got a big number of (unpolished) Rust packages that do work.

Can you share these package definitions somewhere? I'd like
to help out and see how many intersections (duplicates) we
both have in our unfinished rust crates work.

> > days. I hope you don't mind if I list you as a go-to person for getting
> > involved in upstream (Guix) to fix up the rust-build-system.
> 
> Okay.

-- 
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
  WWW: https://n0.is

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]