Google Summer of Code - some ideas

Google Summer of Code - some ideas

[Aurélien Aptel]

Hi,

I would like to work on Emacs itself for this year GSoC. I have
several projects in mind.

- Modules & FFI

I'm interested in adding a way to load compiled modules dynamically
and possibly add a FFI. I say possibly because the more I look into
into the more I realize it's not really helpful for complex stuff and
simple stuff are easy to do with a module. I know Joakim Verona
suggested to look into GObjectIntrospection which he implemented in
his xwidget branch but I have not looked at it yet. It's possible that
this project overlaps a bit with Daimrod's project [1] in which case
we could both work on it, I don't think it would be a problem.

I've also already posted on the mailing list about it [2] if you want
to know more.

- Internal documentation

I have started a "Hacker Guide" on the wiki few month ago [3] hoping
some people would jump in to help me (didn't happen :( ). Anyway Eli
gave me some advice and suggestions to improve it but I didn't have
the time to work on it. Ultimately this could end up in the manual. I
could also clean/fix some things as I document them.

- Support for library <FOO>

By support I mean expose the lib API to Emacs Lisp. I don't know which
lib could be added but I'm sure people here could make some
suggestions. This can be done to extend Elisp or to make it faster (or
both :).

All these are just ideas, you can always suggest something different
for any of them.

As for the credentials, I've already several contributions related to Emacs :
- raw strings patch which you can read more about on my blog [4]
- under-"waving" patch (accepted) [5]
- I've worked on org-mode in last year GSoC

Also I've already signed/sent the copyright assignment, so that's
done. Thought I should mention it.

So, If an emacs hacker is interested in mentoring any of these
projects please manifest yourself :) !


1: http://comments.gmane.org/gmane.emacs.devel/158990
2: http://comments.gmane.org/gmane.emacs.devel/151243
3: http://www.emacswiki.org/emacs/HackerGuide
4: http://definitelyaplug.b0.cx/post/raw-strings/
5: http://comments.gmane.org/gmane.emacs.devel/147958

Re: Google Summer of Code - some ideas

[Aurélien Aptel]

On Sun, Apr 21, 2013 at 6:46 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> So, If an emacs hacker is interested in mentoring any of these
> projects please manifest yourself :) !

I should mention some things :
- being a mentor does not take too much time (GSoC FAQ says 5 hours a
week is a reasonable estimate).
- a project can have multiple mentors (less work for each !)
- I'll be asking most of my questions to the mailling list or the irc channel

As far as I know, anyone sufficiently involved in Emacs can mentor.
You basically have to review my progress, give some advice when I'm
stuck and check up on me if I don't give any news in a long time.

If you're still interested head to the GSoC webpage where you can
register (you need a google account I think)

http://www.google-melange.com/gsoc/homepage/google/gsoc2013

Re: Google Summer of Code - some ideas

[Bastien]

Hi Aurélien,

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Sun, Apr 21, 2013 at 6:46 PM, Aurélien Aptel
> <aurelien.aptel+emacs@gmail.com> wrote:
>> So, If an emacs hacker is interested in mentoring any of these
>> projects please manifest yourself :) !
>
> I should mention some things :
> - being a mentor does not take too much time (GSoC FAQ says 5 hours a
> week is a reasonable estimate).

Well, depends.

I was a not-so-good mentor last year because I had two students to
take care of (including you), and I think it does take time to be a
good one.

There is always this temptation to say "Well, you're a grown up,
just make progress and report blockages.  But I don't think it is
very effective -- I least I think we lost of lot of your precious
time and energy last year but relying to much on this.

> - a project can have multiple mentors (less work for each !)

... if they are well organized enough :)

> - I'll be asking most of my questions to the mailling list or the
> irc channel

That's very good.  When we switched to this way of working last
year I think it was more effective.

> As far as I know, anyone sufficiently involved in Emacs can mentor.
> You basically have to review my progress, give some advice when I'm
> stuck and check up on me if I don't give any news in a long time.

Let me insist on this: this is soft-mentoring and can work, but I
think "deep" mentoring would much more worth--both for you and for
the project.

> If you're still interested head to the GSoC webpage where you can
> register (you need a google account I think)
>
> http://www.google-melange.com/gsoc/homepage/google/gsoc2013

Another thing: one point about GSoC is to "recruit" new contributors
for the project.  I was happy with your contributions, but slightly
disappointed that you did not involve more into Org's community after
it was done: finishing the redmine sync, promoting your library, etc.

It's fine, because obviously there is no obligation here, and maybe
I was a bit naive about this -- I had this picture that finally we'll
work as a close team with all those new GSoC students (3 just for Org!)

Anyway, again, each mentor will have his expectation and I was faulty
of not having time enough for you.  And I'm sure you'll contribute
with great code and ideas, so I'd be happy to see you step onboard
again.

-- 
 Bastien

Re: Google Summer of Code - some ideas

[Aurélien Aptel]

Thank you Bastien for your input. I was hoping that you post.

On Sun, Apr 21, 2013 at 7:25 PM, Bastien <bzg@gnu.org> wrote:
> Another thing: one point about GSoC is to "recruit" new contributors
> for the project.  I was happy with your contributions, but slightly
> disappointed that you did not involve more into Org's community after
> it was done: finishing the redmine sync, promoting your library, etc.

I completely understand. I took a *long* pause after it and I kind of
let you down, sorry. I was (and still am) not satisfied with the
design of org-sync and was reluctant to work more on it. But now that
I have more time I actually started to work again on it. I'll talk
about it more with you off-list.

> It's fine, because obviously there is no obligation here, and maybe
> I was a bit naive about this -- I had this picture that finally we'll
> work as a close team with all those new GSoC students (3 just for Org!)

I'm sorry you felt "used", it was definitely not my intention :)
On the other hand, the gsoc made me more confident and Emacs did gain
a new (albeit very small) contributor !

> Anyway, again, each mentor will have his expectation and I was faulty
> of not having time enough for you.  And I'm sure you'll contribute
> with great code and ideas, so I'd be happy to see you step onboard
> again.

Thank you !

Re: Google Summer of Code - some ideas

[Bastien]

Hi Aurélien,

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I completely understand. I took a *long* pause after it and I kind of
> let you down, sorry. 

Well, no harm at all, as I told you I realized that my expectations
were perhaps a bit too idealistic -- it's just normal to take a break,
and there was no let down, I was just wondering what I could have done
to keep you (and other GSoC-ers) in the loop.

> I was (and still am) not satisfied with the
> design of org-sync and was reluctant to work more on it. But now that
> I have more time I actually started to work again on it. I'll talk
> about it more with you off-list.

Please do!  

>> It's fine, because obviously there is no obligation here, and maybe
>> I was a bit naive about this -- I had this picture that finally we'll
>> work as a close team with all those new GSoC students (3 just for Org!)
>
> I'm sorry you felt "used", it was definitely not my intention :)
> On the other hand, the gsoc made me more confident and Emacs did gain
> a new (albeit very small) contributor !

Yes, hence my reiterated encouragements for this next GSoC, 
after all, this is a learning process!

Best,

-- 
 Bastien

Re: Google Summer of Code - some ideas

[joakim]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Hi,
>
> I would like to work on Emacs itself for this year GSoC. I have
> several projects in mind.
>
> - Modules & FFI
>
> I'm interested in adding a way to load compiled modules dynamically
> and possibly add a FFI. I say possibly because the more I look into
> into the more I realize it's not really helpful for complex stuff and
> simple stuff are easy to do with a module. I know Joakim Verona
> suggested to look into GObjectIntrospection which he implemented in
> his xwidget branch but I have not looked at it yet. It's possible that
> this project overlaps a bit with Daimrod's project [1] in which case
> we could both work on it, I don't think it would be a problem.

I unsurprisingly favour GObjectIntrospection. GIR is based on libffi,
with loads of sugar on.

If you wind up coding some other approach(I'm not opposed to this) we
should at least try to be compatible so we can do code re-use.

For instance, atm, xwgir(my emacs gir interface) can only handle
scalars, not structs. So, that remains to be coded for xwgir, and could
presumably be re-used for lower level libffi calling.

(one needs to make some functions to create structs, fill them in, make
them gc compatible and some other things in the same vein)

>
> I've also already posted on the mailing list about it [2] if you want
> to know more.
>
> - Internal documentation
>
> I have started a "Hacker Guide" on the wiki few month ago [3] hoping
> some people would jump in to help me (didn't happen :( ). Anyway Eli
> gave me some advice and suggestions to improve it but I didn't have
> the time to work on it. Ultimately this could end up in the manual. I
> could also clean/fix some things as I document them.
>
> - Support for library <FOO>
>
> By support I mean expose the lib API to Emacs Lisp. I don't know which
> lib could be added but I'm sure people here could make some
> suggestions. This can be done to extend Elisp or to make it faster (or
> both :).

FYI there is a reluctancy in the community to add dependencies.
I made a libmagic patch that was rejected for instance.

I agree with the patch rejection now, it's better to add something like
xwgir, and then add gir wrappers for your favorite libs. That way the
entire gnome system benefits. Its also easier to accomodate RMS library
gpl compliance symbol request this way.

> All these are just ideas, you can always suggest something different
> for any of them.
>
> As for the credentials, I've already several contributions related to Emacs :
> - raw strings patch which you can read more about on my blog [4]
> - under-"waving" patch (accepted) [5]
> - I've worked on org-mode in last year GSoC
>
> Also I've already signed/sent the copyright assignment, so that's
> done. Thought I should mention it.
>
> So, If an emacs hacker is interested in mentoring any of these
> projects please manifest yourself :) !
>
>
> 1: http://comments.gmane.org/gmane.emacs.devel/1589902: http://comments.gmane.org/gmane.emacs.devel/1512433: http://www.emacswiki.org/emacs/HackerGuide4: http://definitelyaplug.b0.cx/post/raw-strings/5: http://comments.gmane.org/gmane.emacs.devel/147958
>

-- 
Joakim Verona

Re: Google Summer of Code - some ideas

[Stefan Monnier]

> - Modules & FFI
> I'm interested in adding a way to load compiled modules dynamically
> and possibly add a FFI.

That would be great.

> I say possibly because the more I look into into the more I realize
> it's not really helpful for complex stuff and simple stuff are easy to
> do with a module.

Things like libxml2 and libgnutls should use such a mechanism instead of
the one they use currently.

> - Support for library <FOO>
> By support I mean expose the lib API to Emacs Lisp. I don't know which
> lib could be added but I'm sure people here could make
> some suggestions.

Adding support for other libraries is problematic currently, so I'd
rather try and avoid it until we have something like an FFI, so that we
can add libraries without having to recompile Emacs.


        Stefan

Re: Google Summer of Code - some ideas

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 22 Apr 2013 10:34:39 -0400
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> > - Modules & FFI
> > I'm interested in adding a way to load compiled modules dynamically
> > and possibly add a FFI.
> 
> That would be great.
> 
> > I say possibly because the more I look into into the more I realize
> > it's not really helpful for complex stuff and simple stuff are easy to
> > do with a module.
> 
> Things like libxml2 and libgnutls should use such a mechanism instead of
> the one they use currently.

Would loading them dynamically with dlopen and dlsym, similar to what
the MS-Windows build does now, be good enough?  If not, why not?

Re: Google Summer of Code - some ideas

[Stefan Monnier]

>> Things like libxml2 and libgnutls should use such a mechanism instead of
>> the one they use currently.
> Would loading them dynamically with dlopen and dlsym, similar to what
> the MS-Windows build does now, be good enough?

Depends what you mean by "good enough".  By "should" above I meant that
it would be a good basic test of the capability of a new FFI.


        Stefan

Re: Google Summer of Code - some ideas

[Aurélien Aptel]

On Mon, Apr 22, 2013 at 4:34 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> I say possibly because the more I look into into the more I realize
>> it's not really helpful for complex stuff and simple stuff are easy to
>> do with a module.
>
> Things like libxml2 and libgnutls should use such a mechanism instead of
> the one they use currently.

I agree.

> Adding support for other libraries is problematic currently, so I'd
> rather try and avoid it until we have something like an FFI, so that we
> can add libraries without having to recompile Emacs.

I understand. My project could then be:

- add support for compiled modules
- rewrite libxml2 and libgnutls as modules
- complete GIR implementation

It looks like a lot of work so I can't guarantee I'll be done by the
end of the summer but I'll surely be closer to the end than the start
:)

Re: Google Summer of Code - some ideas

[Eli Zaretskii]

> Date: Mon, 22 Apr 2013 19:51:55 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> I understand. My project could then be:
> 
> - add support for compiled modules
> - rewrite libxml2 and libgnutls as modules

libxml2 and GnuTLS are projects, not just libraries developed by Emacs
in-house.  Rewriting them is probably a terrible waste of resources.
Perhaps you meant to write some kind of wrapper for them (if that is
needed, though I cannot see why).

Why isn't dynamic loading the solution we want?

Re: Google Summer of Code - some ideas

[Jérémie Courrèges-Anglas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 22 Apr 2013 19:51:55 +0200
>> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
>> Cc: Emacs development discussions <emacs-devel@gnu.org>
>> 
>> I understand. My project could then be:
>> 
>> - add support for compiled modules
>> - rewrite libxml2 and libgnutls as modules
>
> libxml2 and GnuTLS are projects, not just libraries developed by Emacs
> in-house.  Rewriting them is probably a terrible waste of resources.
> Perhaps you meant to write some kind of wrapper for them (if that is
> needed, though I cannot see why).
>
> Why isn't dynamic loading the solution we want?

I'm probably not knowledgeable enough about this subject, but... is it
really easier to get this (raw dlopening) right?  If my Emacs build
dlopens gnutls with a major 39, and a few days later my system gets
a 40th gnutls major version because of ABI change, then I will run into
problems.  And I can't count on my package tools dependancy system to
notice the ABI change, unless I artificially encode the dependancy on
gnutls or take care to bump my Emacs package to force a rebuild.

Perhaps are there workarounds or methods to implement this safely in
Emacs, but I fear that it would make life harder for some users and
packagers.

I don't know if libffi or GObject Introspection provide a way to solve
this kind of problem, but if they do then that'd be way better, IMHO.

Please correct me if my guesses are wrong.

-- 
Jérémie Courrèges-Anglas
PGP Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

Re: Google Summer of Code - some ideas

[Eli Zaretskii]

> From: jca+emacs@wxcvbn.org (Jérémie Courrèges-Anglas)
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>         monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Mon, 22 Apr 2013 22:16:17 +0200
> 
> I'm probably not knowledgeable enough about this subject, but... is it
> really easier to get this (raw dlopening) right?  If my Emacs build
> dlopens gnutls with a major 39, and a few days later my system gets
> a 40th gnutls major version because of ABI change, then I will run into
> problems.  And I can't count on my package tools dependancy system to
> notice the ABI change, unless I artificially encode the dependancy on
> gnutls or take care to bump my Emacs package to force a rebuild.
> 
> Perhaps are there workarounds or methods to implement this safely in
> Emacs, but I fear that it would make life harder for some users and
> packagers.

Emacs already has a solution for all this, in the MS-Windows build
(which already loads these libraries dynamically).  So what's needed
is to use the same arrangement on other platforms, which currently
link statically against these libraries.

In  nutshell, the version against which Emacs was compiled is recorded
at build time, and then tested against the library at load time.

Re: Google Summer of Code - some ideas

[Jérémie Courrèges-Anglas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: jca+emacs@wxcvbn.org (Jérémie Courrèges-Anglas)
>> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>>         monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Mon, 22 Apr 2013 22:16:17 +0200
>> 
>> I'm probably not knowledgeable enough about this subject, but... is it
>> really easier to get this (raw dlopening) right?  If my Emacs build
>> dlopens gnutls with a major 39, and a few days later my system gets
>> a 40th gnutls major version because of ABI change, then I will run into
>> problems.  And I can't count on my package tools dependancy system to
>> notice the ABI change, unless I artificially encode the dependancy on
>> gnutls or take care to bump my Emacs package to force a rebuild.
>> 
>> Perhaps are there workarounds or methods to implement this safely in
>> Emacs, but I fear that it would make life harder for some users and
>> packagers.
>
> Emacs already has a solution for all this, in the MS-Windows build
> (which already loads these libraries dynamically).  So what's needed
> is to use the same arrangement on other platforms, which currently
> link statically against these libraries.

I think you meant "link dynamically", not "statically".

> In  nutshell, the version against which Emacs was compiled is recorded
> at build time, and then tested against the library at load time.

I trust you that there must have good reasons to do this on "windows",
but I fail to see the bonus on Unix-like distros out there that have
proper package / library / dependancies management.  Startup time?  Not
failing at startup if a shared lib that we linked against isn't present
anymore?

I'm a bit late at this discussion, so please forgive me if this has
already been discussed.

Aurélien: I trimmed you from the CC list since gmail doesn't seem to
like my ipvsh^Wipv6 prefix...
-- 
Jérémie Courrèges-Anglas
PGP Key fingerprint: 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494

Re: Google Summer of Code - some ideas

[Eli Zaretskii]

> From: jca+emacs@wxcvbn.org (Jérémie Courrèges-Anglas)
> Cc: aurelien.aptel+emacs@gmail.com, monnier@iro.umontreal.ca,
>         emacs-devel@gnu.org
> Date: Mon, 22 Apr 2013 22:39:37 +0200
> 
> > Emacs already has a solution for all this, in the MS-Windows build
> > (which already loads these libraries dynamically).  So what's needed
> > is to use the same arrangement on other platforms, which currently
> > link statically against these libraries.
> 
> I think you meant "link dynamically", not "statically".

No, I meant statically.

> I trust you that there must have good reasons to do this on "windows",
> but I fail to see the bonus on Unix-like distros out there that have
> proper package / library / dependancies management.

Then what is this thread all about?  Stefan said:

> > - Modules & FFI
> > I'm interested in adding a way to load compiled modules dynamically
> > and possibly add a FFI.
> 
> That would be great.
> 
> > I say possibly because the more I look into into the more I realize
> > it's not really helpful for complex stuff and simple stuff are easy to
> > do with a module.
> 
> Things like libxml2 and libgnutls should use such a mechanism instead of
> the one they use currently.

If all of this is already solved, what else do we need that would load
libxml2 and libgnutls dynamically?

Re: Google Summer of Code - some ideas

[Eli Zaretskii]

> Date: Tue, 23 Apr 2013 05:38:19 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: aurelien.aptel+emacs@gmail.com, monnier@iro.umontreal.ca,
> 	emacs-devel@gnu.org
> 
> > From: jca+emacs@wxcvbn.org (Jérémie Courrèges-Anglas)
> > Cc: aurelien.aptel+emacs@gmail.com, monnier@iro.umontreal.ca,
> >         emacs-devel@gnu.org
> > Date: Mon, 22 Apr 2013 22:39:37 +0200
> > 
> > > Emacs already has a solution for all this, in the MS-Windows build
> > > (which already loads these libraries dynamically).  So what's needed
> > > is to use the same arrangement on other platforms, which currently
> > > link statically against these libraries.
> > 
> > I think you meant "link dynamically", not "statically".
> 
> No, I meant statically.

Actually, that was misleading.  I meant that the library needs to be
accessible to the linker at link time, either as a static library or
as a shared library.  The Windows build does not have that
restriction.

Re: Google Summer of Code - some ideas

[Lennart Borgman]

On Sun, Apr 21, 2013 at 6:46 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
>
> Hi,
>
> I would like to work on Emacs itself for this year GSoC. I have
> several projects in mind.

Is it perhaps possible now to make a reliable multi major mode support?