[ELPA] Package proposal: gnus-mock

[ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Hi,

I've just pushed the branch scratch/gnus-mock to ELPA, containing a
package I'd like to add there.

It's called "Gnus Mock", and provides a dummy test installation for
Gnus, which you can use for working on Gnus features and testing Gnus
bugs without endangering your own Gnus setup. I'm hoping this makes it
easier to do work on Gnus -- it's often hard to hack on without a full
working installation, and no one wants to risk their own mail on that.

In a nutshell, you run M-x gnus-mock-start, which first copies a dummy
Gnus installation (including mails) into a temporary directory, boots up
a new "emacs -Q", and points all Gnus-related variables at the temporary
directory. You can start Gnus as usual, hack on whatever you like,
accidentally delete data, and when it all gets too messy you can just
kill the secondary Emacs, and start over afresh.

Two questions:

1. I find the data directory like so:

(defconst gnus-mock-data-dir
(file-name-as-directory (expand-file-name
"data"
(file-name-directory load-file-name)))
   "Source directory for Gnus mock data.")

   That seems to work fine, but I wanted to check there wasn't a
   better/safer way to do it.

2. There's a small Python script in there that acts as a dummy sendmail
   command: when you send an email from a Gnus mock installation, it
   hands it off to the Python script, which boomerangs it back to a
   folder in the local installation, so you can send yourself messages
   and see what they look like. The script is called with
   "#!/usr/bin/env python", which I assume will be fine for Unix-y
   platforms, but maybe not work for Windows. I'd like it to work for
   Windows -- does anyone have suggestions for a more portable way of
   doing this?


Future plans include more different pre-installed servers, including
checking for a "dovecot" executable and setting up an nnimap server if
it's found.

Comments very welcome!

Eric

Re: [ELPA] Package proposal: gnus-mock

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Wed, 10 Oct 2018 11:49:56 -0700
> 
> 1. I find the data directory like so:
> 
> (defconst gnus-mock-data-dir
> (file-name-as-directory (expand-file-name
> "data"
> (file-name-directory load-file-name)))
>    "Source directory for Gnus mock data.")

Why not use data-directory?

> 2. There's a small Python script in there that acts as a dummy sendmail
>    command: when you send an email from a Gnus mock installation, it
>    hands it off to the Python script, which boomerangs it back to a
>    folder in the local installation, so you can send yourself messages
>    and see what they look like. The script is called with
>    "#!/usr/bin/env python", which I assume will be fine for Unix-y
>    platforms, but maybe not work for Windows. I'd like it to work for
>    Windows -- does anyone have suggestions for a more portable way of
>    doing this?

Issue a shell command "python SCRIPT"?

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Wed, 10 Oct 2018 11:49:56 -0700
>> 
>> 1. I find the data directory like so:
>> 
>> (defconst gnus-mock-data-dir
>> (file-name-as-directory (expand-file-name
>> "data"
>> (file-name-directory load-file-name)))
>>    "Source directory for Gnus mock data.")
>
> Why not use data-directory?

I think I was unclear -- this is the data directory that ships with the
package itself (and ends up installed in
$package-user-dir/gnus-mock-X/data), not Emacs' data directory.
Basically I need to build an absolute path starting from the place the
package is installed, and `load-file-name' seems like the only way to do
that.

>> 2. There's a small Python script in there that acts as a dummy sendmail
>> command: when you send an email from a Gnus mock installation, it
>> hands it off to the Python script, which boomerangs it back to a
>> folder in the local installation, so you can send yourself messages
>>    and see what they look like. The script is called with
>>    "#!/usr/bin/env python", which I assume will be fine for Unix-y
>>    platforms, but maybe not work for Windows. I'd like it to work for
>>    Windows -- does anyone have suggestions for a more portable way of
>>    doing this?
>
> Issue a shell command "python SCRIPT"?

That would work on Windows? (I know nothing about Windows.)

The problem is that I'm injecting this program at the lowest level
possible, as the value of `sendmail-program', so that
`message-send-mail-with-sendmail' works correctly, and all aspects of
Gnus/message code can be tested. `sendmail-program' is called with
`call-process-region', so it needs to be an executable.

Would there be some way to write an executable wrapper around the
python program? What would that look like on Windows?

Thanks,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Wed, 10 Oct 2018 12:53:51 -0700
> 
> >> 2. There's a small Python script in there that acts as a dummy sendmail
> >> command: when you send an email from a Gnus mock installation, it
> >> hands it off to the Python script, which boomerangs it back to a
> >> folder in the local installation, so you can send yourself messages
> >>    and see what they look like. The script is called with
> >>    "#!/usr/bin/env python", which I assume will be fine for Unix-y
> >>    platforms, but maybe not work for Windows. I'd like it to work for
> >>    Windows -- does anyone have suggestions for a more portable way of
> >>    doing this?
> >
> > Issue a shell command "python SCRIPT"?
> 
> That would work on Windows? (I know nothing about Windows.)

Yes, it should.  'python' is a normal executable, so Windows should
know how to run it.  It's the assumption that the OS understands the
hash-bang signature of a script that's unportable, it is
Unix-specific.

> The problem is that I'm injecting this program at the lowest level
> possible, as the value of `sendmail-program', so that
> `message-send-mail-with-sendmail' works correctly, and all aspects of
> Gnus/message code can be tested. `sendmail-program' is called with
> `call-process-region', so it needs to be an executable.

python.exe is an executable.

> Would there be some way to write an executable wrapper around the
> python program? What would that look like on Windows?

It's possible (with CPython, AFAIK), but why ask users to do something
like that?  Invoking the interpreter directly is much easier, IMO.

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Wed, 10 Oct 2018 12:53:51 -0700
>> 
>> >> 2. There's a small Python script in there that acts as a dummy sendmail
>> >> command: when you send an email from a Gnus mock installation, it
>> >> hands it off to the Python script, which boomerangs it back to a
>> >> folder in the local installation, so you can send yourself messages
>> >>    and see what they look like. The script is called with
>> >>    "#!/usr/bin/env python", which I assume will be fine for Unix-y
>> >>    platforms, but maybe not work for Windows. I'd like it to work for
>> >>    Windows -- does anyone have suggestions for a more portable way of
>> >>    doing this?
>> >
>> > Issue a shell command "python SCRIPT"?
>> 
>> That would work on Windows? (I know nothing about Windows.)
>
> Yes, it should.  'python' is a normal executable, so Windows should
> know how to run it.  It's the assumption that the OS understands the
> hash-bang signature of a script that's unportable, it is
> Unix-specific.
>
>> The problem is that I'm injecting this program at the lowest level
>> possible, as the value of `sendmail-program', so that
>> `message-send-mail-with-sendmail' works correctly, and all aspects of
>> Gnus/message code can be tested. `sendmail-program' is called with
>> `call-process-region', so it needs to be an executable.
>
> python.exe is an executable.
>
>> Would there be some way to write an executable wrapper around the
>> python program? What would that look like on Windows?
>
> It's possible (with CPython, AFAIK), but why ask users to do something
> like that?  Invoking the interpreter directly is much easier, IMO.

I don't want to make the users do anything, I'm just saying that I can't
change `message-send-mail-with-sendmail' to use
`shell-command-on-region' instead of `call-process-region'. I could
override `message-send-mail-with-sendmail' to use a custom function for
Windows users, but I'd rather not do that in case the original function
is exactly what someone is trying to test.

Eric

Re: [ELPA] Package proposal: gnus-mock

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Fri, 12 Oct 2018 10:24:21 -0700
> 
> I don't want to make the users do anything, I'm just saying that I can't
> change `message-send-mail-with-sendmail' to use
> `shell-command-on-region' instead of `call-process-region'.

call-process-region accepts arguments to the program, doesn't it?

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Fri, 12 Oct 2018 10:24:21 -0700
>> 
>> I don't want to make the users do anything, I'm just saying that I can't
>> change `message-send-mail-with-sendmail' to use
>> `shell-command-on-region' instead of `call-process-region'.
>
> call-process-region accepts arguments to the program, doesn't it?

I don't have any control over the call, though. It looks like this:

(apply
 'call-process-region
 (append
  (list (point-min) (point-max) sendmail-program
	nil errbuf nil "-oi")
  message-sendmail-extra-arguments
 (and (null message-sendmail-f-is-evil)
       (list "-f" (message-sendmail-envelope-from)))
 (if (null message-interactive) '("-oem" "-odb"))
  (if resend-to-addresses
      (list resend-to-addresses)
    '("-t"))))

All I've got to work with is the value of `sendmail-program'. I can't
even stick the script name in `message-sendmail-extra-arguments',
because Python will complain about the "-oi" argument.

All I can think of is providing a separate executable for Windows users,
but I don't know what that executable would look like.

Thanks again,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Yuri Khan]

On Sat, Oct 13, 2018 at 2:03 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> All I've got to work with is the value of `sendmail-program'. I can't
> even stick the script name in `message-sendmail-extra-arguments',
> because Python will complain about the "-oi" argument.
>
> All I can think of is providing a separate executable for Windows users,
> but I don't know what that executable would look like.

Pretty much like this, if my Windows scripting has not rusted away
completely yet:

=== fake-sendmail.cmd
@echo off
set ERRORLEVEL=
call python fake-sendmail.py %*
exit /b %ERRORLEVEL%
=== end fake-sendmail.cmd

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Yuri Khan <yurivkhan@gmail.com> writes:

> On Sat, Oct 13, 2018 at 2:03 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> All I've got to work with is the value of `sendmail-program'. I can't
>> even stick the script name in `message-sendmail-extra-arguments',
>> because Python will complain about the "-oi" argument.
>>
>> All I can think of is providing a separate executable for Windows users,
>> but I don't know what that executable would look like.
>
> Pretty much like this, if my Windows scripting has not rusted away
> completely yet:
>
> === fake-sendmail.cmd
> @echo off
> set ERRORLEVEL=
> call python fake-sendmail.py %*
> exit /b %ERRORLEVEL%
> === end fake-sendmail.cmd

Cool! That looks convincing to me :) I'm going to commit it, and see if
anyone complains.

Thanks,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Yuri Khan]

On Sat, Oct 13, 2018 at 2:58 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> > === fake-sendmail.cmd

> Cool! That looks convincing to me :) I'm going to commit it, and see if
> anyone complains.

In the interest of full disclosure, I must say I have not actually run
that script. You might want to find someone with ready access to a
Windows machine and at least average troubleshooting skills to test
it.

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Yuri Khan <yurivkhan@gmail.com> writes:

> On Sat, Oct 13, 2018 at 2:58 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> > === fake-sendmail.cmd
>
>> Cool! That looks convincing to me :) I'm going to commit it, and see if
>> anyone complains.
>
> In the interest of full disclosure, I must say I have not actually run
> that script. You might want to find someone with ready access to a
> Windows machine and at least average troubleshooting skills to test
> it.

Understood!

Re: [ELPA] Package proposal: gnus-mock

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Fri, 12 Oct 2018 12:02:13 -0700
> 
> > call-process-region accepts arguments to the program, doesn't it?
> 
> I don't have any control over the call, though. It looks like this:
> 
> (apply
>  'call-process-region
>  (append
>   (list (point-min) (point-max) sendmail-program
> 	nil errbuf nil "-oi")
>   message-sendmail-extra-arguments

I think the usual way of overcoming this is to use feedmail.el.

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Fri, 12 Oct 2018 12:02:13 -0700
>> 
>> > call-process-region accepts arguments to the program, doesn't it?
>> 
>> I don't have any control over the call, though. It looks like this:
>> 
>> (apply
>>  'call-process-region
>>  (append
>>   (list (point-min) (point-max) sendmail-program
>> 	nil errbuf nil "-oi")
>>   message-sendmail-extra-arguments
>
> I think the usual way of overcoming this is to use feedmail.el.

That's something I haven't looked into before, thank you. I'm going to
try Yuri's solution in the interim, but will look into this as well.

Thanks again,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Stefan Monnier]

> I've just pushed the branch scratch/gnus-mock to ELPA, containing a
> package I'd like to add there.

Looks great!  Feel free to move it to a non-scratch branch (e.g. using
"Version: 0" if you still don't want to release it as a GNU ELPA package).

> It's called "Gnus Mock", and provides a dummy test installation for
> Gnus, which you can use for working on Gnus features and testing Gnus
> bugs without endangering your own Gnus setup. I'm hoping this makes it
> easier to do work on Gnus -- it's often hard to hack on without a full
> working installation, and no one wants to risk their own mail on that.

And here I was, thinking it was mostly designed for use by automated tests.

> (defconst gnus-mock-data-dir
>           (file-name-as-directory (expand-file-name
>                                    "data"
>                                    (file-name-directory load-file-name)))
>    "Source directory for Gnus mock data.")
>
>    That seems to work fine, but I wanted to check there wasn't a
>    better/safer way to do it.

I think that's about as good as it gets currently.
It's pretty reliable/safe in this use case (it gets more tricky if you
need to find such files during byte-compilation of your file).


        Stefan

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've just pushed the branch scratch/gnus-mock to ELPA, containing a
>> package I'd like to add there.
>
> Looks great!  Feel free to move it to a non-scratch branch (e.g. using
> "Version: 0" if you still don't want to release it as a GNU ELPA package).

Cool. If I'm going to work on this in ELPA, shouldn't I just move it
into master?

>> It's called "Gnus Mock", and provides a dummy test installation for
>> Gnus, which you can use for working on Gnus features and testing Gnus
>> bugs without endangering your own Gnus setup. I'm hoping this makes it
>> easier to do work on Gnus -- it's often hard to hack on without a full
>> working installation, and no one wants to risk their own mail on that.
>
> And here I was, thinking it was mostly designed for use by automated tests.

I actually hadn't thought of that. There's no reason you couldn't use it
that way, though currently there's no hook for automatically running
code after the child Emacs process boots up. I'll add an option for
appending code to the init file that's loaded by default.

Of course, Gnus doesn't have much separation between server logic and
user interaction. You'd end up with one of those test routines that
simulates user mouse clicks, ugh.

>> (defconst gnus-mock-data-dir
>>           (file-name-as-directory (expand-file-name
>>                                    "data"
>>                                    (file-name-directory load-file-name)))
>>    "Source directory for Gnus mock data.")
>>
>>    That seems to work fine, but I wanted to check there wasn't a
>>    better/safer way to do it.
>
> I think that's about as good as it gets currently.
> It's pretty reliable/safe in this use case (it gets more tricky if you
> need to find such files during byte-compilation of your file).

Okay. I'm hoping Eli's comment was based on a misunderstanding, we'll
see.

Thanks,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Stefan Monnier]

> Cool. If I'm going to work on this in ELPA, shouldn't I just move it
> into master?

Sure.  Tho you can also put it into an externals/gnus-mock.
Up to you.


        Stefan

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Cool. If I'm going to work on this in ELPA, shouldn't I just move it
>> into master?
>
> Sure.  Tho you can also put it into an externals/gnus-mock.
> Up to you.

I don't have any particular need to keep it separate...

I just noticed that the ELPA .gitignore filters out empty directories (is
that the \#*\# rule?), which messes up the maildirs in this package
(they need empty new/ and tmp/ directories).

Can I get special dispensation for this? Or maybe an external branch is
the way to go after all, if it can have its own .gitignore rules?

Re: [ELPA] Package proposal: gnus-mock

[Yuri Khan]

On Thu, Oct 11, 2018 at 3:59 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> I just noticed that the ELPA .gitignore filters out empty directories (is
> that the \#*\# rule?), which messes up the maildirs in this package
> (they need empty new/ and tmp/ directories).

That is not about .gitignore. Git just doesn’t keep empty directories.

Your options are (a) to put a file there, named so that it won’t
interfere with your code’s functioning; or (b) arrange for empty
directories to be created at installation|configuration|run time.

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Yuri Khan <yurivkhan@gmail.com> writes:

> On Thu, Oct 11, 2018 at 3:59 AM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> I just noticed that the ELPA .gitignore filters out empty directories (is
>> that the \#*\# rule?), which messes up the maildirs in this package
>> (they need empty new/ and tmp/ directories).
>
> That is not about .gitignore. Git just doesn’t keep empty directories.

Interesting, I've never run into that problem before.

> Your options are (a) to put a file there, named so that it won’t
> interfere with your code’s functioning; or (b) arrange for empty
> directories to be created at installation|configuration|run time.

I think I'll go with b, that seems less prone to weirdness.

Thanks for the tip!

Eric

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've just pushed the branch scratch/gnus-mock to ELPA, containing a
>> package I'd like to add there.
>
> Looks great!  Feel free to move it to a non-scratch branch (e.g. using
> "Version: 0" if you still don't want to release it as a GNU ELPA package).
>
>> It's called "Gnus Mock", and provides a dummy test installation for
>> Gnus, which you can use for working on Gnus features and testing Gnus
>> bugs without endangering your own Gnus setup. I'm hoping this makes it
>> easier to do work on Gnus -- it's often hard to hack on without a full
>> working installation, and no one wants to risk their own mail on that.
>
> And here I was, thinking it was mostly designed for use by automated tests.
>
>> (defconst gnus-mock-data-dir
>>           (file-name-as-directory (expand-file-name
>>                                    "data"
>>                                    (file-name-directory load-file-name)))
>>    "Source directory for Gnus mock data.")
>>
>>    That seems to work fine, but I wanted to check there wasn't a
>>    better/safer way to do it.
>
> I think that's about as good as it gets currently.
> It's pretty reliable/safe in this use case (it gets more tricky if you
> need to find such files during byte-compilation of your file).

I went to install this properly (instead of running it from the Elpa
repository), and realized that the data directory that's supposed to
come with it doesn't get installed. The data directory is visible in the
repo:

https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/packages/gnus-mock

But apparently doesn't get installed via the package manager. Is it
getting filtered out by something else?

Thanks,
Eric

Re: [ELPA] Package proposal: gnus-mock

[Stefan Monnier]

> But apparently doesn't get installed via the package manager. Is it
> getting filtered out by something else?

Since it only has a single .el file, the GNU ELPA scripts assume this is
a single-file package.  I installed a patch which should solve this
problem,


        Stefan

Re: [ELPA] Package proposal: gnus-mock

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But apparently doesn't get installed via the package manager. Is it
>> getting filtered out by something else?
>
> Since it only has a single .el file, the GNU ELPA scripts assume this is
> a single-file package.  I installed a patch which should solve this
> problem,

That's a new one for me -- thanks for the fix!