Fwd: mailclient-send-it usage of browse-url

Fwd: mailclient-send-it usage of browse-url

[chad]

On Apr 15, 2010, at 1:14 AM, <christian.lynbech@tieto.com> <christian.lynbech@tieto.com> wrote:

>>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> Stefan> The problem is that for most users configuring sendmail is difficult
> 
> This is true but it could be that there are ways of using sendmail where
> one would be less reliant of its configuration.
> 
> This was the situation for me when trying to change to using the
> sendmail client. The default value of `mail-specify-envelope-from' is
> nil meaning that we rely on sendmail to generate the envelope and this
> requires sendmail to be setup correctly which apparently isn't the
> default on OSX. However, by setting the variable to true, emacs
> generates the envelope and now mails are getting delivered since they
> have a legitimate envelope which many mailservers wisely seem to
> require.
> 
> The point I am trying ot make here (without being a great sendmail
> expert) that it may be possible to excert enough external control over
> sendmail to ensure delivery, at least for nodes directly connected to
> the internet. If you are in a situation where you need to authenticate,
> other approaches must be used.
> 
> The biggest problem with sendmail (at least on OSX) is that (as David
> mentioned) it may fail silently. Sendmail will in a number of cases
> happily put the mail into the queue and then fail to deliver it later.

As has previously been mentioned, a great number of users have internet 
connections that cannot use sendmail without arcane configuration, because
their network connection blocks the straightforward connection in an attempt
at spam-bot mitigation.  No emacs default variable will fix this.  The user will
not notice that their mail was dropped into a black hole, because the system 
will not inform them -- it will simply and silently drop the mail.  This is not an
abstract concern -- David mentioned that it happened recently that a tester
complained about bug reports going unanswered when in fact his bug reports
were being swallowed by this sort of firewall when the send-mail function was
accidentally set to use sendmail in a beta release.

Until someone writes M-x telepathically-fix-smtp-auth-settings, sendmail cannot
easily be configured by end users.  Until sendmail can easily be configured by 
end users, using `open' is a far, far better choice.

*Chad

Re: mailclient-send-it usage of browse-url

[Lennart Borgman]

On Thu, Apr 15, 2010 at 6:49 PM, chad <yandros@gmail.com> wrote:
>
> Until someone writes M-x telepathically-fix-smtp-auth-settings, sendmail cannot
> easily be configured by end users.  Until sendmail can easily be configured by
> end users, using `open' is a far, far better choice.


Better, but not good enough IMO since it does not work for web mail
and a lot of users might have that. (Perhaps especially those that are
making free software at home. I consider those as an important
target.)

As I have already said (without any response) for those users
browse-url with an intermediate html file (see my prev) reply might be
a better solution. And it probably works for mostly all who have a web
browser.

Re: mailclient-send-it usage of browse-url

[Chad Brown]

On Apr 15, 2010, at 10:15 AM, Lennart Borgman wrote:
> Better, but not good enough IMO since it does not work for web mail
> and a lot of users might have that. (Perhaps especially those that are
> making free software at home. I consider those as an important
> target.)
> 
> As I have already said (without any response) for those users
> browse-url with an intermediate html file (see my prev) reply might be
> a better solution. And it probably works for mostly all who have a web
> browser.

All the web mail systems I know today make it pretty easy to attach a file
to an outgoing message; maybe that would work better than putting the 
(fairly large) text in the clipboard/cut-buffer?

Re: mailclient-send-it usage of browse-url

[Lennart Borgman]

On Thu, Apr 15, 2010 at 7:22 PM, Chad Brown <yandros@mit.edu> wrote:
>
>
> On Apr 15, 2010, at 10:15 AM, Lennart Borgman wrote:
>> Better, but not good enough IMO since it does not work for web mail
>> and a lot of users might have that. (Perhaps especially those that are
>> making free software at home. I consider those as an important
>> target.)
>>
>> As I have already said (without any response) for those users
>> browse-url with an intermediate html file (see my prev) reply might be
>> a better solution. And it probably works for mostly all who have a web
>> browser.
>
> All the web mail systems I know today make it pretty easy to attach a file
> to an outgoing message; maybe that would work better than putting the
> (fairly large) text in the clipboard/cut-buffer?


But what is on the clipboard in this case is the message body for the
bug mail message. I do not think that should be sent as an attachment.

Re: mailclient-send-it usage of browse-url

[Stefan Monnier]

> Better, but not good enough IMO since it does not work for web mail
> and a lot of users might have that.

Maybe it won't quite do the best possible thing, but it does solve the
main problem: using an interface where the user is likely to send a bug
report that will just get silently dropped somewhere along the way.

Furthermore, the current code tries mostly to let the "system" figure out
what MUA to use, so if a user is using webmail as her MUA, then the
"system" should know about it; and if the "system" can't be told to use
that webmail system, then it's a problem in the "system".


        Stefan

Re: mailclient-send-it usage of browse-url

[Lennart Borgman]

On Fri, Apr 16, 2010 at 2:14 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Better, but not good enough IMO since it does not work for web mail
>> and a lot of users might have that.
>
> Maybe it won't quite do the best possible thing, but it does solve the
> main problem: using an interface where the user is likely to send a bug
> report that will just get silently dropped somewhere along the way.
>
> Furthermore, the current code tries mostly to let the "system" figure out
> what MUA to use, so if a user is using webmail as her MUA, then the
> "system" should know about it; and if the "system" can't be told to use
> that webmail system, then it's a problem in the "system".


Yes, it is a system failure, but are we not trying to get around these
here for the purpose of getting the bug reports? I believe the two
step procedure of first displaying a web page and then letting the
user click a mail link there is a little bit more likely to succeed.

I myself has resorted to copy-and-paste when sending bug reports
because of the bad integration with web mail.

Here is how gmail web mail works currently with the best possible w32
system integration (affixa + gmail):

- When "opening" a mailto url affixa catches it and saves it to the
gmail draft folder. And it ask some not very good question first.
- This means there is an empty mail in the draft folder addressed to
bug-gnu-emacs.
- To send the message I have to open the draft folder in the gmail web
interface and then copy the message from the clipboard to it.

Of course I do not do that. I just create a new message from the gmail
web interface.

However even without installing the affixa firefox knows that I am
using gmail. Unfortunately firefox does not care to tell w32 that so
it only works from web pages, not from w32 "open" mailto calls. (That
is why I am suggesting the two step workaround.)

Re: mailclient-send-it usage of browse-url

[Stefan Monnier]

> Yes, it is a system failure, but are we not trying to get around these
> here for the purpose of getting the bug reports?

No.  We have to assume that the "system"'s mail works.
In the past the "system" mail was basically /usr/sbin/sendmail, and
nowadays it's xdg-email/open/....


        Stefan

Re: mailclient-send-it usage of browse-url

[Lennart Borgman]

On Fri, Apr 16, 2010 at 3:08 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Yes, it is a system failure, but are we not trying to get around these
>> here for the purpose of getting the bug reports?
>
> No.  We have to assume that the "system"'s mail works.
> In the past the "system" mail was basically /usr/sbin/sendmail, and
> nowadays it's xdg-email/open/....


I do not think that the systems's mail is well defined today. It looks
to me like web mail interfaces are becoming more and more important.
They live in the web browser and are often well integrated with web
browsers.

Web pages are important enough to be integrated with the desk top
everywhere (though I am not aware of any real standards). Web mail
does not seem to be integrated very well with the desk top anywhere
(maybe because you can go through web pages).

Re: mailclient-send-it usage of browse-url

[Stefan Monnier]

>>> Yes, it is a system failure, but are we not trying to get around these
>>> here for the purpose of getting the bug reports?
>> No.  We have to assume that the "system"'s mail works.
>> In the past the "system" mail was basically /usr/sbin/sendmail, and
>> nowadays it's xdg-email/open/....
> I do not think that the systems's mail is well defined today.

It is.

> It looks to me like web mail interfaces are becoming more and more
> important.  They live in the web browser and are often well integrated
> with web browsers.

Why is that relevant to Emacs?  If they're important and
xdg-email/open/... don't support it, then please send a bug-report to
the corresponding maintainers.


        Stefan

mailclient-send-it usage of browse-url

[Christian Lynbech]

I would like to question the wisdom of letting `mailclient-send-it'
blindly relying on `browse-url' to send mail, especially since this is
set up as the default mail sending client on mac and windows.

After switching back to emacs23 on my powermac, I suddenly started
getting strange behaviour when trying to send mail. I finally pulled
myself together and traced what happened when I tried to send a
message. I was quite surprised to see `browse-url' turn up. I am using
emacs-w3m as my default browser and furthermore have adviced
`browse-url' to automatically switch to my web frame so strange things
was happening.

It is unclear to me why this has never been a problem with Carbon Emacs
and whether there is something screwy in my emacs-w3m setup. I run
identical (and pretty bleeding edge) versions of emacs-w3m and gnus in
both emacs 22 and emacs 23. 

I am certainly not fully on top of how emacs-w3m is supposed to handle
mailto. My `w3m-mailto-url-function' variable is nil and then the
documentation says control goes back to `mail-user-agent' which sort of
looks like the receipe of an infinite loop to me.

It is of course clever to appeal to a system handling of mailto: but
since that OSX comes with a sendmail client, one could argue that we
should do what we do for other unix systems (which I guess uses sendmail
as the default). 

Admittedly, it didn't work out of the box (I needed to also set
`mail-specify-envelope-from' to true) so using mailto may still be a
good idea but shouldn't we then not at least force the choice of
handling agent (ie. call the `open' shell command or the moral
equivalent of `browse-url-safari'). Browse-url seems a shaky foundation
for mail sending given it is anyway not well-specified what it really
points to.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[David Reitter]

Christian,

On Apr 13, 2010, at 3:51 PM, Christian Lynbech wrote:

> I would like to question the wisdom of letting `mailclient-send-it'
> blindly relying on `browse-url' to send mail, especially since this is
> set up as the default mail sending client on mac and windows.


> I am certainly not fully on top of how emacs-w3m is supposed to handle
> mailto. My `w3m-mailto-url-function' variable is nil and then the
> documentation says control goes back to `mail-user-agent' which sort of
> looks like the receipe of an infinite loop to me.

>  Browse-url seems a shaky foundation
> for mail sending given it is anyway not well-specified what it really
> points to.


first of all, the reason why we don't go via `sendmail' is that mail gets swallowed by firewalls and spam filters.  I have recent first-hand experience with this when a user complained about bug reports vanishing after the send mail function got accidentally switched back to sendmail in a beta release of Aquamacs.

I thought of recommending to switch back to `sendmail' when Apple fixed their sendmail mechanism (it was broken at some point, prompting mailclient.el), but the above experience made me discard that thought.

Second, as you state, the "from" field isn't filled in right (at least on the two systems you mention), and third, users will have configured their preferred mail client, where the message is supposed to be archived.

"mailto:" URLs are an established standard ( http://www.ietf.org/rfc/rfc1738.txt ) and most browsers handle it correctly, i.e. by calling the system's mail function.  I know of no more appropriate system-specific mechanism to let the default mail client handle mail on OS X.  I think there was a way to send e-mail on Windows, which was very populars among writers of viruses and trojans.

There are issues with long messages not going through GMail (via "Google Mail Notifier"), but this is due to Google's servers and long URLs, and a bug on their side.

So, the problem lies in the infinite loop that you point out and one could and should do something about it.  Perhaps "mailclient" shouldn't use browse-url in case of an infinite loop, but that may be difficult to detect (asynchronous API in some situations).  
Perhaps w3m (being a third party package) should detect the send-mail-function setting and temporarily bind it to `sendmail-send-it' or so.

Sending mail isn't trivial.  But we really don't want to lose bug reports or user's mail.  The mailclient solution is standards-compliant and does the right thing in the majority of configurations.  I agree: pretty it is not.

Re: mailclient-send-it usage of browse-url

[Christian Lynbech]

>>>>> "David" == David Reitter <david.reitter@gmail.com> writes:

David> So, the problem lies in the infinite loop that you point out and
David> one could and should do something about it.  Perhaps "mailclient"
David> shouldn't use browse-url in case of an infinite loop, but that
David> may be difficult to detect (asynchronous API in some situations).

While it could be that emacs-w3m does not do the right thing with
mailto, it is still so that one cannot really know what `browse-url'
really points to.

Wouldn't it be a solution to just directly do what `browse-url' was
normally expected to do rather than relying on the user or the
third-party package having set everything up correctly?

                             -- Christian

Re: mailclient-send-it usage of browse-url

[Jeff Clough]

Christian Lynbech <christian@defun.dk> writes:

> While it could be that emacs-w3m does not do the right thing with
> mailto, it is still so that one cannot really know what `browse-url'
> really points to.
>
> Wouldn't it be a solution to just directly do what `browse-url' was
> normally expected to do rather than relying on the user or the
> third-party package having set everything up correctly?

I believe the issue is that it is important for Emacs to have a way to
send mail messages even if the user doesn't use it for mail.  Sending
bug reports is a great example of this.

The problem is that there is not one universal way to ask an operating
system "How do I send a mail message?" and get an honest answer.  Under
Unix, you have tools like sendmail that are virtually guaranteed to be
there.  Under Windows, the Mac and other systems, not so much.

A clever way around this is to fire up what Emacs thinks is the system
browser and feed it a mailto: link.  It's a hack, but given the
alternatives it's probably the right thing.  browse-url should work out
of the box on most systems.

If engineering were going to be done to make this better, it should
probably start with finding a way to really ask the question "What is
your mail client?"  I think most of the machines this is an issue for
now have some variable "Mail User Agent" buried in a configuration file
or registry now.  I don't know that that makes the problem really any
easier to solve than it was before.

Jeff

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "Jeff" == Jeff Clough <jeff@chaosphere.com> writes:

Jeff> The problem is that there is not one universal way to ask an operating
Jeff> system "How do I send a mail message?" and get an honest answer.  Under
Jeff> Unix, you have tools like sendmail that are virtually guaranteed to be
Jeff> there.  Under Windows, the Mac and other systems, not so much.

I do not about Windows, but on Mac you can (I believe) rely on the
*presence* of sendmail just as much as on other UNIX like systems.

However, it is true that there is higher risk of meeting a
mis-configured sendmail installation on a Mac because this is not
directly the default route out of the box but I believe that to be more
of an historical accident. Sendmail is however a very flexible tool and
it is probably mostly a case of emacs taking more responsability for
setting up the outgoing mail and relying less on the system end
environment defaults (cf. `mail-specify-envelope-from').

UNIX systems have traditionally a well-configured sendmail since this is
the expectation of many things and sendmail used to be the *only* way
out, however this is not always the case and probably will be less so in
the future as more and more users are using graphical clients such as
thunderbird where you configure mail sending directly. On my work PC,
which also runs Linux, I needed to switch emacs to using smtpmail since
local sendmail was not supported nor maintained by IT.

Jeff> A clever way around this is to fire up what Emacs thinks is the system
Jeff> browser and feed it a mailto: link.  It's a hack, but given the
Jeff> alternatives it's probably the right thing.  browse-url should work out
Jeff> of the box on most systems.

I am not contesting the use of mailto:, merely the use of browse-url.

First of all you somehow need to set up browse-url. If it works out of
the box, it only does so because somebody took the time to figure what
the default should be on a Mac. In fact, looking at the default setup
for browse-url on emacs23 , it does not seem to be working out of the
box on a Mac. "browse-url.el" will search for a number of programs such
as firefox or konqueror but not Safari (which is all you can rely on on
a Mac) and if you do not have any of the browsers in your path (perhaps
because you use some desktop environment menu rather than setting up
path), it fails. If you are using emacs-w3m which I think is a quite
reasonable thing to do, the mailto: hack fails.

Secondly, browse-url is a user command and while few people may be as
multi-frame-frenzied as me, there is none the less a significant risk
that the user has tampered with the command.

Thirdly, we are already doing system specific things. Using mailclient
is done specifically for darwin and windows, otherwise the default is
sendmail. If we know we are on darwin, we might as well default to using
the Safari browser, since that has the same prominence on a Mac as
sendmail has on Linux, or even use the "open" command which is supposed
to be able to match file types to applications and then leave the
matching of functionality to application to the system.

My point is that using `browse-url' is a brittle foundation for an
otherwise clever hack. 

In the very least one should consider making the browsing function used
for mailclient configurable and then default to `browse-url-default-browser',
since we are anyway wanting to invoke the *system behaviour* for mailto:.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[David Reitter]

On Apr 14, 2010, at 9:07 AM, <christian.lynbech@tieto.com> wrote:
> 
> I do not about Windows, but on Mac you can (I believe) rely on the
> *presence* of sendmail just as much as on other UNIX like systems.

Yes, you can rely on the presence, but not on its function.  Mail will be swallowed in many cases, as I have pointed out, and that is true independently of the system.  (I only expect GNU/Linux users to be more savvy about their system / Emacs configuration.)
> 
>  If it works out of
> the box, it only does so because somebody took the time to figure what
> the default should be on a Mac.

The default is flexible.

> In fact, looking at the default setup
> for browse-url on emacs23 , it does not seem to be working out of the
> box on a Mac. "browse-url.el" will search for a number of programs such
> as firefox or konqueror but not Safari (which is all you can rely on on
> a Mac) and if you do not have any of the browsers in your path (perhaps
> because you use some desktop environment menu rather than setting up
> path), it fails.

Can you point to the code that does that?
`browse-url-browser-function' is initialized (defcustom) and set to `browse-url-default-macosx-browser' on a Mac, which is a function that calls "open".  That will do the right thing in all relevant cases of a http:// or mailto: URLs.

The system keeps track of the user-preferred mail client and web browser.

Could you investigate why that doesn't work for you?

> If you are using emacs-w3m which I think is a quite
> reasonable thing to do, the mailto: hack fails.

Why is it a hack again?
It relies on accepted and standardized URLs.  I have previously pointed out the standard to you.

Brittle, maybe.  But less brittle than most other solutions...

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "David" == David Reitter <david.reitter@gmail.com> writes:

David> Can you point to the code that does that?

I probably have been jumping to conclusions based on a too cursory look
at browse-url.el. I will do a more proper investigation on my mac when I
get home and report back.

>> If you are using emacs-w3m which I think is a quite
>> reasonable thing to do, the mailto: hack fails.

David> Why is it a hack again?

I didn't mean any disrespect to the approach. I think I have seen others
referring to it as a hack, even an unpretty one, but I am personally
fine with it. 

I should perhaps also state for the record that this is not about
helping me personally; I have something that works now. 

As I see it, the issue is with the default behaviour for which we have
the goal of providing as robustly as possible a means of sending an
email such that even a novice with an unconfigured emacs on darwin can
send back a bug report.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[chad]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]


On Apr 15, 2010, at 12:46 AM, <christian.lynbech@tieto.com> <christian.lynbech@tieto.com> wrote:
> I probably have been jumping to conclusions based on a too cursory look
> at browse-url.el. I will do a more proper investigation on my mac when I
> get home and report back.
> 
>>> If you are using emacs-w3m which I think is a quite
>>> reasonable thing to do, the mailto: hack fails.
> [...]
> 
> I should perhaps also state for the record that this is not about
> helping me personally; I have something that works now. 
> 
> As I see it, the issue is with the default behaviour for which we have
> the goal of providing as robustly as possible a means of sending an
> email such that even a novice with an unconfigured emacs on darwin can
> send back a bug report.

The default does this, and it is all about helping you.  You somehow broke
the default in your configuration.  Since you haven't told us how you broke
your own configuration, and the only potentially interesting part of the breakage
to *us* is if you broke it doing something that users shouldn't expect to cause the
breakage, then all the rest of the noise about browse-url and the like is just that
-- noise -- until you determine that whatever caused the issue should not have
done so.

Let us know if we can help you!
*Chad


[-- Attachment #2: Type: text/html, Size: 2301 bytes --]

Re: mailclient-send-it usage of browse-url

[Jeff Clough]

<christian.lynbech@tieto.com> writes:

>>>>>> "Jeff" == Jeff Clough <jeff@chaosphere.com> writes:
>
> Jeff> The problem is that there is not one universal way to ask an operating
> Jeff> system "How do I send a mail message?" and get an honest answer.  Under
> Jeff> Unix, you have tools like sendmail that are virtually guaranteed to be
> Jeff> there.  Under Windows, the Mac and other systems, not so much.
>
> I do not about Windows, but on Mac you can (I believe) rely on the
> *presence* of sendmail just as much as on other UNIX like systems.

With Windows your options are mailclient-send-it and hope for the best
or configure smtpmail-send-it to work.  (This is one item on a very long
list of things that caused my most recent move back to Linux.)

> First of all you somehow need to set up browse-url. If it works out of
> the box, it only does so because somebody took the time to figure what
> the default should be on a Mac. In fact, looking at the default setup
> for browse-url on emacs23 , it does not seem to be working out of the
> box on a Mac. "browse-url.el" will search for a number of programs such
> as firefox or konqueror but not Safari (which is all you can rely on on
> a Mac) and if you do not have any of the browsers in your path (perhaps
> because you use some desktop environment menu rather than setting up
> path), it fails. If you are using emacs-w3m which I think is a quite
> reasonable thing to do, the mailto: hack fails.

It does seem lame that it's not looking for Safari, but then I don't
know if it looks for IE on Windows.  This may be a philosophical point
and not a technical one, so for better or worse it just might not
happen.

> Secondly, browse-url is a user command and while few people may be as
> multi-frame-frenzied as me, there is none the less a significant risk
> that the user has tampered with the command.

I think a case can be made that tampering with a browser-oriented thing
shouldn't break an email-oriented thing when it is not obvious to the
end user that that's what's going to happen.  A case could also be made
that if one knows how to tamper with browse-url, they know how to
configure smtp mail correctly.

> My point is that using `browse-url' is a brittle foundation for an
> otherwise clever hack. 

I don't think this is so much that browse-url is brittle, just that a
case can be made that it should work for the more defaulty browsers.
The overall issue of having Emacs get the right answer to "What is your
mail client?" might still ought to be addressed in a way that doesn't
rely on browsers at all, which may be more possible now than it was
however long ago.

Jeff

Re: mailclient-send-it usage of browse-url

[David Reitter]

On Apr 14, 2010, at 9:36 AM, Jeff Clough wrote:
> 
> It does seem lame that it's not looking for Safari, but then I don't
> know if it looks for IE on Windows.  This may be a philosophical point
> and not a technical one, so for better or worse it just might not
> happen.

You are misinformed.  It will use Safari if that's what the system default is.  It is not looking for any specific browser on OS X.  It is doing the right thing and using the browser that the user has configured on their system.  This is actually Safari on the Mac by default.

> I think a case can be made that tampering with a browser-oriented thing
> shouldn't break an email-oriented thing when it is not obvious to the
> end user that that's what's going to happen.  

The command is `browse-url', and it is URL as much as it is BROWSE.  
But you absolutely have a point there - I can see how users may get confused.

> 
> The overall issue of having Emacs get the right answer to "What is your
> mail client?" might still ought to be addressed in a way that doesn't
> rely on browsers at all, 

Once more: as it is now, Emacs does not rely on browsers at all on the Mac in its default configuration.
It uses a system mechanism to handle arbitrary URLs, specifically, mailto: URLs.  This does NOT invoke the browser.  It invokes the system's standard mail client.  

If the user configures a specific browser for Emacs, then yes, we go through the browser, and perhaps that is not the best thing to do in that case.  See my earlier e-mail: bind the browser-function variable to its default before calling browse-url.
 
Are there any cases where this would cause problems?

Re: mailclient-send-it usage of browse-url

[Chad Brown]

Calling `open' is clearly the right thing to do here, and it's what the code does by default.  Christian Lynbech should investigate how/why his installation of the emacs-w3m package caused the unexpected change to the system defaults.  I would not expect emacs-w3m to make those changes itself, so it's possible that Christian configured the change himself and forgot or didn't realize the consequences of the change.  

Without making some configuration change to either Emacs or the system, browse-url on a mailto URL does the right thing - calls open, which calls the configured mail client, which (by default) is Mail.app.  Even in the odd case where Mail.app isn't the right tool for sending mail, it will be obvious to the user what is going on, and only they can figure out the next steps.

*Chad

Re: mailclient-send-it usage of browse-url

[Jeff Clough]

David Reitter <david.reitter@gmail.com> writes:

>> It does seem lame that it's not looking for Safari, but then I don't
>> know if it looks for IE on Windows.  This may be a philosophical point
>> and not a technical one, so for better or worse it just might not
>> happen.
>
> You are misinformed.  It will use Safari if that's what the system default is.  It is not looking for any specific browser on OS X.  It is doing the right thing and using the browser that the user has configured on their system.  This is actually Safari on the Mac by default.

Ah, this makes more sense.

>> The overall issue of having Emacs get the right answer to "What is your
>> mail client?" might still ought to be addressed in a way that doesn't
>> rely on browsers at all, 
>
> Once more: as it is now, Emacs does not rely on browsers at all on the Mac in its default configuration.
> It uses a system mechanism to handle arbitrary URLs, specifically, mailto: URLs.  This does NOT invoke the browser.  It invokes the system's standard mail client.  

So it throws a mailto: URL at the OS and the OS is responsible for
invoking the associated application?  If that is true, then that is very
cool.

Jeff

Re: mailclient-send-it usage of browse-url

[David Reitter]

On Apr 14, 2010, at 12:41 PM, Jeff Clough wrote:
> 
> So it throws a mailto: URL at the OS and the OS is responsible for
> invoking the associated application?  If that is true, then that is very
> cool.

Yup.

It should really work like that on GNU/Linux as well; but we can't rely on the middleware to do this there - someone please correct me if that's wrong.

Re: mailclient-send-it usage of browse-url

[Lennart Borgman]

On Wed, Apr 14, 2010 at 7:46 PM, David Reitter <david.reitter@gmail.com> wrote:
> On Apr 14, 2010, at 12:41 PM, Jeff Clough wrote:
>>
>> So it throws a mailto: URL at the OS and the OS is responsible for
>> invoking the associated application?  If that is true, then that is very
>> cool.
>
> Yup.
>
> It should really work like that on GNU/Linux as well; but we can't rely on the middleware to do this there - someone please correct me if that's wrong.


I think this will not work very well in some cases for those using web
mail. For some of those I believe mailto:-links only works from within
the browser. (At least that is what I see on w32 with gmail.)

A solution to this is writing a temporary html file. This could also
include more instructions, something like this:

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Bug Mail Sender for Emacs</title>
  </head>
  <body style="width:50em;">
    <h1>Bug Mail Sender for Emacs</h1>
    <p>
      The content of your bug report has been placed on the clipboard.
      Click on the link below to start your mail program and then copy
      the content from the clipboard.
    </p>
    <p>
      <a href="mailto:bug-gnu-emacs@gnu.org?subject=The%20subject"
         >bug-gnu-emacs@gnu.org?subject=The subject</a>
    </p>
    <hr style="margin-top:4em" />
    <p>
      <b>Note:</b> Should the above link for some reason fail then
      please start your mail program yourself and send the report the
      address above. Below is the message body again in case you lost
      it in the clipboard:
    </p>
    <textarea cols="72" rows="20" readonly="readonly"
              style="font-size:small; padding:10px;">
Here should be a copy of the users bug report.
    </textarea>
  </body>
</html>

Re: mailclient-send-it usage of browse-url

[Stephen J. Turnbull]

David Reitter writes:

 > It should really work like that on GNU/Linux as well; but we can't
 > rely on the middleware to do this there - someone please correct me
 > if that's wrong.

There's a standard (freedesktop.org, IIRC) wadget called xdg-open,
which may do the right thing.  If not, an RFE would be in order, I
think.

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "Stephen" == Stephen J Turnbull <stephen@xemacs.org> writes:

Stephen> There's a standard (freedesktop.org, IIRC) wadget called xdg-open,
Stephen> which may do the right thing.  If not, an RFE would be in order, I
Stephen> think.

Can you really rely on the freedesktop.org standards being ubiquitously
available on Linux systems? It probably is ok for people running Gnome
or KDE but my hunch is that although the desktop systems are on the
rise, it still doesn't have the same wide reaching aboundance as, say,
sendmail or X11.

I do not know what a wadget is but it does not seem right to rely on
things that somebody (the user or a distribution) needs to install on
the system to be available unless we can be really sure that it really
is abundant. It would of course be good to utilise it if present but
that is another matter.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[Stephen J. Turnbull]

christian.lynbech@tieto.com writes:
 > >>>>> "Stephen" == Stephen J Turnbull <stephen@xemacs.org> writes:
 > 
 > Stephen> There's a standard (freedesktop.org, IIRC) wadget called
 > Stephen> xdg-open, which may do the right thing.  If not, an RFE
 > Stephen> would be in order, I think.
 > 
 > Can you really rely on the freedesktop.org standards being ubiquitously
 > available on Linux systems?

No, but then the only thing you can rely on on "Linux" systems is the
Linux kernel; with libc variants and busybox, it's no longer the case
that you're even guaranteed the basic minimum of GNU userspace.  Even
with GNU/Linux, you know you'll have glibc, but you can't count on
bash, even.

So where xdg-open (or maybe xdg-email as cited in another post) works,
it is presumably the right thing to do.  If it's not there, then you
fall back to Plan B.

And who knows, if Emacs uses it, it might become an industry standard,
as well as a fd.o standard.

 > I do not know what a wadget is

According to Theodore Sturgeon, a [wadget] is "a [ ]ed, inefficient
[stone] age excuse for a [mechanism]".  Even if you don't read ancient
science fiction, the word looks unreliable, doesn't it?<wink>

Re: mailclient-send-it usage of browse-url

[Richard Stallman]

    Can you really rely on the freedesktop.org standards being ubiquitously
    available on Linux systems?

Linux is a kernel.  Are you talking about variants of the GNU system
in which the kernel is Linux?

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

Richard> Linux is a kernel.  Are you talking about variants of the GNU
Richard> system in which the kernel is Linux?

Yes, that was what was in my mind although I should probably have
referred more specifically to desktop systems such as Gnome or
KDE.

Following the freedesktop.org standard is of course not kernel dependent
nor specific to GNU systems even if I personally do not know if there
are examples of systems adhering to that standard outside of the
UNIX-like/X11 context.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[Richard Stallman]

    Yes, that was what was in my mind although I should probably have
    referred more specifically to desktop systems such as Gnome or
    KDE.

I think the things you're talking are about graphical environments.

When you talk about something that is basically the GNU system, would
you please not call it "Linux"?  Calling it that way is unfair to us,
but much worse is that it leads people not to think about their freedom.
See http://www.gnu.org/gnu/gnu-linux-faq.html,
http://www.gnu.org/gnu/the-gnu-project.html, and
http://www.gnu.org/gnu/why-gnu-linux.html.

Re: mailclient-send-it usage of browse-url

[David De La Harpe Golden]

Jeff Clough wrote:


> I don't think this is so much that browse-url is brittle, just that a
> case can be made that it should work for the more defaulty browsers.
> The overall issue of having Emacs get the right answer to "What is your
> mail client?" might still ought to be addressed in a way that doesn't
> rely on browsers at all, which may be more possible now than it was
> however long ago.
> 

FWIW, on freedesktop.org desktops, "xdg-email" opens the user's 
preferred mail composer (and takes args for specifying subject/body etc.)

Last time around, the issue of what happens when the user's preferred 
mail composer desktop setting was then set to emacsclient was of concern 
due to deadlock opportunities.

Re: mailclient-send-it usage of browse-url

[Stefan Monnier]

> However, it is true that there is higher risk of meeting a
> mis-configured sendmail installation on a Mac because this is not

The problem is that for most users configuring sendmail is difficult
(because they can't reliably send email without going though some
SMTP-AUTH step, so they'd have to give their password to the system's
sendmail).
The world of SMTP is very very sad nowadays.


        Stefan

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> The problem is that for most users configuring sendmail is difficult

This is true but it could be that there are ways of using sendmail where
one would be less reliant of its configuration.

This was the situation for me when trying to change to using the
sendmail client. The default value of `mail-specify-envelope-from' is
nil meaning that we rely on sendmail to generate the envelope and this
requires sendmail to be setup correctly which apparently isn't the
default on OSX. However, by setting the variable to true, emacs
generates the envelope and now mails are getting delivered since they
have a legitimate envelope which many mailservers wisely seem to
require.

The point I am trying ot make here (without being a great sendmail
expert) that it may be possible to excert enough external control over
sendmail to ensure delivery, at least for nodes directly connected to
the internet. If you are in a situation where you need to authenticate,
other approaches must be used.

The biggest problem with sendmail (at least on OSX) is that (as David
mentioned) it may fail silently. Sendmail will in a number of cases
happily put the mail into the queue and then fail to deliver it later.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[David Reitter]

On Apr 13, 2010, at 11:22 PM, Christian Lynbech wrote:
> 
> While it could be that emacs-w3m does not do the right thing with
> mailto, it is still so that one cannot really know what `browse-url'
> really points to.

Well, that's Emacs, or Lisp, if you will.  You never know how a function is implemented...  Users/third-party packages can override a lot.

> Wouldn't it be a solution to just directly do what `browse-url' was
> normally expected to do rather than relying on the user or the
> third-party package having set everything up correctly?

Have a look at browse-url.el.  What would this be?
It's entirely system-dependent, so stopping short of replicating the code that determines `browse-url-browser-function', one could

(let ((browse-url-browser-function (default-value 'browse-url-browser-function)))
   (browse-url mailto-url))

If you're set up correctly, this would let the system handle the mailto URL, and the system would give control back to Emacs, because Emacs is your mail client.  And there you have your infinite loop again - it's just a wider loop.

The real problem here lies in the design.  `send-mail-function' is supposed to specify a function that sends e-mail off without further user interaction.  Because we can't do that out-of-the-box, we have to do something at a different level in the "control hierarchy": call the mail client and let the user edit the e-mail again (which is what mailto URLs lead to).
 
Perhaps we need a mechanism to leave send-mail-function nil by default, yet allow bug reporting functions to go through the mail client by default.

Or, one would set up a trivial little script on Savannah to receive a bug report by HTTP and then send the mail off.  That's roughly how most modern software solves the problem.

Re: mailclient-send-it usage of browse-url

[christian.lynbech]

>>>>> "David" == David Reitter <david.reitter@gmail.com> writes:

David> On Apr 13, 2010, at 11:22 PM, Christian Lynbech wrote:
>> 
>> While it could be that emacs-w3m does not do the right thing with
>> mailto, it is still so that one cannot really know what `browse-url'
>> really points to.

David> Well, that's Emacs, or Lisp, if you will.  You never know how a
David> function is implemented...  Users/third-party packages can
David> override a lot.

True, but I think there is a difference between, say,
`calendar-lunar-phases' which does something completely deterministic
(however it is implemented) and then `browse-url' which is just an
interface to a host of different applications. I *know* that
`browse-url' may mean different things on different systems.

>> Wouldn't it be a solution to just directly do what `browse-url' was
>> normally expected to do rather than relying on the user or the
>> third-party package having set everything up correctly?

David> Have a look at browse-url.el.  What would this be?

As I just wrote to Jeff, I would actually it by using Safari directly
when on Darwin. We are anyway doing special things for Darwin (choosing
mailclient over sendmail) and on Linux we are fine with selecting a
system function we know to be there, why couldn't we just do the same
for Darwin? It is not that we want to involve the user in our calling of
a browser, we only wants something that works, and we can rely on Safari
being on a Darwin as much as we can rely on sendmail being on a Linux
system.

Alternatively, we should fix `browse-url-default-browser' to also work
on Darwin and Windows and use that instead of `browse-url'.



------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: mailclient-send-it usage of browse-url

[David Reitter]

On Apr 14, 2010, at 9:20 AM, <christian.lynbech@tieto.com> wrote:
> 
> a browser, we only wants something that works, and we can rely on Safari
> being on a Darwin as much as we can rely on sendmail being on a Linux
> system.

You cannot rely on Safari being the correct browser on Darwin.  You definitely don't want to use it to display HTTP material.

You don't want to call Safari with a mailto URL.  That's slow, because it starts up Safari.  

The current solution goes through "open", which starts the correct mail client without going through a web browser.
On Aquamacs I'm even using the correct Cocoa function to handle URLs rather than calling open to handle a different special case.  But "open" is as correct as any of the solutions circulated.

If you want to force "open" over user's settings for the browser, then that might be more stable, but in your case it would still lead back to Emacs if everything is set up right.