Re: Several problems

Re: Several problems

[Ted Zlatanov]

Hello David and Dmitry,

I see the following situation (using the Git mirror):

* Dmitry's commit 4c19675328d0de84cc3181cfc118973f591e8243 broke GnuTLS
  connections and possibly more. I couldn't even do the initial
  connection.

* the followup fix was good for the initial connection, but larger
  amounts of data fail:

commit 85113e79cb92f1d2798db5754a9edd54567d2b0e
Author: Dmitry Antipov <dmantipov@yandex.ru>
Date:   Mon Jul 28 18:50:55 2014 +0400

    Fix Gnus-related issues reported by David Kastrup <dak@gnu.org> in
    <http://lists.gnu.org/archive/html/emacs-devel/2014-07/msg00370.html>.
    * atimer.c (timerfd_callback): Always read expiration data.
    Add comment.
    (turn_on_atimers) [HAVE_TIMERFD]: Disarm timerfd timer.
    * process.c (add_timer_wait_descriptor): Add timer descriptor
    to input_wait_mask and non_process_wait_mask as well.

I've only tested with GnuTLS connections.

The symptom seems to be that GnuTLS will read data, then gets a retry,
and finally hangs forever. I can duplicate it by trying to read the last
200 messages in emacs-devel and I believe even as little as 16K will
trigger the behavior. But if I read the last 1 message only (`C-u 1
RET') then things work.  If this is indeed a timer problem, it will be
hard to replicate but I hope this is enough data.

It would be nice if we could automate the testing of this behavior.  It
can probably be done with a TLS server (`gnutls-serv' for instance) but
in my experience that's a very tricky type of test that tends to throw
spurious errors.  So it may just be something we live with.

I have probably missed the discussion about this timerfd feature (having
just come back from vacation), so please forgive my ignorance, but is
this something that should always be enabled, or can it be configurable?
If it interferes with GnuTLS connections, maybe it should be disabled
for them?

I am following up to emacs-devel instead of opening a bug report because
that's where David Kastrup posted initially.  If you prefer a bug
report, let me know.

Thanks!
Ted

Re: Several problems

[Ted Zlatanov]

I can confirm after testing that reverting Dmitry's timerfd commits in a
local Git repo brings GnuTLS back to usable:

commit 58d06704bf38581a7fc4e7e44702176bd9d65c76
Author: Ted Zlatanov <tzz@lifelogs.com>
Date:   Wed Jul 30 16:04:08 2014 -0400

    Revert "On GNU/Linux, use timerfd for asynchronous timers."
    
    This reverts commit 4c19675328d0de84cc3181cfc118973f591e8243.
    
    Conflicts:
        src/ChangeLog
        test/ChangeLog

commit fbee9036f9d2d786948b7be9447e8d1a6b24030b
Author: Ted Zlatanov <tzz@lifelogs.com>
Date:   Wed Jul 30 16:02:53 2014 -0400

    Revert "Fix Gnus-related issues reported by David Kastrup <dak@gnu.org> in"
    
    This reverts commit 85113e79cb92f1d2798db5754a9edd54567d2b0e.

I hope that's helpful :)  The only conflicts are in the ChangeLog files
as usual, so this is a simple workaround until we figure the problem out.

Thanks
Ted

Re: Several problems

[Dmitry Antipov]

On 07/31/2014 12:02 AM, Ted Zlatanov wrote:

> The symptom seems to be that GnuTLS will read data, then gets a retry,
> and finally hangs forever. I can duplicate it by trying to read the last
> 200 messages in emacs-devel and I believe even as little as 16K will
> trigger the behavior. But if I read the last 1 message only (`C-u 1
> RET') then things work.  If this is indeed a timer problem, it will be
> hard to replicate but I hope this is enough data.

Is it possible to create a simple recipe?  I'm not familiar enough with Gnus, and
would like to see something like in http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16603#5.

> If you prefer a bug report, let me know.

Yes, please.

Dmitry

Gnus encrypted connections hang (was: Several problems)

[Ted Zlatanov]

On Thu, 31 Jul 2014 08:22:08 +0400 Dmitry Antipov <dmantipov@yandex.ru> wrote: 

DA> On 07/31/2014 12:02 AM, Ted Zlatanov wrote:
>> The symptom seems to be that GnuTLS will read data, then gets a retry,
>> and finally hangs forever. I can duplicate it by trying to read the last
>> 200 messages in emacs-devel and I believe even as little as 16K will
>> trigger the behavior. But if I read the last 1 message only (`C-u 1
>> RET') then things work.  If this is indeed a timer problem, it will be
>> hard to replicate but I hope this is enough data.

DA> Is it possible to create a simple recipe?  I'm not familiar enough with Gnus, and
DA> would like to see something like in http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16603#5.

>> If you prefer a bug report, let me know.

DA> Yes, please.

Hmm, I'd like some help here.  I can't replicate the problem easily.
Using Lars' recipe from that bug report, I did:

#+begin_src emacs-lisp
(require 'gnus-group)
(setq gnutls-verbose 9)
(let ((gnus-bug-group-download-format-alist
       '((emacs . "https://code.jquery.com/jquery-2.1.1.js"))))
  (gnus-read-ephemeral-emacs-bug-group 16577))
#+end_src

which locks up for me reliably the *first* time it's run, in a way
similar to what I reported earlier, but there's no output in *Messages*.
It's hitting a static URL with some parameters and transferring a few KB
of data.

OTOH here's a recipe that doesn't lock up, but transfers a fairly large
file quickly.  Because it doesn't lock up, I believe the issue is
somehow tied to interaction with the process.

#+begin_src emacs-lisp
(let ((p (open-gnutls-stream "tls" "tls" "code.jquery.com" "443")))
  (with-current-buffer "tls"
    (process-send-string p "GET /jquery-2.1.1.js\n")))
#+end_src

Ideally I'd like to provide you an exact recipe to replicate what Gnus
does but it's fairly complicated. Lars would be better at figuring this
out. Perhaps the first recipe above is enough. If you or others have
ideas, please let me know.  I'll open the bug as soon as I have a recipe.

I also asked about the reason for introducing this change recently, and
if it can be made configurable and optional, at least. Right now it
makes Gnus and perhaps other parts of Emacs unusable.  Please point me
to the discussion, if any, about this change so I can learn.  If there
was no previous discussion, could you explain a bit?

Thanks
Ted

Re: Gnus encrypted connections hang

[Ted Zlatanov]

Another recipe that doesn't require Gnus to be loaded (:

#+begin_src lisp
(setq newsticker-url-list '(("broke" "https://code.jquery.com/jquery-2.1.1.js" nil nil nil)))
(newsticker-treeview)
#+end_src

It hangs reliably for me when it tries to get the "broke" feed.

Ted

Re: Gnus encrypted connections hang

[Dmitry Antipov]

On 08/01/2014 03:41 AM, Ted Zlatanov wrote:

> Another recipe that doesn't require Gnus to be loaded (:
>
> #+begin_src lisp
> (setq newsticker-url-list '(("broke" "https://code.jquery.com/jquery-2.1.1.js" nil nil nil)))
> (newsticker-treeview)
> #+end_src
>
> It hangs reliably for me when it tries to get the "broke" feed.

Good recipe, thanks.  Please try trunk revision >= 117622.

Dmitry

Re: Gnus encrypted connections hang

[Dmitry Antipov]

On 08/01/2014 03:23 AM, Ted Zlatanov wrote:

> I also asked about the reason for introducing this change recently,
> and if it can be made configurable and optional, at least.

The latter is done in trunk revision 117621.  If you suspect a weird
issue with timers, re-run with EMACS_IGNORE_TIMERFD environment
variable set (and report your problems, of course).

> If there was no previous discussion, could you explain a bit?

Emacs' input reading is very complicated, and asynchronous signals
complicates it even more.  With timerfd and its counterpart signalfd,
it's (in theory) possible to build a completely synchronous event
loop and (hopefully) simplify a lot of things.  Unfortunately this
is GNU/Linux-specific.

Dmitry

Re: Gnus encrypted connections hang

[Ted Zlatanov]

On Fri, 01 Aug 2014 11:13:31 +0400 Dmitry Antipov <dmantipov@yandex.ru> wrote: 

DA> On 08/01/2014 03:23 AM, Ted Zlatanov wrote:
>> I also asked about the reason for introducing this change recently,
>> and if it can be made configurable and optional, at least.

DA> The latter is done in trunk revision 117621.  If you suspect a weird
DA> issue with timers, re-run with EMACS_IGNORE_TIMERFD environment
DA> variable set (and report your problems, of course).

Oh, nice. This morning GMail was very slow, almost unusable. I was able
to determine that it was not a problem from your changes by running with
and without that environment variable. Very helpful!

>> If there was no previous discussion, could you explain a bit?

DA> Emacs' input reading is very complicated, and asynchronous signals
DA> complicates it even more.  With timerfd and its counterpart signalfd,
DA> it's (in theory) possible to build a completely synchronous event
DA> loop and (hopefully) simplify a lot of things.  Unfortunately this
DA> is GNU/Linux-specific.

Thanks for explaining. That's a very sensible change. I recall Mac OS X
has a similar mechanism, "Grand Central Dispatch" maybe? W32 has such
facilities as well. Maybe they'll also be used eventually.

Ted