I think I caught it now. This should be fixed by the following commit (just
pushed)
711e14d * lisp/emacs-lisp/package.el: Don't always propagate async errors

Does it work for you now?