I think I caught it now. This should be fixed by the following commit (just pushed)
711e14d * lisp/emacs-lisp/package.el: Don't always propagate async errors