bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Brady Trainor]

Hello,

When I setup lsp-java in a Docker to verify setup steps, I get
intermittent errors as package.el tries to install a requirement from
elpa.gnu. If I experiment with different OSes, different emacs versions,
it is difficult to reproduce. With the description below, of a
Dockerfile and an init.el, the issue happens seems to happen more
frequently if I have iterated on restart a few times in a row.

The init.el is as follows.

--8<---------------cut here---------------start------------->8---
(setq package-archives
      '(("gnu" . "https://elpa.gnu.org/packages/")
        ("melpa" . "https://melpa.org/packages/")))

(package-initialize)

(unless package-archive-contents
  (package-refresh-contents))

(package-install 'lsp-java)
--8<---------------cut here---------------end--------------->8---

The contents of the Dockerfile are as follows.

--8<---------------cut here---------------start------------->8---
FROM silex/emacs

ADD .emacs.d /root/.emacs.d

CMD "emacs" "--debug-init"
--8<---------------cut here---------------end--------------->8---

The Dockerfile and the .emacs.d/init.el are in the same directory. Then,
in that directory, I call with the following.

--8<---------------cut here---------------start------------->8---
docker build . -t this:one && docker run --rm -it this:one
--8<---------------cut here---------------end--------------->8---

I then get the following error, but neither 0% nor 100% of the time.

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (file-error "https://elpa.gnu.org/packages/spinner-1.7.3.el" "Bad Request") signal(file-error ("https://elpa.gnu.org/packages/spinner-1.7.3.el" "Bad Request")) package-install-from-archive(#s(package-desc :name spinner :version (1 7 3) :summary "Add spinners and progress-bars to the mode-line for ongoing operations" :reqs nil :kind single :archive "gnu" :dir nil :extras ((:keywords "processes" "mode-line") (:authors ("Artur Malabarba" . "emacs@endlessparentheses.com")) (:maintainer "Artur Malabarba" . "emacs@endlessparentheses.com") (:url . "https://github.com/Malabarba/spinner.el")) :signed nil))
  mapc(package-install-from-archive (#s(package-desc :name request :version (20190621 1622) :summary "Compatible layer for URL request in Emacs" :reqs ((emacs (24 4))) :kind single :archive "melpa" :dir nil :extras ((:commit . "ef9f509b5579f2b529a20e43f7e0f061e641fec5") (:authors ("Takafumi Arakaki <aka.tkf at gmail.com>")) (:maintainer "Takafumi Arakaki <aka.tkf at gmail.com>") (:url . "https://github.com/tkf/emacs-request")) :signed nil) #s(package-desc :name dash :version (20190424 1804) :summary "A modern list library for Emacs" :reqs nil :kind single :archive "melpa" :dir nil :extras ((:commit . "77f3bf40c9c85386a50f2dab3dc950513f6f88bd") (:keywords "lists") (:authors ("Magnar Sveen" . "magnars@gmail.com")) (:maintainer "Magnar Sveen" . "magnars@gmail.com")) :signed nil) #s(package-desc :name dash-functional :version (20180107 1618) :summary "Collection of useful combinators for Emacs Lisp" :reqs ((dash (2 0 0)) (emacs (24))) :kind single :archive "melpa" :dir nil :extras ((:commit . "77f3bf40c9c85386a50f2dab3dc950513f6f88bd") (:keywords "lisp" "functions" "combinators")) :signed nil) #s(package-desc :name ht :version (20190611 2131) :summary "The missing hash table library for Emacs" :reqs ((dash (2 12 0))) :kind single :archive "melpa" :dir nil :extras ((:commit . "5650a8cd190badb49d28d21e72a2f55c9380de7b") (:keywords "hash table" "hash map" "hash") (:authors ("Wilfred Hughes" . "me@wilfred.me.uk")) (:maintainer "Wilfred Hughes" . "me@wilfred.me.uk")) :signed nil) #s(package-desc :name s :version (20180406 808) :summary "The long lost Emacs string manipulation library." :reqs nil :kind single :archive "melpa" :dir nil :extras ((:commit . "03410e6a7a2b11e47e1fea3b7d9899c7df26435e") (:keywords "strings") (:authors ("Magnar Sveen" . "magnars@gmail.com")) (:maintainer "Magnar Sveen" . "magnars@gmail.com")) :signed nil) #s(package-desc :name f :version (20190109 906) :summary "Modern API for working with files and directories" :reqs ((s (1 7 0)) (dash (2 2 0))) :kind single :archive "melpa" :dir nil :extras ((:commit . "8191672377816a1975414cc1f116fd3b94b30bd0") (:keywords "files" "directories") (:authors ("Johan Andersson" . "johan.rejeep@gmail.com")) (:maintainer "Johan Andersson" . "johan.rejeep@gmail.com") (:url . "http://github.com/rejeep/f.el")) :signed nil) #s(package-desc :name markdown-mode :version (20190305 319) :summary "Major mode for Markdown-formatted text" :reqs ((emacs (24 4)) (cl-lib (0 5))) :kind single :archive "melpa" :dir nil :extras ((:commit . "115f77df9755c6a453f3e5d9623ff885d207ea82") (:keywords "markdown" "github flavored markdown" "itex") (:authors ("Jason R. Blevins" . "jblevins@xbeta.org")) (:maintainer "Jason R. Blevins" . "jblevins@xbeta.org") (:url . "https://jblevins.org/projects/markdown-mode/")) :signed nil) #s(package-desc :name spinner :version (1 7 3) :summary "Add spinners and progress-bars to the mode-line for ongoing operations" :reqs nil :kind single :archive "gnu" :dir nil :extras ((:keywords "processes" "mode-line") (:authors ("Artur Malabarba" . "emacs@endlessparentheses.com")) (:maintainer "Artur Malabarba" . "emacs@endlessparentheses.com") (:url . "https://github.com/Malabarba/spinner.el")) :signed nil) #s(package-desc :name lsp-mode :version (20190629 546) :summary "LSP mode" :reqs ((emacs (25 1)) (dash (2 14 1)) (dash-functional (2 14 1)) (f (0 20 0)) (ht (2 0)) (spinner (1 7 3)) (markdown-mode (2 3))) :kind tar :archive "melpa" :dir nil :extras ((:commit . "9f89a3413865343a1a1f794fe45b31f72e56912e") (:keywords "languages") (:authors ("Vibhav Pant, Fangrui Song, Ivan Yonchovski")) (:maintainer "Vibhav Pant, Fangrui Song, Ivan Yonchovski") (:url . "https://github.com/emacs-lsp/lsp-mode")) :signed nil) #s(package-desc :name lsp-java :version (20190623 659) :summary "Java support for lsp-mode" :reqs ((emacs (25 1)) (lsp-mode (6 0)) (markdown-mode (2 3)) (dash (2 14 1)) (f (0 20 0)) (ht (2 0)) (dash-functional (1 2 0)) (request (0 3 0))) :kind tar :archive "melpa" :dir nil :extras ((:commit . "305eb56aecc7180c294cd06ab7a8dd62cba40f70") (:keywords "java") (:url . "https://github.com/emacs-lsp/lsp-java")) :signed nil)))
  package-download-transaction((#s(package-desc :name request :version (20190621 1622) :summary "Compatible layer for URL request in Emacs" :reqs ((emacs (24 4))) :kind single :archive "melpa" :dir nil :extras ((:commit . "ef9f509b5579f2b529a20e43f7e0f061e641fec5") (:authors ("Takafumi Arakaki <aka.tkf at gmail.com>")) (:maintainer "Takafumi Arakaki <aka.tkf at gmail.com>") (:url . "https://github.com/tkf/emacs-request")) :signed nil) #s(package-desc :name dash :version (20190424 1804) :summary "A modern list library for Emacs" :reqs nil :kind single :archive "melpa" :dir nil :extras ((:commit . "77f3bf40c9c85386a50f2dab3dc950513f6f88bd") (:keywords "lists") (:authors ("Magnar Sveen" . "magnars@gmail.com")) (:maintainer "Magnar Sveen" . "magnars@gmail.com")) :signed nil) #s(package-desc :name dash-functional :version (20180107 1618) :summary "Collection of useful combinators for Emacs Lisp" :reqs ((dash (2 0 0)) (emacs (24))) :kind single :archive "melpa" :dir nil :extras ((:commit . "77f3bf40c9c85386a50f2dab3dc950513f6f88bd") (:keywords "lisp" "functions" "combinators")) :signed nil) #s(package-desc :name ht :version (20190611 2131) :summary "The missing hash table library for Emacs" :reqs ((dash (2 12 0))) :kind single :archive "melpa" :dir nil :extras ((:commit . "5650a8cd190badb49d28d21e72a2f55c9380de7b") (:keywords "hash table" "hash map" "hash") (:authors ("Wilfred Hughes" . "me@wilfred.me.uk")) (:maintainer "Wilfred Hughes" . "me@wilfred.me.uk")) :signed nil) #s(package-desc :name s :version (20180406 808) :summary "The long lost Emacs string manipulation library." :reqs nil :kind single :archive "melpa" :dir nil :extras ((:commit . "03410e6a7a2b11e47e1fea3b7d9899c7df26435e") (:keywords "strings") (:authors ("Magnar Sveen" . "magnars@gmail.com")) (:maintainer "Magnar Sveen" . "magnars@gmail.com")) :signed nil) #s(package-desc :name f :version (20190109 906) :summary "Modern API for working with files and directories" :reqs ((s (1 7 0)) (dash (2 2 0))) :kind single :archive "melpa" :dir nil :extras ((:commit . "8191672377816a1975414cc1f116fd3b94b30bd0") (:keywords "files" "directories") (:authors ("Johan Andersson" . "johan.rejeep@gmail.com")) (:maintainer "Johan Andersson" . "johan.rejeep@gmail.com") (:url . "http://github.com/rejeep/f.el")) :signed nil) #s(package-desc :name markdown-mode :version (20190305 319) :summary "Major mode for Markdown-formatted text" :reqs ((emacs (24 4)) (cl-lib (0 5))) :kind single :archive "melpa" :dir nil :extras ((:commit . "115f77df9755c6a453f3e5d9623ff885d207ea82") (:keywords "markdown" "github flavored markdown" "itex") (:authors ("Jason R. Blevins" . "jblevins@xbeta.org")) (:maintainer "Jason R. Blevins" . "jblevins@xbeta.org") (:url . "https://jblevins.org/projects/markdown-mode/")) :signed nil) #s(package-desc :name spinner :version (1 7 3) :summary "Add spinners and progress-bars to the mode-line for ongoing operations" :reqs nil :kind single :archive "gnu" :dir nil :extras ((:keywords "processes" "mode-line") (:authors ("Artur Malabarba" . "emacs@endlessparentheses.com")) (:maintainer "Artur Malabarba" . "emacs@endlessparentheses.com") (:url . "https://github.com/Malabarba/spinner.el")) :signed nil) #s(package-desc :name lsp-mode :version (20190629 546) :summary "LSP mode" :reqs ((emacs (25 1)) (dash (2 14 1)) (dash-functional (2 14 1)) (f (0 20 0)) (ht (2 0)) (spinner (1 7 3)) (markdown-mode (2 3))) :kind tar :archive "melpa" :dir nil :extras ((:commit . "9f89a3413865343a1a1f794fe45b31f72e56912e") (:keywords "languages") (:authors ("Vibhav Pant, Fangrui Song, Ivan Yonchovski")) (:maintainer "Vibhav Pant, Fangrui Song, Ivan Yonchovski") (:url . "https://github.com/emacs-lsp/lsp-mode")) :signed nil) #s(package-desc :name lsp-java :version (20190623 659) :summary "Java support for lsp-mode" :reqs ((emacs (25 1)) (lsp-mode (6 0)) (markdown-mode (2 3)) (dash (2 14 1)) (f (0 20 0)) (ht (2 0)) (dash-functional (1 2 0)) (request (0 3 0))) :kind tar :archive "melpa" :dir nil :extras ((:commit . "305eb56aecc7180c294cd06ab7a8dd62cba40f70") (:keywords "java") (:url . "https://github.com/emacs-lsp/lsp-java")) :signed nil)))
  package-install(lsp-java)
  eval-buffer(#<buffer  *load*> nil "/root/.emacs.d/init.el" nil t)  ; Reading at buffer position 241
  load-with-code-conversion("/root/.emacs.d/init.el" "/root/.emacs.d/init.el" t t)
  load("/root/.emacs.d/init" t t)
  #f(compiled-function () #<bytecode 0x25ead9>)()
  command-line()
  normal-top-level()
--8<---------------cut here---------------end--------------->8---

I hope someone else can reproduce, and/or give advice on how I can
attempt to further isolate the issue.

Notes:

- It is not idempotent. If I restart emacs in the same image, it does
  not happen.
- If I install interactively, it does not happen.
- I tried briefly on macOS, and on Ubuntu in a VPS. It did not happen.
- If I use for example (mapc 'package-install '(spinner lsp-java)), it
  does not happen.

Thank you!

-Brady

Sent from mu4e

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Lars Ingebrigtsen]

Brady Trainor <emacs@tangential.info> writes:

> I then get the following error, but neither 0% nor 100% of the time.

The backtrace looks kinda oddly formatted, but it's basically (I think)...

> Debugger entered--Lisp error: (file-error "https://elpa.gnu.org/packages/spinner-1.7.3.el" "Bad Request") 
>   mapc(package-install-from-archive
>   package-download-transaction(
>   package-install(lsp-java)

So basically you're getting "400 Bad Request" from ELPA.  This wasn't
just a bad network connection?  The FSF moved their servers the other
week -- are you still seeing these problems?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Brady Trainor]

Hello Lars,

On Mon, Jul 08 2019, Lars Ingebrigtsen wrote:

> So basically you're getting "400 Bad Request" from ELPA.  This wasn't
> just a bad network connection?  The FSF moved their servers the other
> week -- are you still seeing these problems?

No. Please note, (package-install 'lsp-java) gets the error, but not
(package-install 'spinner).

I have some adjustments to the previous recipe that seem to give me the
error 100% of the time now.

Dockerfile:

--8<---------------cut here---------------start------------->8---
FROM silex/emacs

ADD .emacs.d /root/.emacs.d

ARG ANY_VAR=unknown

RUN emacs -batch -l ~/.emacs.d/init.el

WORKDIR /root/
CMD "bash"
--8<---------------cut here---------------end--------------->8---

.emacs.d/init.el:

--8<---------------cut here---------------start------------->8---
(setq debug-on-error t)

(setq package-archives
      '(("gnu" . "https://elpa.gnu.org/packages/")
        ("melpa" . "https://melpa.org/packages/")))

(package-initialize)

(unless package-archive-contents
  (package-refresh-contents))

(package-install 'lsp-java)
--8<---------------cut here---------------end--------------->8---

docker command:

--8<---------------cut here---------------start------------->8---
docker build . --build-arg ANY_VAR=$(date +%s) -t this:one && docker run --rm -it this:one
--8<---------------cut here---------------end--------------->8---

So the INCUBATOR_VER has been added to ensure starting emacs as if it's
the first time each time.

Thank you,

-Brady

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Lars Ingebrigtsen]

Brady Trainor <emacs@tangential.info> writes:

>> So basically you're getting "400 Bad Request" from ELPA.  This wasn't
>> just a bad network connection?  The FSF moved their servers the other
>> week -- are you still seeing these problems?
>
> No. Please note, (package-install 'lsp-java) gets the error, but not
> (package-install 'spinner).

Are both packages from the same package repository?

> I have some adjustments to the previous recipe that seem to give me the
> error 100% of the time now.

Can you reproduce the error without Docker?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Brady Trainor]

On Thu, Jul 11, 2019, at 7:01 AM, Lars Ingebrigtsen wrote:

> Are both packages from the same package repository?

No.

> Can you reproduce the error without Docker?

No.

-Brady

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Lars Ingebrigtsen]

"Brady Trainor" <emacs@tangential.info> writes:

> On Thu, Jul 11, 2019, at 7:01 AM, Lars Ingebrigtsen wrote:
>
>> Are both packages from the same package repository?
>
> No.
>
>> Can you reproduce the error without Docker?
>
> No.

So is the problem really that you can't access one of the package
repositories when running within Docker?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Brady Trainor]

On Fri, Jul 12 2019, Lars Ingebrigtsen wrote:

> So is the problem really that you can't access one of the package
> repositories when running within Docker?

Individually, Emacs seems able to access elpa.gnu and melpa. But
something about the transitive dependency seems to break. I've heard
speculation that this could be a race condition, but that's a bit over
my head.

With (package-install 'spinner), it works.

With (mapc 'package-install '(spinner lsp-java)), it works.

With (package-install 'lsp-java), it installs some dependencies
successfully, then gives the failure at spinner that I shared.

And yes, I've only been able to reproduce in Docker. One the one hand,
maybe this is a failure in how Docker is being set up. On the other
hand, maybe this is a pathological example of some imperfection in
Emacs.

Thank you!

-Brady

P.S. My usecase for Docker is mostly to have reproducible setup
instructions for local software, or providing minimal broken
examples. I.e., in the previous, I was trying to check on an editing
environment for Java. And rather than nix, Docker seemed more
approachable for collaborating on recipes.

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Christopher Cooper]

[-- Attachment #1: Type: text/plain, Size: 882 bytes --]

I can reproduce this issue on Arch Linux, Emacs 26.2. To reproduce, I run
emacs -Q and eval the following in *scratch*

(package-initialize)
(package-install 'spinner)

Looks like the actual response says:
> Your browser sent a request that this server could not understand.
> Reason: You're speaking plain HTTP to an SSL-enabled server port.
> Instead use the HTTPS scheme to access this URL, please.

Since this only happens on HTTPS, it's a bit hard to MITM to figre out
what's actually going down. However, Wireshark shows some interesting
stuff: Emacs seems to complete a TLS handshake, then after some packets,
sends a TCP RST. Then, astonishingly, Emacs sends a plaintext HTTP request
_on port 443_ to the server.

I can provide the Wireshark pcap if needed, though I doubt it's too
helpful. Let me know if there's any other way I can help debug.

Thanks,
Christopher Cooper

[-- Attachment #2: Type: text/html, Size: 1206 bytes --]

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Noam Postavsky]

Christopher Cooper <cooperc@ocf.berkeley.edu> writes:

> I can reproduce this issue on Arch Linux, Emacs 26.2.

Does this happen with 26.2.90?  Otherwise you might just be hitting
Bug#36749/36725/34341. 

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Robert Pluim]

>>>>> On Mon, 22 Jul 2019 23:05:51 -0400, Noam Postavsky <npostavs@gmail.com> said:

    Noam> Christopher Cooper <cooperc@ocf.berkeley.edu> writes:
    >> I can reproduce this issue on Arch Linux, Emacs 26.2.

    Noam> Does this happen with 26.2.90?  Otherwise you might just be hitting
    Noam> Bug#36749/36725/34341. 

It could be those, or it could be Bug#36660 (fixed in emacs-27). Does
the second request go to the same IP address as the first?

Robert

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Christopher Cooper]

Sorry for the delay in responding.

> Does this happen with 26.2.90?  Otherwise you might just be hitting
> Bug#36749/36725/34341.

This does appear to be fixed on 26.2.90.

>  Does the second request go to the same IP address as the first?

The IP is the same. So I think this is a dupe of 36749/36725/34341 as Noam said.

Christopher

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Stefan Kangas]

Brady Trainor <emacs@tangential.info> writes:

> On Fri, Jul 12 2019, Lars Ingebrigtsen wrote:
>
>> So is the problem really that you can't access one of the package
>> repositories when running within Docker?
>
> Individually, Emacs seems able to access elpa.gnu and melpa. But
> something about the transitive dependency seems to break. I've heard
> speculation that this could be a race condition, but that's a bit over
> my head.
>
> With (package-install 'spinner), it works.
>
> With (mapc 'package-install '(spinner lsp-java)), it works.
>
> With (package-install 'lsp-java), it installs some dependencies
> successfully, then gives the failure at spinner that I shared.
>
> And yes, I've only been able to reproduce in Docker. One the one hand,
> maybe this is a failure in how Docker is being set up. On the other
> hand, maybe this is a pathological example of some imperfection in
> Emacs.

Can you reproduce this using Emacs 27.1?  The discussion in this thread
seemed to indicate this was due to a bug which is fixed in that version.

Thanks.

bug#36499: 26.2.90; file-error spinner-1.7.3.el, Bad Request

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> Brady Trainor <emacs@tangential.info> writes:
>
>> On Fri, Jul 12 2019, Lars Ingebrigtsen wrote:
>>
>>> So is the problem really that you can't access one of the package
>>> repositories when running within Docker?
>>
>> Individually, Emacs seems able to access elpa.gnu and melpa. But
>> something about the transitive dependency seems to break. I've heard
>> speculation that this could be a race condition, but that's a bit over
>> my head.
>>
>> With (package-install 'spinner), it works.
>>
>> With (mapc 'package-install '(spinner lsp-java)), it works.
>>
>> With (package-install 'lsp-java), it installs some dependencies
>> successfully, then gives the failure at spinner that I shared.
>>
>> And yes, I've only been able to reproduce in Docker. One the one hand,
>> maybe this is a failure in how Docker is being set up. On the other
>> hand, maybe this is a pathological example of some imperfection in
>> Emacs.
>
> Can you reproduce this using Emacs 27.1?  The discussion in this thread
> seemed to indicate this was due to a bug which is fixed in that version.

More information was requested, but none was given within 4 weeks, so
I'm closing this bug.  If this is still an issue, please reply to this
email (use "Reply to all" in your email client) and we can reopen the
bug report.

Best regards,
Stefan Kangas