Re: [maintenance] Compressed JSON files and served file extension?

[Attila Lendvai <attila@lendvai.name>]

TL;DR the filename shouldn't contain the .gz extension, and the HTTP standard is crap ("If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding.").

use curl --compressed

the details:

the Content-Encoding response header instructs the client on how to decode the __transfer payload__ that the server is serving.

i.e. proper HTTP clients should automatically decode the content as instructed by the Content-Encoding response header, or at the very least warn that they do not understand the response encoding...

but that should not happen, because the HTTP request can contain an Accept-Encoding header that tells the server what the client understands, and it defaults to unprocessed raw data ('identity')...

except that the standard allows the server to ignore the Accept-Encoding request header.

well, this is the theory, but both wget and curl don't automatically decode the content. curl at least can be instructed to do so, which arguably should be its default:

curl --compressed https://guix.gnu.org/sources.json | less

--verbose can be used to inspect the reques/response headers (printed to stderr):

curl --verbose https://guix.gnu.org/sources.json >/dev/null
curl --verbose --compressed https://guix.gnu.org/sources.json >/dev/null

here's a detailed discussion of this very question:

https://stackoverflow.com/questions/8364640/how-to-properly-handle-a-gzipped-page-when-using-curl

so, in an ideal world wget and curl should transparently decode the content according to the Content-Encoding response header, and nginx should not respond with a compressed content when the client is not sending an Accept-Encoding request header.

the pragmatic solution is to use curl --compressed in scripts, and/or add it to your ~/.curlrc:

# to automatically decode responses with some of
# the supported Content-Encoding
compressed

HTH,

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“We need people in our lives with whom we can be as open as possible. To have real conversations with people may seem like such a simple, obvious suggestion, but it involves courage and risk.”
	— Thomas Moore (1940–)