bug#35927: 26.2; url-http-end-of-headers sets to 1 when cache is working

bug#35927: 26.2; url-http-end-of-headers sets to 1 when cache is working

[Xu Chunyang]

Today I enable URL cache via

    (setq url-automatic-caching t
          url-cache-expire-time (* 6 3600))

and I notice url-http-end-of-headers isn't the end of the headers when
cache is working. This breaks some of my code, I assume it is the end of
the header so I use (goto-char url-http-end-of-headers) before parsing
the response body. Is this expected or a bug?

    (with-current-buffer (url-retrieve-synchronously "http://example.com")
      (message "=> %S" url-http-end-of-headers))
    "=> #<marker at 347 in  *http example.com:80*>"
    
    (with-current-buffer (url-retrieve-synchronously "http://example.com")
      (message "=> %S" url-http-end-of-headers))
    "=> #<marker at 1 in  *http example.com:80*-325608>"
    
    (with-current-buffer (url-retrieve-synchronously "http://example.com")
      (message "=> %S" url-http-end-of-headers))
    "=> #<marker at 1 in  *http example.com:80*-887290>"

bug#35927: 26.2; url-http-end-of-headers sets to 1 when cache is working

[Basil L. Contovounesios]

Xu Chunyang <mail@xuchunyang.me> writes:

> Today I enable URL cache via
>
>     (setq url-automatic-caching t
>           url-cache-expire-time (* 6 3600))
>
> and I notice url-http-end-of-headers isn't the end of the headers when
> cache is working. This breaks some of my code, I assume it is the end of
> the header so I use (goto-char url-http-end-of-headers) before parsing
> the response body. Is this expected or a bug?
>
>     (with-current-buffer (url-retrieve-synchronously "http://example.com")
>       (message "=> %S" url-http-end-of-headers))
>     "=> #<marker at 347 in  *http example.com:80*>"
>     
>     (with-current-buffer (url-retrieve-synchronously "http://example.com")
>       (message "=> %S" url-http-end-of-headers))
>     "=> #<marker at 1 in  *http example.com:80*-325608>"
>     
>     (with-current-buffer (url-retrieve-synchronously "http://example.com")
>       (message "=> %S" url-http-end-of-headers))
>     "=> #<marker at 1 in  *http example.com:80*-887290>"

I can't seem to reproduce this on emacs-26 or master:

0. HOME=$(mktemp -d) emacs -Q
1. (let ((get (lambda ()
                (with-current-buffer
                    (url-retrieve-synchronously "https://gnu.org")
                  (message "%s" url-http-end-of-headers)))))
     (dotimes (_ 3) (funcall get))
     (setq url-automatic-caching t
           url-cache-expire-time (* 6 3600))
     (dotimes (_ 3) (funcall get)))
2. C-x C-e

Contacting host: gnu.org:443
#<marker at 470 in  *http www.gnu.org:443*>
Contacting host: gnu.org:443
#<marker at 469 in  *http www.gnu.org:443*-93923>
#<marker at 469 in  *http www.gnu.org:443*-669963>
Contacting host: gnu.org:443
#<marker at 469 in  *http www.gnu.org:443*-843863>
#<marker at 469 in  *http www.gnu.org:443*-298952>
#<marker at 469 in  *http www.gnu.org:443*-323128>

What am I doing wrong?

-- 
Basil

In GNU Emacs 26.2.50 (build 8, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2019-05-27 built on thunk
Repository revision: c4d4dcf17e407a3c68e150f22b9756ef6c943070
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description:	Debian GNU/Linux 10 (buster)

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4'
 --config-cache --prefix=/home/blc/.local --program-suffix=26
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 --with-mailutils --with-x-toolkit=lucid --with-modules
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
LCMS2

Important settings:
  value of $LANG: en_IE.UTF-8
  locale-coding-system: utf-8-unix


In GNU Emacs 27.0.50 (build 6, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2019-05-25 built on thunk
Repository revision: ec4b2c888ad12c255accf1bccdac12f173c96047
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description: Debian GNU/Linux 10 (buster)

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
 --prefix=/home/blc/.local --with-mailutils --with-x-toolkit=lucid
 --with-modules --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

bug#35927: 26.2; url-http-end-of-headers sets to 1 when cache is working

[xuchunyang]

On Mon, May 27, 2019, at 11:21 PM, Basil L. Contovounesios wrote:
> Xu Chunyang <mail@xuchunyang.me> writes:
> 
> > Today I enable URL cache via
> >
> >     (setq url-automatic-caching t
> >           url-cache-expire-time (* 6 3600))
> >
> > and I notice url-http-end-of-headers isn't the end of the headers when
> > cache is working. This breaks some of my code, I assume it is the end of
> > the header so I use (goto-char url-http-end-of-headers) before parsing
> > the response body. Is this expected or a bug?
> >
> >     (with-current-buffer (url-retrieve-synchronously "http://example.com")
> >       (message "=> %S" url-http-end-of-headers))
> >     "=> #<marker at 347 in  *http example.com:80*>"
> >     
> >     (with-current-buffer (url-retrieve-synchronously "http://example.com")
> >       (message "=> %S" url-http-end-of-headers))
> >     "=> #<marker at 1 in  *http example.com:80*-325608>"
> >     
> >     (with-current-buffer (url-retrieve-synchronously "http://example.com")
> >       (message "=> %S" url-http-end-of-headers))
> >     "=> #<marker at 1 in  *http example.com:80*-887290>"
> 
> I can't seem to reproduce this on emacs-26 or master:
> 
> 0. HOME=$(mktemp -d) emacs -Q
> 1. (let ((get (lambda ()
>                 (with-current-buffer
>                     (url-retrieve-synchronously "https://gnu.org")
>                   (message "%s" url-http-end-of-headers)))))
>      (dotimes (_ 3) (funcall get))
>      (setq url-automatic-caching t
>            url-cache-expire-time (* 6 3600))
>      (dotimes (_ 3) (funcall get)))
> 2. C-x C-e
> 
> Contacting host: gnu.org:443
> #<marker at 470 in  *http www.gnu.org:443*>
> Contacting host: gnu.org:443
> #<marker at 469 in  *http www.gnu.org:443*-93923>
> #<marker at 469 in  *http www.gnu.org:443*-669963>
> Contacting host: gnu.org:443
> #<marker at 469 in  *http www.gnu.org:443*-843863>
> #<marker at 469 in  *http www.gnu.org:443*-298952>
> #<marker at 469 in  *http www.gnu.org:443*-323128>
> 
> What am I doing wrong?

Can you give http://example.com a try? It seems it's  fine with https://gnu.org.

> -- 
> Basil
> 
> In GNU Emacs 26.2.50 (build 8, x86_64-pc-linux-gnu, X toolkit, Xaw3d 
> scroll bars)
>  of 2019-05-27 built on thunk
> Repository revision: c4d4dcf17e407a3c68e150f22b9756ef6c943070
> Windowing system distributor 'The X.Org Foundation', version 
> 11.0.12003000
> System Description:	Debian GNU/Linux 10 (buster)
> 
> Configured using:
>  'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4'
>  --config-cache --prefix=/home/blc/.local --program-suffix=26
>  --enable-checking=yes,glyphs --enable-check-lisp-object-type
>  --with-mailutils --with-x-toolkit=lucid --with-modules
>  --with-file-notification=yes --with-x'
> 
> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
> GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
> ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
> LCMS2
> 
> Important settings:
>   value of $LANG: en_IE.UTF-8
>   locale-coding-system: utf-8-unix
> 
> 
> In GNU Emacs 27.0.50 (build 6, x86_64-pc-linux-gnu, X toolkit, Xaw3d 
> scroll bars)
>  of 2019-05-25 built on thunk
> Repository revision: ec4b2c888ad12c255accf1bccdac12f173c96047
> Repository branch: master
> Windowing system distributor 'The X.Org Foundation', version 
> 11.0.12003000
> System Description: Debian GNU/Linux 10 (buster)
> 
> Configured using:
>  'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
>  --prefix=/home/blc/.local --with-mailutils --with-x-toolkit=lucid
>  --with-modules --with-file-notification=yes --with-x'
> 
> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY
> INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
> TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
> PDUMPER LCMS2 GMP
>

bug#35927: 26.2; url-http-end-of-headers sets to 1 when cache is working

[Lars Ingebrigtsen]

Xu Chunyang <mail@xuchunyang.me> writes:

> and I notice url-http-end-of-headers isn't the end of the headers when
> cache is working. This breaks some of my code, I assume it is the end of
> the header so I use (goto-char url-http-end-of-headers) before parsing
> the response body. Is this expected or a bug?
>
>     (with-current-buffer (url-retrieve-synchronously "http://example.com")
>       (message "=> %S" url-http-end-of-headers))
>     "=> #<marker at 347 in  *http example.com:80*>"

It's expected, I think.  url-http-end-of-headers is an internal variable
that url-http.el uses to keep track of things, and is not meant to be
used by the callers.

To find the end of the headers, you have to search for a blank line.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no