bug#31427: 27.0.50; libxml-parse-html-region stripped a big node

[Noam Postavsky <npostavs@gmail.com>]

tags 31427 + notabug
quit

stardiviner <numbchild@gmail.com> writes:

> As the background here: https://github.com/AdamNiederer/elquery/issues/5
>
> ,----
> | (search "\"content" ; changing to "\"menu" reveals the <div id="menu">
> |         (prin1-to-string (with-temp-buffer
> |                            (insert-string (with-current-buffer (url-retrieve-synchronously "https://nginx.org/en/docs/dirindex.html")
> |                                             (buffer-string)))
> |                            (let ((tree (libxml-parse-html-region (point-min) (point-max))))
> |                              tree))))
> `----
>
> libxml-parse-html-region stipped the <div id="content"> node (which is big).

The problem is that the buffer given by `url-retrieve-synchronously' is
unibyte, then you insert the unibyte buffer-string into a multibyte
buffer (the temp buffer), which gives bad results.

The functions `search' and `insert-string' don't exist, so I
reformulated the code above:

    (let ((case-fold-search nil)
          (str (prin1-to-string
            (with-temp-buffer
              (insert (with-current-buffer (url-retrieve-synchronously "https://nginx.org/en/docs/dirindex.html")
                (buffer-string)))
              (let ((tree (libxml-parse-html-region (point-min) (point-max))))
            tree)))))
      (and (string-match "\"content" str)
           (match-string 0 str)))
    ;=> nil

    (let ((case-fold-search nil)
          (str (prin1-to-string
            (with-temp-buffer
              (set-buffer-multibyte nil) ; <--------- set the temp buffer to unibyte
              (insert (with-current-buffer (url-retrieve-synchronously "https://nginx.org/en/docs/dirindex.html")
                (buffer-string)))
              (let ((tree (libxml-parse-html-region (point-min) (point-max))))
            tree)))))
      (and (string-match "\"content" str)
           (match-string 0 str)))
    ;=> "\"content"

Instead of setting the temp buffer to multibyte, it would probably be
more correct to check the "Content-Type: text/html; charset=utf-8" HTTP
header, and decode it accordingly.  It looks like the libxml-parse-*
functions assume utf-8, so it works out either way in this particular
case.