bug#35969: 26.2, Excorporate

[Collin Day <dcday137@gmail.com>]

Did a git pull and followed instructions above.

There are some issues.  First, I need to run M-x package-install <RET>
excorporate <RET>  four times because I see (each line after each
invocation)

package--with-response-buffer-1:
https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
allowed
package--with-response-buffer-1:
https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
package--with-response-buffer-1:
https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
allowed

After the 4th call it compiles and becomes available.

M-x customize-group <RET> excorporate, set up for no autoconfig right
now, hit apply for current sessions, save not availiable....

M-x excorporate <RET>

enter uname and password

Contacting host: outlook.office365.com:443
error in process filter: exco--parse-xml-in-current-buffer: Server
response is not an XML document
error in process filter: Server response is not an XML documen

*http outlook.office.365.com:443*

HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
request-id: f8e80f64-b51d-4d66-88ef-4ee7f09814f3
X-WSSecurity-Enabled: True
X-WSSecurity-For: Logon
X-FederationTrustTokenIssuerUri: urn:federation:MicrosoftOnline
X-WSSecurity-SymmetricKey-Enabled: True
X-WSSecurity-X509Cert-Enabled: True
X-OAuth-Enabled: True
X-Powered-By: ASP.NET
X-FEServer: SN4PR0401CA0031
WWW-Authenticate: Basic Realm=""
Date: Tue, 09 Jul 2019 22:48:22 GMT
Content-Length: 0

URL-DEBUG
http -> Saw end of headers... ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> url-http-parse-headers called in ( *http outlook.office365.com:443*)
http -> url-http-parse-response called in ( *http outlook.office365.com:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Found existing connection: outlook.office365.com:443 #<process
outlook.office365.com>
http -> Reusing existing connection: outlook.office365.com:443
http -> Marking connection as busy: outlook.office365.com:443
#<process outlook.office365.com>
http -> getting referer from buffer: buffer:#<buffer  *http
outlook.office365.com:443*> target-url:#s(url "https" nil nil
"outlook.office365.com" nil "/EWS/Services.wsdl" nil nil t nil t t)
lastloc:nil
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process
outlook.office365.com>'
http -> url-http-parse-response called in ( *http
outlook.office365.com:443*-535296)
http -> error response: 400
http -> Marking connection as free: outlook.office365.com:443
#<process outlook.office365.com>
http -> Activating callback in buffer ( *http
outlook.office365.com:443*-535296): #[257
"p\303\304\305\306\307 !\310\"\311$\216
@\312=\203$\0\313\301\314\315\316\302\"#\210\317\300\320\"\202/\0\313\301\321\322
#\210\317\300\323\")\207" [fsm-exco--fsm-0 (:identifier ("my email" .
"https://outlook.office365.com/EWS/Excahnge.asmx") :mail-address "my
email" :retrying nil :autodiscovery-urls nil :service-url
"https://outlook.office365.com/EWS/Excahnge.asmx" :service-xml nil
:service-wsdl nil :next-state-after-success nil :failure-message nil
:server-version nil) "https://outlook.office365.com/EWS/Services.wsdl"
make-byte-code 0 "\301\300!\205    \0\302\300!\207" vconcat vector
[buffer-live-p kill-buffer] 2 :error plist-put :failure-message format
"Failed to retrieve %s" fsm-send :unrecoverable-error :service-xml
exco--parse-xml-in-current-buffer :success] 8 "

(fn STATUS)"] ((:peer (:certificates ((:version 3 :serial-number
"0a:48:28:08:de:bc:a4:10:92:0f:87:6a:63:05:a1:e9" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-18" :valid-to "2020-11-18" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ae:b5:9e:3f:4e:c3:72:a8:c0:fc:24:4e:24:b4:d6:8d:99:b0:b9:e1"
:certificate-id
"sha1:05:b8:36:4c:33:e5:63:7e:fd:88:e0:a3:b8:7e:7d:cf:6f:8d:d0:d4")
(:version 3 :serial-number
"01:9e:c1:c6:bd:3f:59:7b:b2:0c:33:38:e5:51:d8:77" :issuer
"C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert Global Root CA"
:valid-from "2015-08-04" :valid-to "2030-08-04" :subject
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ac:13:d5:79:03:66:f3:cc:ff:fe:7c:23:71:a7:61:5d:39:20:89:6e"
:certificate-id
"sha1:81:b6:8d:6c:d2:f2:21:f8:f5:34:e6:77:52:3b:b2:36:bb:a1:dc:56"))
:certificate (:version 3 :serial-number
"0a:48:28:08:de:bc:a4:10:92:0f:87:6a:63:05:a1:e9" :issuer
"C=US,O=DigiCert Inc,CN=DigiCert Cloud Services CA-1" :valid-from
"2018-11-18" :valid-to "2020-11-18" :subject
"C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,CN=outlook.com"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:ae:b5:9e:3f:4e:c3:72:a8:c0:fc:24:4e:24:b4:d6:8d:99:b0:b9:e1"
:certificate-id
"sha1:05:b8:36:4c:33:e5:63:7e:fd:88:e0:a3:b8:7e:7d:cf:6f:8d:d0:d4")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM"
:mac "AEAD")))

So it appears not to work, at least on the current head as of today....

Thanks!

-C

On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137@gmail.com> wrote:
>
> Sorry, I have not had the chance to.  A lot has been going on at my place of employment.  It has crossed my mind, and as soon as I have a chance, I will try what you suggested above.  Thanks, sorry for the inconvenience.
>
> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>>
>> Hi,
>>
>> Were you able to complete building Emacs?  I'd like to know if my patch
>> is needed on top of Andreas's to make Excorporate work through your
>> proxy.
>>
>> I'm not sure why the built version would interfere with the current
>> installation.  If you were doing "make install" to a common prefix, that
>> might explain it.  Instead you can try something like this:
>>
>> cd emacs-master [your github.com checkout]
>> make
>> mkdir test-home
>> HOME=`pwd`/test-home ./src/emacs -Q
>>
>> That will ensure you're only running the built Emacs, and completely
>> ignoring the packages installed in your home directory.
>>
>> Thomas
>>
>> "tenspd137 ." <dcday137@gmail.com> writes:
>>
>> > I haven't tried the first item on the list yet - ie reloading Andreas'
>> > function.  i have tried to build emacs from
>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
>> > official repo from my firewall for whatever reason.  I am sure it is
>> > something here.  Anyway, I was having problems because my built
>> > version (27.0.x) was getting seemingly mixed up with my current
>> > installation.  I can try again later, but right now things are pretty
>> > busy at work.
>> >
>> > Thanks.
>> >
>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> "tenspd137 ." <dcday137@gmail.com> writes:
>> >>
>> >> > So - I am not sure if I did it correctly, but I copied this function
>> >> > with Andreas' changes into a file:
>> >> >
>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
>> >> >   (let* ((process-buffer (current-buffer))
>> >> >          (proc (get-buffer-process process-buffer)))
>> >> >     (goto-char (point-min))
>> >> >     (when (re-search-forward "^\r?\n" nil t)
>> >> >       (backward-char 1)
>> >> >       ;; Saw the end of the headers
>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) (point)))
>> >> >       (url-http-parse-response)
>> >> >       (cond
>> >> >        ((null url-http-response-status)
>> >> >         ;; We got back a headerless malformed response from the
>> >> >         ;; server.
>> >> >         (url-http-activate-callback)
>> >> >         (error "Malformed response from proxy, fail!"))
>> >> >        ((= url-http-response-status 200)
>> >> >         (if (gnutls-available-p)
>> >> >             (condition-case e
>> >> >                 (let ((tls-connection (gnutls-negotiate
>> >> >                                        :process proc
>> >> >                                        :hostname (url-host url-current-object)
>> >> >                                        :verify-error nil)))
>> >> >                   ;; check certificate validity
>> >> >                   (setq tls-connection
>> >> >                         (nsm-verify-connection tls-connection
>> >> >                                                (url-host url-current-object)
>> >> >                                                (url-port url-current-object)))
>> >> >                   (with-current-buffer process-buffer (erase-buffer))
>> >> >                   (set-process-buffer tls-connection process-buffer)
>> >> >                   (setq url-http-after-change-function
>> >> >                         'url-http-wait-for-headers-change-function)
>> >> >                   (set-process-filter tls-connection 'url-http-generic-filter)
>> >> >                   (process-send-string tls-connection
>> >> >                                        ;; Use the non-proxy form of the request
>> >> >                                        (let (url-http-proxy)
>> >> >                                          (url-http-create-request))))
>> >> >               (gnutls-error
>> >> >                (url-http-activate-callback)
>> >> >                (error "gnutls-error: %s" e))
>> >> >               (error
>> >> >                (url-http-activate-callback)
>> >> >                (error "error: %s" e)))
>> >> >           (error "error: gnutls support needed!")))
>> >> >        (t
>> >> >         (url-http-debug "error response: %d" url-http-response-status)
>> >> >         (url-http-activate-callback))))))
>> >> >
>> >> > and then loaded it before running excorporate.  After that, I did M-x
>> >> > excorporate, and the minibuffer returns:  error in process filter:
>> >> > Server response is not an XML document
>> >>
>> >> In this scenario, if you immediately (without restarting Emacs/reloading
>> >> anything) re-run M-x excorporate does it still fail?  I just want to
>> >> make sure that's not a transient failure.  If it does fail the second
>> >> time, can you post the HTTP response from the server?
>> >>
>> >> > When I do the similar test by loading the url-http-create-request with
>> >> > Thomas's changes, I can get a connection and grab my schedule
>> >> > through the proxy.
>> >>
>> >> OK.
>> >>
>> >> > Let me know if I need to try something different.
>> >>
>> >> Are you in a position to build Emacs master tip and retry the experiment
>> >> without patching anything?
>> >>
>> >> Thanks,
>> >> Thomas
>> >>
>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Good to hear that the patch I posted worked!
>> >> >>
>> >> >> Yes, that's the patch that Andreas's commit
>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can tell,
>> >> >> you will not have that in Emacs 26.2.
>> >> >>
>> >> >> Can you try reverting my patch and applying Andreas's patch, and see if
>> >> >> Excorporate still works through the proxy?
>> >> >>
>> >> >> Thanks,
>> >> >> Thomas
>> >> >>
>> >> >> "tenspd137 ." <dcday137@gmail.com> writes:
>> >> >>
>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
>> >> >> > emacs version is 26.2, so I would think it would include the commit
>> >> >> > Andreas is talking about....  I went and looked it up - is this the
>> >> >> > correct commit?
>> >> >> >
>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> >> > index 53798f7..817c5ce 100644
>> >> >> > --- a/lisp/url/url-http.el
>> >> >> > +++ b/lisp/url/url-http.el
>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>> >> >> > retrieval buffer."
>> >> >> > 'url-http-wait-for-headers-change-function)
>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>> >> >> > (process-send-string tls-connection
>> >> >> > - (url-http-create-request)))
>> >> >> > + ;; Use the non-proxy form of the request
>> >> >> > + (let (url-http-proxy)
>> >> >> > + (url-http-create-request))))
>> >> >> > (gnutls-error
>> >> >> > (url-http-activate-callback)
>> >> >> > (error "gnutls-error: %s" e))
>> >> >> >
>> >> >> > Thanks!
>> >> >> >
>> >> >> > -C
>> >> >> >
>> >> >> >
>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab <schwab@linux-m68k.org> wrote:
>> >> >> >>
>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>> >> >> >>
>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>> >> >> >> > index 00803a103a..723d111d58 100644
>> >> >> >> > --- a/lisp/url/url-http.el
>> >> >> >> > +++ b/lisp/url/url-http.el
>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>> >> >> >> >               ;; The request
>> >> >> >> >               (or url-http-method "GET") " "
>> >> >> >> >               (url-http--encode-string
>> >> >> >> > -              (if using-proxy (url-recreate-url url-http-target-url) real-fname))
>> >> >> >> > +              (if (and using-proxy
>> >> >> >> > +                       (not (equal "https" (url-type url-http-target-url))))
>> >> >> >> > +                  (url-recreate-url url-http-target-url)
>> >> >> >> > +                real-fname))
>> >> >> >>
>> >> >> >> That should already be handled by commit 84613dae5c.
>> >> >> >>
>> >> >> >> Andreas.
>> >> >> >>
>> >> >> >> --
>> >> >> >> Andreas Schwab, schwab@linux-m68k.org
>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
>> >> >> >> "And now for something completely different."