bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[bdju via Bug reports for GNU Guix]

I have hit this issue before, but not in a while. I don't know for sure
when the issue came back as it only matters for the initial login or
visit of the offending sites, so if you manage to get past you'll be
good until your cookie expires or gets deleted somehow. I got logged out
of many sites in qutebrowser recently and was made aware of the issue
being back. I also tested in icecat where some sites I still had a
cookie, but I managed to find one with a browser check and it indeed
never completes (can leave it open hours or even more than a day).

Some sites to test with include:
https://www.livechart.me/ (must click login button, main page works)
https://www.gitlab.com/ (again must click log in to start the check)

Since this is happening in both icecat and qutebrowser with different
user agents and everything, I suspect a guix-related issue. I found that
at least one other person on IRC was also experiencing the infinite
browser checks. I use a few sites daily that are now unusable on my Guix
System machine due to these browser checks, so a fix would be very much
appreciated if anyone could figure this out.

guix version:
guix (GNU Guix) eb5e650e09dd096c066278918456f3e989f7b9d9
running Guix System as my distro
Issue first noticed (again) under a week ago, but could be older.
In the past I fixed it by spoofing a firefox user agent the same way in
both browsers, but it seemed like it was not working to do that anymore.
Currently I have both browsers set to default UAs again, also not
working.
There is also a relevant closed issue on qutebrowser's github:
https://github.com/qutebrowser/qutebrowser/issues/7208

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[Maxim Cournoyer]

Hello,

"bdju" <bdju@tilde.team> writes:

> I have hit this issue before, but not in a while. I don't know for sure
> when the issue came back as it only matters for the initial login or
> visit of the offending sites, so if you manage to get past you'll be
> good until your cookie expires or gets deleted somehow. I got logged out
> of many sites in qutebrowser recently and was made aware of the issue
> being back. I also tested in icecat where some sites I still had a
> cookie, but I managed to find one with a browser check and it indeed
> never completes (can leave it open hours or even more than a day).

> Some sites to test with include:
> https://www.livechart.me/ (must click login button, main page works)
> https://www.gitlab.com/ (again must click log in to start the check)
>
> Since this is happening in both icecat and qutebrowser with different
> user agents and everything, I suspect a guix-related issue. I found that
> at least one other person on IRC was also experiencing the infinite
> browser checks. I use a few sites daily that are now unusable on my Guix
> System machine due to these browser checks, so a fix would be very much
> appreciated if anyone could figure this out.

I've had that too with Gitlab when using Icecat.  Sadly, it has nothing
to do with Guix but with how Cloudfare and the website identifies
browsers.

For example, I had found out that by using a Windows Firefox 83 user
agent, I was able to login into Gitlab (using this plugin:
https://gitlab.com/ntninja/user-agent-switcher).  I reported the issue
to Gitlab, and they could apparently fixed it on their side (not yet
deployed) [0]

[0]  https://gitlab.com/gitlab-org/gitlab/-/issues/345328

I think other sites or CloudFare must be similarly faulty, or require
fingerprinting which is guarded against out-of-the-box in IceCat.

Closing, as I doubt Guix has something to do with it.  If you find
something to the contrary, let us know!

-- 
Thanks,
Maxim

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[bdju via Bug reports for GNU Guix]

On Wed Dec 14, 2022 at 10:32 PM CST, Maxim Cournoyer wrote:
> Hello,
>
> "bdju" <bdju@tilde.team> writes:
>
> > I have hit this issue before, but not in a while. I don't know for sure
> > when the issue came back as it only matters for the initial login or
> > visit of the offending sites, so if you manage to get past you'll be
> > good until your cookie expires or gets deleted somehow. I got logged out
> > of many sites in qutebrowser recently and was made aware of the issue
> > being back. I also tested in icecat where some sites I still had a
> > cookie, but I managed to find one with a browser check and it indeed
> > never completes (can leave it open hours or even more than a day).
>
> > Some sites to test with include:
> > https://www.livechart.me/ (must click login button, main page works)
> > https://www.gitlab.com/ (again must click log in to start the check)
> >
> > Since this is happening in both icecat and qutebrowser with different
> > user agents and everything, I suspect a guix-related issue. I found that
> > at least one other person on IRC was also experiencing the infinite
> > browser checks. I use a few sites daily that are now unusable on my Guix
> > System machine due to these browser checks, so a fix would be very much
> > appreciated if anyone could figure this out.
>
> I've had that too with Gitlab when using Icecat.  Sadly, it has nothing
> to do with Guix but with how Cloudfare and the website identifies
> browsers.
>
> For example, I had found out that by using a Windows Firefox 83 user
> agent, I was able to login into Gitlab (using this plugin:
> https://gitlab.com/ntninja/user-agent-switcher).  I reported the issue
> to Gitlab, and they could apparently fixed it on their side (not yet
> deployed) [0]
>
> [0]  https://gitlab.com/gitlab-org/gitlab/-/issues/345328
>
> I think other sites or CloudFare must be similarly faulty, or require
> fingerprinting which is guarded against out-of-the-box in IceCat.
>
> Closing, as I doubt Guix has something to do with it.  If you find
> something to the contrary, let us know!
>
> -- 
> Thanks,
> Maxim

I too have fixed it in the past by switching my user agent. I opened and
closed a similar bug some months or years back. That solution stopped
working. I have consulted with folks in the qutebrowser IRC about this
issue several times and it is not affecting everyone there, so it
definitely seems guix-related to me. Something about our packages must
make the browser(s) look odd to these infernal browser checks. I know
that the issue is cloudflare essentially bullying those of us with more
niche setups, but cloudflare has sadly infected most of the Web, so we
have to play their game. I fear the number of qutebrowser users on guix
is in the single digits and so often I'm running into odd problems with
no solution, that again rarely seems to affect people on the other
distros when I go to ask for help. I think our Qt version is almost
always old which probably doesn't help.
I doubt I have the proper evidence to convince you this is guix's fault,
but just know that if you close this issue my problem may never be
solved (although it may never get solved either way honestly).

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[Maxim Cournoyer]

Hi,

"bdju" <bdju@tilde.team> writes:

[...]

>> I've had that too with Gitlab when using Icecat.  Sadly, it has nothing
>> to do with Guix but with how Cloudfare and the website identifies
>> browsers.
>>
>> For example, I had found out that by using a Windows Firefox 83 user
>> agent, I was able to login into Gitlab (using this plugin:
>> https://gitlab.com/ntninja/user-agent-switcher).  I reported the issue
>> to Gitlab, and they could apparently fixed it on their side (not yet
>> deployed) [0]
>>
>> [0]  https://gitlab.com/gitlab-org/gitlab/-/issues/345328
>>
>> I think other sites or CloudFare must be similarly faulty, or require
>> fingerprinting which is guarded against out-of-the-box in IceCat.
>>
>> Closing, as I doubt Guix has something to do with it.  If you find
>> something to the contrary, let us know!

> I too have fixed it in the past by switching my user agent. I opened and
> closed a similar bug some months or years back. That solution stopped
> working. I have consulted with folks in the qutebrowser IRC about this
> issue several times and it is not affecting everyone there, so it
> definitely seems guix-related to me. Something about our packages must
> make the browser(s) look odd to these infernal browser checks.

That's a good lead; could you please test qutebrowser in Guix vs
qutebrowser on another distribution yourself and confirm this hypothesis
(that it works elsewhere?), and post your finings here?  If you can do
that and post your finding, I we can reopen the ticket, as we'll have
something actionable to look at.

-- 
Thanks,
Maxim

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[bdju via Bug reports for GNU Guix]

On Thu Dec 15, 2022 at 7:26 AM CST, Maxim Cournoyer wrote:
> That's a good lead; could you please test qutebrowser in Guix vs
> qutebrowser on another distribution yourself and confirm this hypothesis
> (that it works elsewhere?), and post your finings here?  If you can do
> that and post your finding, I we can reopen the ticket, as we'll have
> something actionable to look at.
>

I'm able to pass the browser check in a matter of seconds and log in to
GitLab from my PineBook Pro running postmarketOS (Alpine-based) using
qutebrowser.

Here is the :version output from the working qutebrowser:
------------------------------------------------------------------------
qutebrowser v2.5.2
Git commit:
Backend: QtWebEngine 5.15.3, based on Chromium 87.0.4280.144
Qt: 5.15.6

CPython: 3.11.1
PyQt: 5.15.7

sip: 6.6.2
colorama: 0.4.6
jinja2: 3.1.2
pygments: 2.13.0
yaml: 6.0
adblock: 0.6.0
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebEngine: 5.15.6
PyQt5.QtWebKitWidgets: no
pdf.js: no
sqlite: 3.40.0
QtNetwork SSL: OpenSSL 3.0.7 1 Nov 2022

Style: QFusionStyle
Platform plugin: wayland
OpenGL: OpenGL ES
Platform: Linux-5.18.0-aarch64-with, 64bit
Linux distribution: postmarketOS edge (alpine)
Frozen: False
Imported from /usr/lib/python3.11/site-packages/qutebrowser
Using Python from /usr/bin/python3
Qt library executable path: /usr/lib/qt5/libexec, data path: /usr/share/qt5

Paths:
cache: /home/bdju/.cache/qutebrowser
config: /home/bdju/.config/qutebrowser
data: /home/bdju/.local/share/qutebrowser
runtime: /run/user/10001/qutebrowser
system data: /usr/share/qutebrowser

Autoconfig loaded: no
Config.py: /home/bdju/.config/qutebrowser/config.py has been loaded
Uptime: 0:02:55
------------------------------------------------------------------------

guix version info for comparison:
------------------------------------------------------------------------
qutebrowser v2.5.2
Git commit: 
Backend: QtWebEngine 5.15.5, based on Chromium 87.0.4280.144
Qt: 5.15.5

CPython: 3.9.9
PyQt: 5.15.5

sip: 6.6.1
colorama: 0.4.4
jinja2: 3.1.1
pygments: 2.12.0
yaml: 6.0
adblock: no
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebEngine: 5.15.5
PyQt5.QtWebKitWidgets: no
pdf.js: no
sqlite: 3.36.0
QtNetwork SSL: OpenSSL 1.1.1s  1 Nov 2022

Style: QFusionStyle
Platform plugin: xcb
OpenGL: Intel Open Source Technology Center, 3.0 Mesa 21.3.8
Platform: Linux-6.0.8-gnu-x86_64-with-glibc2.33, 64bit
Linux distribution: Guix System (unknown)
Frozen: False
Imported from
/gnu/store/675pkhpvbvi0yai1bggkkaj3h1xy2xrb-qutebrowser-2.5.2/lib/python3.9/site-packages/qutebrowser
Using Python from
/gnu/store/avmnzy8djp42r5926cwznz6ls9gablf8-python-wrapper-3.9.9/bin/python
Qt library executable path:
/gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/lib/qt5/libexec,
data path:
/gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/share/qt5
OS Version: 

--- /etc/os-release ---
NAME="Guix System"
ID=guix
PRETTY_NAME="Guix System"
LOGO=guix-icon
DOCUMENTATION_URL="https://guix.gnu.org/en/manual"

Paths:
cache: /home/brad/.cache/qutebrowser
config: /home/brad/.config/qutebrowser
data: /home/brad/.local/share/qutebrowser
runtime: /run/user/1000/qutebrowser

Autoconfig loaded: no
Config.py: /home/brad/.config/qutebrowser/config.py has been loaded
Uptime: 1 day, 3:11:30
------------------------------------------------------------------------

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[Maxim Cournoyer]

reopen 59546
quit

Hello,

"bdju" <bdju@tilde.team> writes:

> On Thu Dec 15, 2022 at 7:26 AM CST, Maxim Cournoyer wrote:
>> That's a good lead; could you please test qutebrowser in Guix vs
>> qutebrowser on another distribution yourself and confirm this hypothesis
>> (that it works elsewhere?), and post your finings here?  If you can do
>> that and post your finding, I we can reopen the ticket, as we'll have
>> something actionable to look at.
>>
>
> I'm able to pass the browser check in a matter of seconds and log in to
> GitLab from my PineBook Pro running postmarketOS (Alpine-based) using
> qutebrowser.
>
> Here is the :version output from the working qutebrowser:
> ------------------------------------------------------------------------
> qutebrowser v2.5.2
> Git commit:
> Backend: QtWebEngine 5.15.3, based on Chromium 87.0.4280.144
> Qt: 5.15.6
>
> CPython: 3.11.1
> PyQt: 5.15.7
>
> sip: 6.6.2
> colorama: 0.4.6
> jinja2: 3.1.2
> pygments: 2.13.0
> yaml: 6.0
> adblock: 0.6.0
> PyQt5.QtWebEngineWidgets: yes
> PyQt5.QtWebEngine: 5.15.6
> PyQt5.QtWebKitWidgets: no
> pdf.js: no
> sqlite: 3.40.0
> QtNetwork SSL: OpenSSL 3.0.7 1 Nov 2022
>
> Style: QFusionStyle
> Platform plugin: wayland
> OpenGL: OpenGL ES
> Platform: Linux-5.18.0-aarch64-with, 64bit
> Linux distribution: postmarketOS edge (alpine)
> Frozen: False
> Imported from /usr/lib/python3.11/site-packages/qutebrowser
> Using Python from /usr/bin/python3
> Qt library executable path: /usr/lib/qt5/libexec, data path: /usr/share/qt5
>
> Paths:
> cache: /home/bdju/.cache/qutebrowser
> config: /home/bdju/.config/qutebrowser
> data: /home/bdju/.local/share/qutebrowser
> runtime: /run/user/10001/qutebrowser
> system data: /usr/share/qutebrowser
>
> Autoconfig loaded: no
> Config.py: /home/bdju/.config/qutebrowser/config.py has been loaded
> Uptime: 0:02:55
> ------------------------------------------------------------------------
>
> guix version info for comparison:
> ------------------------------------------------------------------------
> qutebrowser v2.5.2
> Git commit: 
> Backend: QtWebEngine 5.15.5, based on Chromium 87.0.4280.144
> Qt: 5.15.5
>
> CPython: 3.9.9
> PyQt: 5.15.5
>
> sip: 6.6.1
> colorama: 0.4.4
> jinja2: 3.1.1
> pygments: 2.12.0
> yaml: 6.0
> adblock: no
> PyQt5.QtWebEngineWidgets: yes
> PyQt5.QtWebEngine: 5.15.5
> PyQt5.QtWebKitWidgets: no
> pdf.js: no
> sqlite: 3.36.0
> QtNetwork SSL: OpenSSL 1.1.1s  1 Nov 2022
>
> Style: QFusionStyle
> Platform plugin: xcb
> OpenGL: Intel Open Source Technology Center, 3.0 Mesa 21.3.8
> Platform: Linux-6.0.8-gnu-x86_64-with-glibc2.33, 64bit
> Linux distribution: Guix System (unknown)
> Frozen: False
> Imported from
> /gnu/store/675pkhpvbvi0yai1bggkkaj3h1xy2xrb-qutebrowser-2.5.2/lib/python3.9/site-packages/qutebrowser
> Using Python from
> /gnu/store/avmnzy8djp42r5926cwznz6ls9gablf8-python-wrapper-3.9.9/bin/python
> Qt library executable path:
> /gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/lib/qt5/libexec,
> data path:
> /gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/share/qt5
> OS Version: 
>
> --- /etc/os-release ---
> NAME="Guix System"
> ID=guix
> PRETTY_NAME="Guix System"
> LOGO=guix-icon
> DOCUMENTATION_URL="https://guix.gnu.org/en/manual"
>
> Paths:
> cache: /home/brad/.cache/qutebrowser
> config: /home/brad/.config/qutebrowser
> data: /home/brad/.local/share/qutebrowser
> runtime: /run/user/1000/qutebrowser
>
> Autoconfig loaded: no
> Config.py: /home/brad/.config/qutebrowser/config.py has been loaded
> Uptime: 1 day, 3:11:30
> ------------------------------------------------------------------------

Thanks for sharing.  It looks like we have something to investigate.
It's perhaps related to the way we build qtwebengine; do these sites
work in ungoogled-chromium for you?

Re-opening!

-- 
Thanks,
Maxim

bug#59546: qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

[bdju via Bug reports for GNU Guix]

On Fri Dec 16, 2022 at 7:31 AM CST, Maxim Cournoyer wrote:
> Thanks for sharing.  It looks like we have something to investigate.
> It's perhaps related to the way we build qtwebengine; do these sites
> work in ungoogled-chromium for you?

Just tested in ungoogled-chromium, and yes it seems both gitlab and
livechart worked fine, passing the brower checks quickly on my Guix
System machine. Very interesting.
I tried copying the user agent from ungoogled-chromium (similar but
slightly different from the qutebrowser default) but it didn't seem to
make a difference.