From mboxrd@z Thu Jan  1 00:00:00 1970
Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail
From: Thanos Apollo <public@thanosapollo.org>
Newsgroups: gmane.emacs.devel
Subject: Re: Using tor & url
Date: Sun, 11 Feb 2024 09:43:50 +0200
Message-ID: <87frxzbdbd.fsf@thanosapollo.org>
References: <87ttr67rkt.fsf@thanosapollo.com>
 <CADwFkm=OV5rmDvVuyhrn8HusxsbJRe_THPNUrMNPjLzMssPjMg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214";
	logging-data="28139"; mail-complaints-to="usenet@ciao.gmane.io"
User-Agent: mu4e 1.10.8; emacs 29.2
Cc: Thanos Apollo <public@thanosapollo.com>, emacs-devel@gnu.org, rms@gnu.org
To: Stefan Kangas <stefankangas@gmail.com>
Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Feb 11 10:01:30 2024
Return-path: <emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org>
Envelope-to: ged-emacs-devel@m.gmane-mx.org
Original-Received: from lists.gnu.org ([209.51.188.17])
	by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org>)
	id 1rZ5iJ-00077z-9W
	for ged-emacs-devel@m.gmane-mx.org; Sun, 11 Feb 2024 10:01:29 +0100
Original-Received: from localhost ([::1] helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <emacs-devel-bounces@gnu.org>)
	id 1rZ5hZ-0002tq-EN; Sun, 11 Feb 2024 04:00:41 -0500
Original-Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <public@thanosapollo.org>)
 id 1rZ5RY-0000xi-V8
 for emacs-devel@gnu.org; Sun, 11 Feb 2024 03:44:08 -0500
Original-Received: from out3-smtp.messagingengine.com ([66.111.4.27])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <public@thanosapollo.org>)
 id 1rZ5RW-0007bz-CZ; Sun, 11 Feb 2024 03:44:08 -0500
Original-Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 0ADCC5C0093;
 Sun, 11 Feb 2024 03:44:04 -0500 (EST)
Original-Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Sun, 11 Feb 2024 03:44:04 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 thanosapollo.org; h=cc:cc:content-type:content-type:date:date
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:subject:subject:to:to; s=fm1; t=1707641044;
 x=1707727444; bh=gowleOfiPT8M3hrIrTVKFyS0zaNDZmocMF2UF9c2txk=; b=
 SDlBCd7qg9CyvZN51ulna68TFyrmbezTDfi0o6+FvKfAzTbNm8yVYWR4NQUpmxbP
 UDwhFJqT5YluwMA3YIIHd93dGwejB06f9XEpx3d0Y4/nMYadTUYeOVUUBmn8sKff
 uNVPspY4nEJphG4kqU46JX8jVulW509LpC3FIGqgpDyUcmqgjIu6OYaCUOkq+h+n
 zfO4vK3R81spQNEiAO5YN19NlZQfMXCKjtwTpKvfgHPHVRljo0WKbZzw95weaYH7
 xA4GMffxYsbiYBfaapFwB+Wj3CPrwfZ3ZlZ4MDeHDhFoyunN6AU7p3xuQKNUR+FW
 KnJ7OnyiiedmRkb5QbomuQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:content-type:date:date
 :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:subject:subject:to
 :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm3; t=1707641044; x=1707727444; bh=gowleOfiPT8M3hrIrTVKFyS0zaND
 ZmocMF2UF9c2txk=; b=CqfMMCn9Cg21nJ8Qa9IYqPRxEh4qlR7uQ98tPFap5MLZ
 Yui2kZMgnbKZOiFKUSwGfAswodVaAqPb0tVVG9VRBs1BBtbbIltnATsKXG1noR9b
 9URLOaUICQIJkbKza8kLaZy1FkYCiD6PQy9+WUgq4Fajkj9zktJOFH34z3tTbdQ3
 nWUAMUkFoRfnamH1LFU11VT8gB7Jh6PskQKcs5N5nIk+qMdfmkZEtRmztrXUPQ88
 AZ/7/lsD7XSrriupckTI6PItdCI01KPH9o7qUZ2Nek0JfNsvRU71Dsf/yAlcu3Im
 Ywvz3ONPZbHsnxMPu/885MaLr9SPzmt+sIBhmWVqDA==
X-ME-Sender: <xms:04jIZSFE7CArI2AlLAaMHjDnD8us7NrbEWpncQs3AI_vc5mQWra3TQ>
 <xme:04jIZTXEMP7jPk_gDlWjKeotgSegCSmuUv_PNx5hstSgTLGGdCovpZmjueK-zXdhP
 5AGqxYpjmO24Byw4dc>
X-ME-Received: <xmr:04jIZcKAR9n31fzk2NYwRW_1wdWOzTovIILn1LfdkGcp8T4ZdpzVtw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtgdduvdefucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepfhgfhffvvefuffgjkfggtgesghdtreertdertdenucfhrhhomhepvfhhrghn
 ohhsucetphholhhlohcuoehpuhgslhhitgesthhhrghnohhsrghpohhllhhordhorhhgqe
 enucggtffrrghtthgvrhhnpeelueefgfeuheeghfduveekgedukeegudekveeileefteff
 ieejlefgleejhfdvheenucffohhmrghinhepihhnvhhiughiohhushdrihhopdhgihhthh
 husgdrtghomhdpthhhrghnohhsrghpohhllhhordhorhhgnecuvehluhhsthgvrhfuihii
 vgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhusghlihgtsehthhgrnhhoshgrph
 holhhlohdrohhrgh
X-ME-Proxy: <xmx:04jIZcGK2_SyA8SeXiryvgm96KUvFOdctFUDwG9ENkPIu2GvsRn5SQ>
 <xmx:04jIZYU-UwSRzVLMBKXKjOhfl_-xT838MH9KMXKw8dfmBKpVecfSRQ>
 <xmx:04jIZfO4HxuySHwCKHtuES52aq201HQzK1mAckgTPR7GfgU8Y1vQ1A>
 <xmx:1IjIZfQpvcDYi2OcRPUJIj2D0Z8M0pq9yaAB4heCfppLXed6u1syDA>
Feedback-ID: if38949d4:Fastmail
Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 11 Feb 2024 03:44:02 -0500 (EST)
In-reply-to: <CADwFkm=OV5rmDvVuyhrn8HusxsbJRe_THPNUrMNPjLzMssPjMg@mail.gmail.com>
Received-SPF: pass client-ip=66.111.4.27; envelope-from=public@thanosapollo.org;
 helo=out3-smtp.messagingengine.com
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001,
 T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Mailman-Approved-At: Sun, 11 Feb 2024 04:00:39 -0500
X-BeenThere: emacs-devel@gnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Emacs development discussions." <emacs-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/emacs-devel>,
 <mailto:emacs-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/emacs-devel>
List-Post: <mailto:emacs-devel@gnu.org>
List-Help: <mailto:emacs-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/emacs-devel>,
 <mailto:emacs-devel-request@gnu.org?subject=subscribe>
Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org
Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org
Xref: news.gmane.io gmane.emacs.devel:316109
Archived-At: <http://permalink.gmane.org/gmane.emacs.devel/316109>

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable


Quite a late reply to a dead thread, but for anyone reading this in the
future and is interested in implementing tor support, it might be
useful.

Stefan Kangas <stefankangas@gmail.com> writes:

[...]
>
> From a very quick glance at the code:
>
> - Is there anything more needed to add support for Individuous
>   instances?  See: https://invidious.io/
>

(FYI this has been added)

> - Any chance you could add an option like `vc-tor'?

Is it recommended to use 'socks for this?

e.g using a macro like this one:

=3D=3D=3D=3D
(defmacro yeetube-with-tor-socks (&rest body)
  `(let ((url-gateway-method 'socks)
         (socks-noproxy '("localhost"))
         (socks-server '("Default server" "127.0.0.1" 9050 5)))
     ,@body))
=3D=3D=3D=3D

This was the easiest way I could find to route traffic using
#'url-retrieve-synchronously to tor.

A built in option like this for url to route all traffic through tor
might be possible?

Additionally, I found the performance of url-retrieve-synchronously to
improve when I set 'url-request-extra-headers' to something like this:

=3D=3D=3D=3D

'(("Accept-Language" . "Accept-Language: en-US,en;q=3D0.9")
    ("Accept" . "text/html,application/xhtml+xml,application/xml;q=3D0.9,im=
age/webp,image/apng,*/*;q=3D0.8")
    ("User-Agent" . "Mozilla/5.0 (Windows NT 10.0; rv:122.0) Gecko/20100101=
 Firefox/122.0"))

=3D=3D=3D=3D

I'm using the user-agent from arkenfox user.js
  <https://github.com/arkenfox/user.js/>

I had url-privacy-level set to 'high, but despite that my user-agent was
"URL/Emacs Emacs/29.2 (X11; x86_64-pc-linux-gnu)". Changing this
improved the performance of #'url-retrieve-synchronously (maybe because
it's not a common user-agent and it required further verification?) &
makes websites that utilize user-agent blocking accessible.

Also, when it comes to 'url-mime-language-string', shouldn't that either
be set to "en" by default (instead of nil) or get a locale value using
something like getenv? I was located in Greece with locale set to en-US
but I was getting Spanish by default on most websites (still trying to
figure out why).

=2D-=20
Information wants to be free.

Thanos Apollo
https://thanosapollo.org

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQJMBAEBCAA2FiEEYrdY0PZxmTi8Cc7KM59zbDpyCSgFAmXIiNYYHHB1YmxpY0B0
aGFub3NhcG9sbG8ub3JnAAoJEDOfc2w6cgkoz5YP/A77DrVlbz8YmrFg/3wIIQcn
bvz3aPZuFnjeGq2INY7Fhymo9bQDtrUgms3IweCZWyozfjoBuoxocA/n3iY25JD2
3+XFj4P3l3/NC0IiWwG74RfjpVfq16zSHJdIrbALMWoQ70pHWWLf7nGB3ocM/TLz
MdnD4a2hJbsNDYGqBgpOoBA+pOVQdZ3ot+ZZd+LWxQAiIYeXP6kqanD+KvQ/BpJB
usXdqaGuoWVnGJ2N8kl99TmHpAilS9QuRqmgijboPIuge4FTVgF/PPlSOMy+0orb
zB4wtADbl/NF/AZ8TSbIPkzK4v2M6/+6gkxHZ2LNGz6Afs0qVt4YjbokQVwh3pUX
y/p1v0Ey5JAlCCvx1zByV29g6u2oY/1Bne4jkHmZMkd0YVznpflpiKV32k/NCyB+
+99IRiqhvutGL3Ml33yRSNPvkm/hscI5nHEasuF/Jey5U5IGf0tWS5gnGH1b8XSx
HMns/fdoGJUapLa+1b/y56tGDE+63bVTLe8Znd2rFy93r9HW3ID2PAd5xaTkDxd2
oZ3EhB42TSaFwYMl+AsR0PZaBKdd8xAmuuAaRqTDor38ENthL4gk0awq0fuwCG3G
/fnHKpuy7DnQL5Bm4aQXfDQHar1WqLQYGZXKKp/AEOvQ4lDc6fDoyD1D3b3xxVO5
f/5Qc9GRs7ATrYHrlhbk
=6H1p
-----END PGP SIGNATURE-----
--=-=-=--