URL currently doesn't support custom user agents. This could be
implemented by adding a defcustom string url-user-agent which could be
changed by the
user.
The default value could be:
(format "User-Agent: %sURL/%s\r\n"
         (if url-package-name
         (concat url-package-name "/" url-package-version " ")
         "") url-version)

I have attached a patch to add support for custom user agents in URL.

-- 
Vibhav Pant
vibhavp@gmail.com