Try adding your user to the 'input' group:

(user-account
 ...
 (supplementary-groups '("input" ...)))

...then reconfigure and reboot your system.

If that does not work, please mention which controller you are using and
paste the last lines from the `dmesg` command after plugging it in.