​Don't worry about it.  Everyone knows you are doing a great job for Emacs as are the other major contributors.

As Eli said, there will always be noise and questions.  Your legacy will be in the releases made.

Bob