"Christopher M. Miles" <numbchild@gmail.com> writes:

>          668  28%                                    - org-agenda-prepare-buffers
> ...
>          523  22%                                              - org--tag-add-to-alist

I have a suspicion where the bottleneck is.
Can you try the attached patch?