For me, this is done in the text-mode hook.  But then I remove
auto-fill-mode in several modes like this: