I just made this very mistake while adding a new regexp-error checking feature to xr. Needless to say I now am strongly in favour of turning it into a hard error. Patch attached. It was written for master, but I would suggest it go in emacs-27. The error message could be improved. For the benefit of isearch-forward-regexp, it's probably a good idea if it doesn't start or end in a square bracket.