You can select a single template to be assigned by default to all posts belonging to a term. If the posts belongs to more than a term with custom single templates assigned, the template chosen is that of the term with the highest post count.
Anyway, you can assign single templates on a per post basis.