Let $H$ denote a class of manifolds (such as $SO$ (oriented), $O$ (unoriented), $\operatorname{Spin}$, $\operatorname{Pin}+$, $\operatorname{Pin}-$, manifolds with spin defects, etc.). We define a $2+1$-dimensional $H$-modular TQFT to be one which lives on the boundary of a bordism-invariant $3+1$-dimensional $H$-TQFT. Correspondingly, we define a $H$-modular tensor category to be a $H$-premodular category which leads to a bordism-invariant $3+1$-dimensional TQFT. When $H = SO$, this reproduces the familiar Witten-Reshetikhin-Turaev TQFTs and corresponding modular tensor categories. For other examples of $H$, non-zero $H$-bordism groups in dimensions $4$ or lower lead to interesting complications (anomalies, mapping class group extensions, obstructions to defining the $H$-modular theory on all $H$-manifolds).