Involutive category theory provides a flexible framework to describe involutive structures on algebraic objects, such as anti-linear involutions on complex vector spaces. Motivated by the prominent role of involutions in quantum (field) theory, we develop the involutive analogs of colored operads and their algebras, named colored $\ast$-operads and $\ast$-algebras. Central to the definition of colored $\ast$-operads is the involutive monoidal category of symmetric sequences, which we obtain from a general product-exponential $2$-adjunction whose right adjoint forms involutive functor categories. Using a novel criterion for trivializability of involutive structures, we show that the involutive monoidal category of symmetric sequences admits a very simple description. For $\ast$-algebras over $\ast$-operads we obtain involutive analogs of the usual change of color and operad adjunctions. As an application, we turn the colored operads for algebraic quantum field theory into colored $\ast$-operads. The simplest instance is the associative $\ast$-operad, whose $\ast$-algebras are unital and associative $\ast$-algebras.