Title:Nested hyperedges promote the onset of collective transitions but suppress explosive behavior

Abstract:Higher-order interactions can dramatically reshape collective dynamics, yet how their microscopic organization controls macroscopic critical behavior remains unclear. Here we develop a new theory to study contagion dynamics on hypergraphs and show that nested hyperedges not only facilitate the onset of spreading, but also suppress backward bifurcations, thereby inhibiting explosive behavior. By disentangling contagion pathways, we find that overlap redirects transmission from external links to internal, group-embedded routes -- boosting early activation but making dyadic and triadic channels increasingly redundant. This loss of structural independence quenches the nonlinear amplification required for bistability, progressively smoothing the transition as hyperedges become nested. We observe the same phenomenology in Kuramoto dynamics, pointing to a broadly universal mechanism by which nested higher-order structure governs critical transitions in complex systems.