Revealing the Drivers of CO-to-H2 Conversion Factor Variation and its Impact on Star Formation Efficiency