Speaker Johnson says he's given up on Schumer and Jeffries