Some swing-district conservatives didn't use the 17-day recess to speak with constituents, limiting any exposure to the potential backlash from Trump's actions.