Democratic politicians are the ones that could remove the speaker, not voters.