If you care about every last bit of performance, CPUs are difficult to program for as well, possibly more difficult than a GPU.

Generally GPUs punish you more by bad performance if you don't optimise your code.