This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.