I think the number one thing that bites a lot of container workflows is when you have a separate "base image" build that depends on external, mutable package sources using tools that don't support locking, such as apt.

This gets worse for each additional such layer, for example if you start from ubuntu:whatever and have a system deps layer, then an application deps layer, then static assets, then your app's code. Unless you think about it upfront when designing the pipeline, it can be challenging to trace all that state backwards and figure out how to safely re-build thing-from-x-weeks ago but with a single unsafe package bumped forward.

Every shop with competent devops people present should be set up for this, but it usually doesn't come for free.