Yeah, that's why you think that this is a "puzzle".

Though, it does invalidate every complaint you've made about IP source address priority. The only ways you'll get multiple addresses for a given host are to either hard-code them, or use DNS [0]. Given that you're not using DNS, then you either have automation to update all those hard-coded configs, or you do it all by hand. Either way, you're deliberately recording addresses that you don't control for your LAN hosts, and considering it a "puzzle" when you have trouble reaching those LAN hosts when your upstream provider changes their address.

[0] Given that every resolver I've ever used pulls data from '/etc/hosts', I'm considering that to be part of DNS for the purposes of this conversation. I'm also considering Avahi and friends to be DNS. [1]

[1] And if you've talked to any long-term Avahi user, you've certainly heard about its habit of -on daemon or host restart- erroneously considering host name of the machine it's running on to be "in use" and appending a "-N" to the end of it. The remarkable eagerness of Avahi to engage this collision-avoidance mechanism makes it entirely unsuitable for discovery of LAN hosts whose hostname is actually important. This fact breaks my heart, but Avahi has been doing this shit for a very, very long time, so it's probably never going to get fixed.