I tested on bare metal and virtualized x86-64 with NTP.

The bug reporter [0] tested that the fix worked for Chrony.

Pushed as 898fbb60b2354e82e5b7f259b44dbfed112a83aa

[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43321#5