It happens only when sorting by recency. Does ibuffer re-visit the buffers when re-displaying them?