That doesn't match my experience. I have some unlisted videos that I or a small handful of friends might go back and watch once a year, and it takes several sec

That doesn't match my experience. I have some unlisted videos that I or a small handful of friends might go back and watch once a year, and it takes several seconds of loading before they start playing. It's very noticeably different from the near-instant loading of most videos I watch.