I have a similar issue, but believe its my scope.
Problems like this should only a couple things. The action may be moving around and may require bedding. The barrel may walk when warm, but what happens when you go beyond 5 shots? Do shots 6 through 10 go back and do the exact same pattern?
The other likely suspect often is a scope. I've been down this road recently with the gun I mentioned. While I haven't solved the problem yet, I do know the scope is a problem. My scope is made by a very well known American company.........as I have come to find out, these scopes are getting a reputation for getting damaged by recoil from much more than a .30-06......in addition the adjustments on these scopes are worthless because on many they are not repeatable; say one click left, shoot, no movement, next shot, it moves because the recoil jarred the scope's internals into position. A person I would easily qualify as an "expert" in the field (no pun intended) verified what I suspected about this. So, sorry for ranting, but if you rule everything else out, switch scopes/guns and don't be afraid to return scope to the maker or get a quality one if it proves less than desireable.