The default sorting for answers to questions is by votes.
If there's an accepted answer, it is at the top of the answers "pile" and the top voted answer is listed immediately below that one.
The votes are locked in place to prevent users from initially downvoting rival answers to push them down in the pile and then removing the votes later to recoup the 1 reputation point.
From this Meta Stack Exchange answer
Basically it's to stop people gaming the system.
One of the ways people did this was by down-voting rival answers and then removing that vote when their answer gains popularity. There are others.
So, the side effect of all this was that legitimate vote changes, like the one you describe, are not allowed either.
Since there are a number of people voting, there's room for a certain amount of "noise" in the system, such that if the answer is of low quality, over the long term downvotes will outnumber any upvotes that were given anyway.
At the risk of being long-winded, the bottom line is there's no need to sweat it.