When electrons drift in a metal from lower to higher potential, it does not mean that all the free electrons are moving in the same direction. What happens is that only the drift velocity of free electrons is superimposed on the thermal velocity of electrons in the opposite direction of the electric field. The gain of velocity due to the applied electric field takes place only for a very short time.