When you are in the follower selection window to pick one for an expedition, tapping the follower's name will lead you the follower details window, from where you should be able to remove the follower from current workplace.
But maybe this is not practical? It is hard to allow for a convenient experience while keeping the interface not too cluttered.
