If you attack the same player with two Terisian Mindbreakers, that player will mill half their library as the first Mindbreaker's ability resolves. Then they will mill half of what remains in their library as the second Mindbreaker's ability resolves. They do not mill their entire library unless their library only had three or fewer cards to begin with.