If an opponent gains control of the creature that Pact Weapon is attached to but doesn't also gain control of Pact Weapon, you still control Pact Weapon and still don't lose the game for having 0 or less life as long as it is attached.