The player who controlled Ivory Gargoyle when it was put into a graveyard will skip their next draw step even if Ivory Gargoyle is returned to the battlefield under a different player’s control (because it’s owned by someone else) or isn’t returned to the battlefield at all (because it left the graveyard before the end of the turn, for example).