The exiled cards return to your graveyard immediately after Idol of Endurance leaves the battlefield. Any cards that had left exile before then (most likely because they were cast with Idol of Endurance's second ability) remain in their current zone.