If you choose at least one target for Serene Remembrance, and all of its targets are illegal when Serene Remembrance tries to resolve, it won’t resolve and none of its effects will happen. You won’t shuffle any library and Serene Remembrance will be put into its owner’s graveyard.