Wither applies to any damage dealt to creatures by Village Pillagers. This includes combat damage as well as anything that causes it to deal noncombat damage, such as the effect of its second ability.