You can achieve it by creating a closed Group with just those connections that you want to do the challenges, and then launching a group challenge. A non-member then cannot be invited without joining the group first (which you can decline). You can create several groups, if needed.
