Phone audio starts on watch even when I have bluetooth earbuds in, how to fix?

Using the watch to select answer on a call is useful when I'm biking my phone is packed away, or even if I'm walking around the house and the phone is in the other room.

But it answers to the watch which is undesirable and I have to switch it to the phone, highly undesireable touch-screen push extra step when biking. Didn't have to do that with 955.

How do I get it to default to the phone instead of the watch when answering?