Problems stopping the timer after a bike ride with Garmin Edge 530

After my bike ride I find I have difficulty turning off the timer at the end of the ride. Often I have to take off my gloves and press the button three or four times before I get the screen to save or resume or discard.

This button is the only weak feature I have found on my new Garmin Edge 530 and I am hoping it can be fixed or even better I am doing something wrong and someone can tell me how to correct my error?