It appears that you have the opposite problem of not working when not in activity I think.. I had the same issue and thought it was a bug until I read this answer here (https://forums.garmin.com/sports-fitness/healthandwellness/f/vivoactive-3-3-music/137127/va3-gesture-not-working-well?ReplyFilter=Answers&ReplySortBy=Answers&ReplySortOrder=Descending)
Basically check if "Do Not Disturb" is on and turn it off so as to make the gesture active and turn on the backlight