At work we are like 5 people with Garmin watches, we did a check how precise the time is, there was a lot of differences!! Colleague of mine found solution for this sync time drift:
1. System->Time->Set Time switch to MANUAL
2. System->Time->Time Sync
That should sync time with your phone without excessive delay, it should also set AUTO in step 1.
After this procedure my colleague does not have drifted time anymore.