For me it takes quite a long time, up to several minutes.
I found a potential source of the problem. A quick GPS fix requires a precise time setting, previously I had time set a few minutes ahead. Today will test.
Why don't you just set your time to "Auto"? Indeed, incorrect time of day might affect GPS fix time.
In "Auto" mode the time is taken from the phone instead of GPS. And I had just set the wrong time on my phone.