Tracks drift off of mapped roads - causing problems in later conversion of Track to Route

I was converting a simple Track to a Routes, and got the wonky straight line route connecting the start end and points.

My understanding is that this typically happens when the Track goes "off-road".   

The Track was through well mapped city streets, yet close examination showed the Track deviating slightly from the street, an error in the precision of the gps data. 

I've seen this in a couple of Tracks.  For example my ride will show segments parallel to the road I know I was on. 

When such minor deviations happen, how come the Track to Route algorithm can't deal with such minor minor offsets?  Instead it gives the wonky straight line between the Track start and end points.