Yeah, this can be done, but I think wearing wholesale sunglasses will also not give problem. If you are wearing wholesale reading glasses, find them heavier than you can also go with plastic sunglasses, as plastic is lighter in weight, also you can attach the sunglasses strap, while cycling.
Having had an accident where I landed on my face, I am not willing to wear "wholesale sunglasses". Very grateful my Rudy Project glasses did not shatter or the frames break. Ended up in hospital for two weeks, but still have my eyes. But I also agree that a different mount mechanism would be nice.