Altimeter/Barometer is extremely inaccurate

This issue exists since they introduced the barometer as measurement of elevation, with the adverse impact of the temperature on the sensor.

Initial calibration doesn't help since the temperature will change. If you stay on the flat area you elevation will go down when your temperature rises...

But the watch has a temperature sensor as well, which should be used for compensation. Besides, the barometric altitude measurement is continually calibrated by GPS altitude. I usually see no altitude drift in my exercise logs. I do however have an offset error which I suspect comes from wrong sea level calculation in my area.

Yeah, the temperature sensor in the watch is only there for altimeter temperature compensation. 

My watch calibrates to different altitudes depending on which satellite settings I use. Also, I don't think Garmin actually pulls barometric data from weather stations, so sea level pressure is calculated from absolute pressure and current altitude. For my watch, my barometric pressure is about 6-10 mb higher than that of my iPhone, EDGE 530, and surrounding weather stations. But I haven't yet seen a consumer device that has accurate pressure, as it's such a dynamic thing. I just wonder how much tolerance there is at the factory when calibrating the barometer. For the total range, the % tolerance for a 8mb error is very small and probably well within specification. 

Chris, my understanding is that a temperature increase will lower pressure, therefore would increase elevation value.

I still think the OP should consider a factory reset. I had an issue after the update that if I didn't wait 30sec or more after GPS acquisition for an activity, the altimeter would always be set to something like -490meters. Everything else worked. Maybe there is just a bug and the only way to kick it out is to do a full factory reset. User settings, activity settings, etc, can be backed up and restored. Garmin has instructions somewhere to outline how to do this.

Did a cycle with friends doing the same route at the same time and their elevation readings were 1970m and mine 880m with a Vivoactive 4s. Very frustrating 

Yes. I haven't "calibrated) mine in a while and my pressure reading is now 1273 and elevation is 1989 (real elevation is something like 220m) so it is completely arbitrary...