It happened to me now 2 times, so not always but very annoying.
I start a pool swimming activity and uses drill log for the complete workout. After a while the back button stops working, so I cannot start the next round of drill. So I stop the activity with start/stop button and continue again: that fixes the issue.
However after end of activity it says that data of activity cannot be displayed. And indeed I cannot display any details of the activity on the watch. Some data fortunately transferred to GC.
It only happened to me with pool swimming and this is the second time it happened.