HR sensor does not always work well while swimming

you are not unlucky - Happens because water gets between the sensor and the skin. If you want accurate readings, buy a chest strap and connect it to your watch

I know that with a chest strap the reading is accurate, but what I don't understand is why it doesn't read well only the first 15/20 minutes, then it starts reading well and continues well until the end.

My bold guess is since you worn the watch correctly and tight water might not get between the first couple minutes but eventually will. 

It's a possibility I hadn't thought of. Next time I'll do some tests.

I did some tests, making sure that the water got between my wrist and watch before starting training. I also tried moving the watch during workouts, but the problem remains. Any other suggestions?

Apart from wearing a chest strap, there is not a whole lot you can do. Maybe it works to wear the watch a bit higher on the arm, but not so much it interferes with stroke detection.

I swam for years with a Scosche Rhythm optical HR sensor worn just above a Fenix (models without OHR), and the Scosche also was often problematic during the first part of the session. It is a tough job getting optical HR while in the water. What worked best for the Scosche was toggling it off and on, to get it out of the stubborn erroneous pattern it had gotten into. Which is less easy to do on the Fenix 7, setting wrist HR off and on.

It is just super hard to get right.. before the Garmin Swim was around, I tried Suunto Memory Belt, Wahoo Tickr with memory, a Polar with analog chest strap (5KHz goes through water fine), a Garmin FR60 taped right on top of an ANT+ HR strap, electrode stickers with lengthened leads going to a HR strap module and FR60 in an Otterbox. I just really, really wanted it to work. But spent a lot of time diving to retrieve dislogded gadgets from the bottom of the pool. Fun times!

Had same problems with my old Fenix 6 Pro and for first months with my new Forerunner 955. Then I solved wearing it as Garmin says: find your bone on the upper side of the wrist and wear the watch tight about 1 finger behind it.

Never had isues anymore

I'm wearing it like this, maybe I should wear it even higher?

I had the same issue, same as everyone as it’s well known that HR and pools aren’t great.. anyway, I did 2 things and got rock solid results 1. Wear the watch higher in the wrist where there is a better chance the OHR can work (I think Desfit has a YouTube video on best placement for OHRs) and 2, get a Velcro adjustable wrist band so you can get it reasonably tight. The silicone band that came with the Fenix just kept allowing the watch to slip and move. 

Personally I never had problems like that. I do happen to have my HR rise after some time (10-15 min) during a swim and this is normal body reaction. Maybe this is what you observe as well?