Investigating Workouts not appearing after successful upload

Since this is happening to me from time to time and again yesterday evening, I decided to post it here as it is quite annoying, even frustrating when this happens during some harder challenges (the old ones, the real challenges)
I know there are topics about activities failing to upload already, but I think the root-cause of the problem "missing activities" lie elsewhere which I'll try to explain.

Behavior:
Start workout > Complete workout > Click Save > Observe (grey?) balloon about successful upload or something similar > SILENCE (No green balloon confirming "Successfully Processed")
The file in the AppData\[..]\LocalCache\activities\ folder is gone because it has been uploaded.
Result: Workout not showing in TDA or Connect

Environments:
This happens in both production and beta environment. Last evening I was using the Beta 4.42.2.25140

Investigation
I think the process is failing somewhere in "the tacx cloud" based on my analysis of the logs. One example from a workout on Tacx production:

RaceService|Activity upload started!!
AnalysisLogService|**ACTIVITY:{"loggingId":"c2d8db5e-2eab-4375-8366-f081130fba78","createTime":"2023-01-18T15:46:27.0167554Z","parentCourseCloudId":"70a13470-50a6-4250-96a5-d0663f62a5f3","parentCourseName":"Etape du Tour 2019 - Val Thorens"}
AnalysisLogService|**ACTIVITY_UPLOADS_CREATE:{"uploadsId":5,"activityLoggingId":"c2d8db5e-2eab-4375-8366-f081130fba78","linkedCourseLoggingId":6}
AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ATTEMPT:{"uploadsId":5,"uploadsAttemptId":7,"activityUploadId":9,"courseUploadId":8}
AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START:{"uploadId":9}
AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_FILE:{"uploadId":9}
RaceResultUpload|Upload activity bcb49773-fd97-4032-8d99-17fca15d93b8 succeeded
AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_POLLING:{"uploadId":9}
NotificationService|Toast: Process: This may take a few minutes. You can safely close the application now. Your activity will appear once processing is complete.
ICloudServiceManager|HandleGetUserProfileExceptions --> Tacx.Cloud.Apis.Common.CloudApiException: Error calling UserGetProfile: <!DOCTYPE html>
[...]
<title>api.tacxtraining.com | 524: A timeout occurred</title>
<p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a rel="noopener noreferrer" href="">support.cloudflare.com/.../200171926-Error-524">Additional troubleshooting information here.</a></p>
INFO|39|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_ERROR:{"uploadId":9,"errorMessage":"Error calling ActivityGetByUUID: {\"type\":\""title\":\"Forbidden\",\"status\":403,\"traceId\":\"00-4d3fe6378f3f7325ebc848f3e8699e6e-3c05e0d36c4fc243-01\"}"}">tools.ietf.org/.../rfc7231


These logs confirm that the upload of activity was successful. It seems to fail somewhere during the processing phase. The http-error codes 524 and 403 are evidence for something failing badly.
I did contact regular support in this case. Based on their reaction it seemed to me they receive this question regularly but don't really investigate on a 1-by-1 case. They advised me to wait, maybe the ride was stuck and it would show up later. It never did of course because cloud went boohhhhh.

Back to my activity from last night....
Because I experienced the issue again and was annoyed by it, I decided to dive in the logs once again.
In TDA I saw the grey "uploading" balloon and the activity file got deleted from the activities folder. The workout never showed up.
I saved the logs, analysed them trying to find something common. Here are the relevant bits:

152181|2023-05-11T15:52:11.9607525+00:00|INFO|44|RaceService|Activity upload started!!
152182|2023-05-11T15:52:11.9607525+00:00|INFO|4|AnalysisLogService|**ACTIVITY_FINISH:{"loggingId":"e6ad5481-ed02-49a6-b5dd-01837c621da8"}
152183|2023-05-11T15:52:11.9805666+00:00|INFO|4|RaceService|Uploading e6ad5481-ed02-49a6-b5dd-01837c621da8_20230511025235
152190|2023-05-11T15:52:12.1058586+00:00|INFO|4|AnalysisLogService|**ACTIVITY:{"loggingId":"e6ad5481-ed02-49a6-b5dd-01837c621da8","createTime":"2023-05-11T14:52:35.9062230Z","parentCourseCloudId":"bbb7d3ad-250d-445d-8ef2-992f95e71a2a","parentCourseName":"v5 NorseTour 2 - Geiranger Fjord (Norway)"}
152191|2023-05-11T15:52:12.1058719+00:00|INFO|4|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE:{"uploadsId":5,"activityLoggingId":"e6ad5481-ed02-49a6-b5dd-01837c621da8","linkedCourseLoggingId":6}
152315|2023-05-11T15:52:15.0407547+00:00|INFO|44|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ATTEMPT:{"uploadsId":5,"uploadsAttemptId":7,"activityUploadId":9,"courseUploadId":8}
152316|2023-05-11T15:52:15.0407547+00:00|INFO|44|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START:{"uploadId":9}
152393|2023-05-11T15:52:16.8757217+00:00|INFO|44|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_FILE:{"uploadId":9}
152469|2023-05-11T15:52:18.6309853+00:00|INFO|45|RaceResultUpload|Upload activity 88907cb1-f891-4901-9481-bee31d10c98f succeeded
152471|2023-05-11T15:52:18.6309853+00:00|INFO|45|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_POLLING:{"uploadId":9}
152473|2023-05-11T15:52:18.6309853+00:00|INFO|45|NotificationService|Toast: Process: This may take a few minutes. You can safely close the application now. Your activity will appear once processing is complete.
159207|2023-05-11T15:54:57.2706005+00:00|INFO|33|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_ERROR:{"uploadId":9,"errorMessage":"Error calling ActivityGetByUUID: {\"status\":412,\"traceId\":\"00-397a9d43fbe84143ae49e69b5312364a-55b8f250b3002b2e-01\"}"

The last entry shows a status=412 which is a http code indicating some kind of access issue.
This morning I tested the same course on Beta, saved and it processed successfully. I tried grabbing the logs and comparing them. This time the activity showed up. The logs from this session:

43509|2023-05-12T06:06:22.3690745+00:00|INFO|8|RaceService|Activity upload started!!
43510|2023-05-12T06:06:22.3690745+00:00|INFO|3|AnalysisLogService|**ACTIVITY_FINISH:{"loggingId":"e22f997a-769d-4e75-8bbb-ba5c5e23e2ae"}
43511|2023-05-12T06:06:22.3789222+00:00|INFO|3|RaceService|Uploading e22f997a-769d-4e75-8bbb-ba5c5e23e2ae_20230512060457
43513|2023-05-12T06:06:22.3894377+00:00|INFO|3|AnalysisLogService|**ACTIVITY:{"loggingId":"e22f997a-769d-4e75-8bbb-ba5c5e23e2ae","createTime":"2023-05-12T06:04:57.4441623Z","parentCourseCloudId":"bbb7d3ad-250d-445d-8ef2-992f95e71a2a","parentCourseName":"v5 NorseTour 2 - Geiranger Fjord (Norway)"}
43514|2023-05-12T06:06:22.3894377+00:00|INFO|3|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE:{"uploadsId":5,"activityLoggingId":"e22f997a-769d-4e75-8bbb-ba5c5e23e2ae","linkedCourseLoggingId":6}
43718|2023-05-12T06:06:27.0788943+00:00|INFO|75|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ATTEMPT:{"uploadsId":5,"uploadsAttemptId":7,"activityUploadId":9,"courseUploadId":8}
43719|2023-05-12T06:06:27.0788943+00:00|INFO|75|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START:{"uploadId":9}
43790|2023-05-12T06:06:28.7991833+00:00|INFO|39|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_FILE:{"uploadId":9}
43806|2023-05-12T06:06:29.0791513+00:00|INFO|34|RaceResultUpload|Upload activity 34e4ca49-7e27-4bfa-89df-46689997dd64 succeeded
43808|2023-05-12T06:06:29.0791513+00:00|INFO|34|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_START_POLLING:{"uploadId":9}
43810|2023-05-12T06:06:29.0791513+00:00|INFO|34|NotificationService|Toast: Process: This may take a few minutes. You can safely close the application now. Your activity will appear once processing is complete.
44518|2023-05-12T06:06:52.4489294+00:00|INFO|38|AnalysisLogService|**ACTIVITY_UPLOADS_CREATE_ACTIVITY_FINISH:{"uploadId":9,"cloudId":"34e4ca49-7e27-4bfa-89df-46689997dd64"}
44519|2023-05-12T06:06:52.4489294+00:00|INFO|38|RaceResultUpload|Activity 34e4ca49-7e27-4bfa-89df-46689997dd64 processed successfully by the cloud.
44520|2023-05-12T06:06:52.4489294+00:00|INFO|49|NotificationService|Toast: Succeed: Your activity has been uploaded.

This time ACTIVITY_UPLOADS_CREATE_ACTIVITY_FINISH is logged instead of ACTIVITY_UPLOADS_CREATE_ACTIVITY_ERROR
The activity was processed successfully and showed up @ TDA.

Conclusion
To me this looks like access issues in the "Tacx cloud" or between Cloudflare>Tacx cloud. Once this occurs it will not (or cannot) retry and the workout is gone.

Please please please Tacx! make this process more robust and implement some (more) fault tolerance.
Why is it not retrying? Or maybe don't delete the activity file from disk of the client so quickly. Wait with deleting until the process confirmed that processing was successfully.