Ticket Created
over 4 years ago

WERETECH-9896

Memory fragmentation in FW?

My long running app crashes and I suspect memory fragmentation in the FW.

It ran for 1:45 on my F5 before crashing out of memory: 

Error: Out Of Memory Error
Details: 'Failed invoking <symbol>'
Time: 2020-10-17T02:18:02Z
Part-Number: 006-B2697-00
Firmware-Version: '20.00'
Language-Code: eng
ConnectIQ-Version: 3.1.9
Filename: RACEQS
Appname: raceQs
Stack: 
  - pc: 0x30000e9b


Note that the location 0x30000e9 is well outside the memory addresses of my code.

I'm monitoring the memory usage at the start of each GPS location cycle and it is steady at around 102 kB.

The crash occurred independently of any user input: it was running fine and then, a few minutes later when I next looked at the watch, it had crashed.

I have run the app twice through the simulator using the FIT file generated in the original session, as far as possible performing the same user inputs as on the original session. The memory usage remains around 105kB. The peak memory remains firmly at 117.5kB.

It peaks at the start of the app when it downloads a large data file containing the race courses and rounding marks.

I have had a  small number of similar reports of out of memory errors via ERA on a range of devices:  Forerunner® 245: 5.00, vívoactive® 3: 7.40, all reporting memory crashes, but all at different code locations.

This behaviour indicates, and I say this with much trepidation after many decades of software development, that the issue is not with my code, but with the underlying firmware, operating system or hardware.

Given that it only occurs sporadically and in my test, only after a long period of running (1h 50 mins), I wonder if it is being caused by memory fragmentation in the FW?

And if so, what can I do about it?


P.S.I had a similar problem on this platform a couple of years ago and discovered the fault was with my code: I had a hierarchy of menus and was not Popping them correctly. At the time, the simulator behaved differently and did not show my memory leak. Since then, I am logging memory usage at the start of each GPS callback to capture any such oversights. 

Alan.raceQs
  • over 3 years ago

    There has been quite a long discussion on this problem in this Discussion Forum thread, which didn't yield any conclusive results, so I once again refresh this bug report.

    Here is a summary of the characteristics:

    1. The code is working with limited available space: It's consistently reporting used memory of around 100kb in a 124kb space.

    2. There is no memory creep or crash when running the same data through the simulator by re-running the FIT file (which continued to record after the crash for several minutes until the IQ! logo was cleared). It runs for hours in the sim with a fixed Peak Memory of 110 kb.

    3. Only occurs after running a long time, in excess of one hour.

    4.The crashes are very infrequent: We have hundreds of users, and only a handful of ERA reports.

    5. The crash locations appear random: they  occur at many places in the code. 

    6. ERA reports crashes across a number of devices. Unfortunately the broken ERA server prevents me from reviewing the details.

    I have no idea about how the VM handles memory: all I understand is it manages memory by reference counting.

  • over 4 years ago in reply to Alan.raceQs

    This may not be significant, but often, when starting my app in the sim, I get :

    Error: Out Of Memory Error

    Details: Failed invoking <symbol>

    Stack: 

      - toString() at /private/var/jenkins/workspace/Tech-Rel-CIQ-Mac/mbsimulator/submodules/technology/monkeybrains/virtual-machine/api/Lang.mb:1388 0x30001008 

    Which is immediately and always resolved by re-starting the app, without having to restart the sim .

  • over 4 years ago in reply to jim_m_58

    Yes, I know. But if the sim shows ZERO growth at in memory usage over 90 minutes, I think it's pretty clear nothing is going to happen in 8 hours.

  • over 4 years ago in reply to Alan.raceQs

    If you can break it in the sim, it might be easier to track down.

  • over 4 years ago in reply to Alan.raceQs

    And I'm not trying to break the sim. I'm using it to monitor the memory usage. If it crashes the watch after 90 minutes, I would expect to see a memory usage increase in the sim over that period. I don't.