Stack trace addresses in FW.

I have been down this path before, but I don't think I ever made sense of the response.

Here's an error log report from my watch yesterday. (Nothing in ERA - it's still reporting "A server error has occurred...")

Error: Out Of Memory Error
Details: 'Failed invoking <symbol>'
Time: 2021-11-18T07:58:17Z
Part-Number: 006-B2697-00
Firmware-Version: '24.10'
Language-Code: eng
ConnectIQ-Version: 3.2.5
Store-Id: 74c474f8-4140-4794-afb7-f56817083bd5
Store-Version: 43
Filename: 4C065B22
Appname: raceQs
Stack: 
  - pc: 0x300035c0
  - pc: 0x300033d0
  - pc: 0x10002946
  - pc: 0x10001256

The first two addresses of the stack trace  (- pc: 0x300035c0 - pc: 0x300033d0) do not lie in my code, so why are they being provided to me as the developer?

I suspect a memory leak in the FW VM as I am, (and have been for the past four years!)  meticulously managing the memory in the app. The crash occurred after 70 minutes of faultless running.

Can anything be deduced from the fact that the first two addresses of the stack trace are in the FW VM?

  • That doesn't actually surprise me as the recording is happening in the FW,  Does the time line up with when you backed out of the IQ! ?  What device?

  • In this case I'd probably monitor available memory and the start to log things when memory gets tight..

    And, BTW, that's not as easy as it sounds. Whilst the simulator has a nice feature to monitor "Peak memory usage",  that's not available in CIQ. The best I can find is to report current memory usage with System.getSystemStats().usedMemory.

    I'm logging usedMemory and know that my app runs at around 110 Kb, in normal state on a device. There's absolutely no memory creep when running over long periods of several hours.

    However, very occasionally it crashes after an hour or so on a 128 Kb device  (OK, in reality that's 124) like a Fenix5 or VA-HR , and that's is the mystery I'm trying to solve.

    When running in the simulator, running in my app's  "sim mode" where it spoofs GPS data (position, speed and time) from a previously recorded track file (like replaying a FIT file, but with more bells and whistles enabling controllable start point and time compression) it can report Peak memory of 114Kb .

    I cannot get it to crash in the simulator.

    And that's why I'm curious about why the error log reports VM addresses in the stack trace on an out of memory crash. 

  • Besides the usedmemory there's also a call to get free memory but normally they should be in relation.

    Thinking out of the box could it be there are other ciq apps with a background process ? (Normally apps should run in seperation and shouldn't affect each other but maybe on this device it does, that is why i started with "Thinking out of the box")

    Either way the solution might be to further refactor your code to reduce memory pressure. (It's work intensive but I keep surprising myself that when looking at code again you sometimes see an opportunity to write it in another way, or maybe there's similar code in several places then you have easy gain and can save code memory by introducing a new function)

  • there's also a call to get free memory but normally

      Appreciate your input. used memory + free memory always totals to total memory at around 124 kb.

    could it be there are other ciq apps with a background process ?

    I don't think so. I was running on an F5 with nothing else running own the watch. No opportunity to have another app running.

    refactor your code to reduce memory pressure.

    Today I re-ran the app in the sim using the Activity file generated when it crashed. For 70 minutes it ran, reporting consistent memory usage of between 101 and 103 Kb, with the Peak memory steady at 110.3.  

    I have spent a vast amount of time over the past four years, finding ways to squeeze my design into the available memory, and have got it down to consistently around 102-105 Kb. The vast majority of the time it runs for the full two or three hours of the race. It's just occasionally I get ERA reports of out of memory crashes. This was the first time for months that it crashed for me.

    But since I can't replicate the crash I can't fix it, and I have disappointed users.

    Is it possible that the VM addresses in the stack trace indicate a memory leak in the VM?

  • Is it possible that the VM addresses in the stack trace indicate a memory leak in the VM?

    If that was the case, others would likely be seeing the same thing.

  • 1. From my observation. When you see out of memory error it's to late you don't see it in memory log. For example you can see 100kb and you have 28kb memory reserve. Now, if you create array 29kb it will be memory error and no info about it peak.

    2. I had the error is in line dc.setColor and it was connected with venusq bug (dc was from buffered bitmap).

    3. You haven't shown code but probably error is connected with dc (especially buffered bitmap) or something what is run in background (service/timer).

    4. my last memory error was when used
    string += string;
    and it happens after hours :)

  • Could it be a FRAGMENTATION ERROR  in the VM?

    Look at the characteristics:

    1. The code is working with limited available space: It's consistently reporting used memory of around 100kb in a 124kb space.

    2. There's no evidence of memory creep: It runs for hours in the sim with a fixed Peak Memory of 110 kb.

    3. Only occurs after running a long time, in excess of one hour.

    4.The crashes are very infrequent: We have hundreds of users, and only a handful of ERA reports.

    5. The crash locations appear random: they  occur at many places in the code. 

    I have no idea about how the VM handles memory: all I understand is it manages memory by reference counting. Does anyone know how it manages fragmentation?

    How could I check for fragmentation?

    How could I manage fragmentation?

    [EDIT]

    Searching "fragmentation", I found this old, forgotten, unresolved bug report WERETECH-9896.

  • 3. You haven't shown code but probably error is connected with dc (especially buffered bitmap) or something what is run in background (service/timer).

    Yes, I did: 

    dc.setColor(Graphics.COLOR_BLACK, 0xFFFF00);

    No buffered bitmap.

    Plenty happening in the background with:

    Position.enableLocationEvents(Position.LOCATION_CONTINUOUS, method(:GPSData));

    The GPSData callback function drives the whole system.

  • I would love to see more information on this topic also. Unfortunately I feel like most users are not running into some of these things (or don't care enough) where a more advanced insight would be incredibly helpful in tracking down these things. And it's certainly not at all helpful that the ERA is down with no ETA on getting it fixed and no indication that anything is being done other than a ticket having been opened. ;(

  • I would love to see more information on this topic also.

    Do you have any first-hand experience with similar unexplained Out Of Memory errors?