• Replies 43 replies
  • Subscribers 15 subscribers
  • Views 9726 views
  • Users 0 members are here
Options
Related

Too Many Objects Error

Alan.raceQs
My app works perfectly on the watch in my "Sim" mode (for development/testing) when replaying an old yacht race, but crashes with "Too Many Objects Error" when running with live data from the GPS.
It looks to me like Position.enableLocationEvents(Position.LOCATION_CONTINUOUS, method(:GPSData)); is generating objects, but that doesn't make any sense really.
In my "Sim" mode, I'm calling my app's main processing function with data that I'm retrieving from my web site.
In live mode, I'm calling it with data from the Position.enableLocationEvents callback.
Other than that the two modes are identical.
In "Sim" mode. the app runs for an entire race of over 2 hours.
In Live mode, it crashes after around two minutes.
ERROR: Too Many Objects Error
DETAILS: Failed invoking <symbol>
STORE_ID: 00000000000000000000000000000000
CALLSTACK:
144)*Failed while writing stack trace*

I'm monitoring memory each callback cycle, and memory usage remains static up to the crash.
Memory Total:125272 Used:101208 Free:24056


Sure, I'm pushing the watch, running with around 24Kb free, but with no way to monitor object usage, I'm in a big hole.
The issue has started since I added the latest feature which has increased memory usage by around 30Kb.
Any thoughts (before I slit my throat)?
  • Where do I go now?

    Many thanks to Travis who has worked on the code over the weekend and has seen the crash, but is unable to replicate it.
    Here's the status:


    • We can't reproduce the crash - it appears to be random - re-running with the same data fails to reproduce the crash.
    • It crashes with a "Too many Objects" error but Travis confirms that the app code does not appear to be abusing the objects count
    • and there's no facility to monitor object usage.
    • The app only crashes when run on the watch (not CIQ Sim) using the GPS (not my track replay).
    • It's not happening in the app code.
    • Garmin admits the possibility it's in their code.
    • We can't release a product that is this unstable.
    • Even if Garmin finds a fix, how long will it take to get them to implement it?
  • This is starting to sound like something due to a stray pointer in the VM or FW and something getting corrupted, that might not even be related to GPS...

    The key to when it might get fixed might be based on finding a way to reproduce the problem easily.
  • After a further 10 hours of tweaking and testing, both on land and on water today, I can report that I can increase the MTBF (Mean Time Between Failures) by the following:
    a) Code changes: Removing complexity in almost any area e.g
    chunks of code that process the position data
    the reliance on the compass
    b) Operational procedures. Reduce the number of screen taps that invoke application functions. In the last test of the day, it ran for nearly an hour before I tapped an option that invoked a Ui.ConfirmationDelegate wich caused a crash. This function does not cause a crash when frequently invoked at other times.
    c) I also found that disabling Bluetooth, by leaving my phone at home during the testing, increased the MTBF.

    I think the evidence for it being an issue with the GPS code is weakening, but I don't have an alternate hypothesis as I don't know anything about the underlying architecture.

    If only I could narrow it down to an easily repeatable sequence!
  • I have now completed a further 10 tests.
    Each test consisted of carrying the watch app over a fixed course, with the same initial settings and operating the app in the same manner.
    I have logged the following details
    • Duration of the test time from starting the app to it crashing to the IQ! icon
    • Crash type - "Out of Memory" (MEM) or "Too Many Objects" (OBJ)
    • User activity immediately prior to the crash: None - it crashed without user entry, Wind Set - it crashed as I selected the option to re-set the wind. A normal activity at the end of each leg.
    • Reported crash location: The OBJ error provides a CALLSTACK.


    Test # Duration Type User input Loc'n Code
    1 02:27 MEM None
    2 01:52 OBJ None engine.mc 1250 System.println("In
    3 08:32 OBJ Wind Set
    4 02:31 OBJ None engine.mc 1250 System.println("In
    5 02:35 OBJ None "Writing stack trace"
    6 06:26 MEM None
    7 11:02 MEM None
    8 02:22 OBJ None Engine.mc 546 System.println(
    9 01:41 MEM None
    10 01:44 MEM None

    There is a loose correlation between test duration and crash type: long tests generated MEM errors
    There is a grouping of 7 short tests lasting between 1:40 and 2:35 and a separate group of 3 tests over 8 minutes.
    There is an even distribution of OBJ and MEM errors
    Most of the OBJ errors crashed in my code on a System.println command which gets executed on every GPS cycle (1Hz).
    All but one crash occurred without any user input.

    I am logging memory usage at the end of the main processing function and am seeing Free memory of 23808.

    Without a fix, my app is useless and cannot be released.

    I am convinced the cause lies within the Garmin code which I cannot debug or work around and I urgently request assistance from Garmin to allocate some software engineering resources to resolve the issue.
  • Former Member
    Former Member
    I have replied to the PM you sent me with more details.

    We really need some sample code that can reproduce the error to get an investigation on this issue moving. I have done some looking just based on your reports, but have found no obvious errors in the VM memory heap.
  • I have sent this message to [email][email protected][/email] :

    After two weeks and well over 40 hours of experimentation and code modification I have established a procedure that
    • generates a crash of some sort: either Out of Memory or Too Many Objects
    • with a degree of reliability: it crashes within 12 minutes of starting the test
    • under controlled physical conditions: running on a physical watch along a pre-defined test route,
    • operating the app in an experienced manner: the user has to be familiar with the app and understand the nature of the test environment.
    • but can only be generated using the full app: any removal of code causes a different response, usually extending the MTBF from minutes to hours.


    The request for sample code that can reproduce the issue is beyond my capability.

    I will be able to provide a copy of the code and guidance on setting up a test plan, but this will be time-consuming and operationally challenging for both parties. I am fully motivated to do my part.
    Please advise.
  • Former Member
    Former Member
    I've responded over email, but here is a summary:

    I tested out the app and monitored the number of objects being used.

    After app start: 175
    After GPS connect, increases steadily to: 378
    After setting wind, increases steadily to: 427
    After getting "1ST field to populate, increases steadily to: ~495

    At this point opening the "Set Wind" dialog causes the app to exceed the 512 object limit, and crash.

    I suspect you are using a large array with several doubles stored in each entry. Doubles take up an object in MonkeyC, and you should attempt to use Number or Float instead if possible.
  • After discovering, that the location.toDegrees method returns an array of Doubles, I converted them to Floats and problem solved! (Why they are Doubles, not Floats in the first place is a mystery in itself as Floats provide accuracy down to 0.2 metres which is well inside the position accuracy of the technology!)

    Many thanks to Brian, Travis and the Garmin team for their assistance.
    As a result f this exercise, I suggest the following three recommendations:
    • Can I reinforce my request that the handle/object count be made visible to the developers please?
    • Can you update the API Documentation of the toDegrees method of Class: Toybox::Position::Location to specify that the resultant array is an array of Doubles?
    • And finally, could you add a section to the Programmers guide to warn of the dangers of using doubles. Any values that result from operation of doubles are also typed as Doubles - resulting in an epidemic of these resource-consuming critters.
  • Former Member
    Former Member
    I hope we are able to add the object count in soon. The documentation, we will definitely work on fixing.


    I do think your worst case precision error is more than you have estimated with floating points.
    Here is a link that covers some cases:
    https://sites.google.com/site/trescopter/Home/concepts/required-precision-for-gps-calculations

    This shows that the latitude worst case error ranges from about 0.2 to 1 meters and longitude worst case error ranges from about 0.6 to 2 meters in the US.
    The precision is still within the accuracy of GPS, but you may see some strange behavior when values change around the possible single precision values. All your points will appear to snap to a grid.
  • I think I may have been a bit hasty in my earlier post and underestimated the precision required for the exercise. My on-water testing today exhibited some lack of precision in the results which may be due to loss of precision by using Floats.
    I will explore the use of integers by using your suggestion of x / 180.0 * 0x80000000, but would appreciate a little fleshing out of the reasoning behind the factors.
    I followed the suggested link, but the reference links were broken, so I didn't get the full argument.