Multiple watch faces crashing simultaneously with "out of memory" error (minimal repro app included)

For the last 1-2 weeks, a few watch faces have been crashing on my Epix 2 after it's been running for more than one day. They all crash at the same time, and the only way to get them to load again is to restart the device. I've generally ruled out a memory leak.

The source of the problem seems to be use of Graphics.createBufferedBitmap. This call is "randomly" working fine and randomly causing an out-of-memory issue. ("Randomly" is in quotes because it works fine when the device is freshly started, but after some unknown time, all watch faces making this call fail - including ones I made and one from the store).

Here's what I know:

The 3 watch faces I made (all still in private beta):
- Generally use 70% - 80% memory (on device) - I added a gauge to see it realtime on all my faces
- Memory usage remains constant according to this gauge - there's no memory leak / change in memory over multiple days of consistent use of the same watch face
- 1 of the watch faces worked consistently well with no crashes (memory-related or otherwise) from Dec 1, 2022 through the end of December. The other 2 faces are brand new as of the last 1-2 weeks since the new year.
- All my faces use BufferedBitmap to enable 1hz updating (with a second indicator). For each minute, I draw to the BufferedBitmap and then draw that to the screen. Then, between minutes, if the device is awake, I'll draw the background minute via the BufferedBitmap. Then I'll also draw the second indicator directly to the DC over the top. This allows me to keep the onUpdate call for each second to about 6 milliseconds of execution time.
- All my watch faces use exactly 1 BufferedBitmap set to the full dimensions of the DC. I use anti-aliasing, so I cannot use a pallet. I also do not specify a color depth. This BufferedBitmap is stored in a variable inside the class that extends WatchUi.WatchFace.
All watch faces (ones I made + one I didn't):
- Work fine for the first day or so
- They all crash together at the same time / they all work fine when the device is restarted
- The crash generally happens when the watch face goes from not being displayed to being displayed again (e.g. after the Morning Report or after exiting an activity)

The watch face that's crashing that I didn't make is "Analog Switch"

About a week ago, I began some research into this issue by looking at the error logs on the device and tracking it back to the function where the Graphics.createBufferedBitmap call happens.

To test my hypothesis that this is a system-level issue (and not something related to the design of the watch faces), I created a minimum reproducible watch face (I call it MemoryX for "Memory Explorer"). To create it, I took the new Watch Face project and made very slight modifications to make it use a BufferedBitmap to draw the time to the screen (based on a setting). Once the watch goes into the mode where all the watch faces crash, MemoryX loads fine if I turn off the use of the BufferedBitmap. But if I toggle the setting to make it use the BufferedBitmap, it crashes, too. When using the BufferedBitmap, MemoryX only used 7.3/123.9kB on the simulator. But this crashes on the device.

Questions:

Is anyone else experiencing this or something related? I noticed Gavriel reported another memory issue recently.
Why would they all crash at the same time? Even if one of the faces had an out-of-memory issue, shouldn't that be isolated?
It's odd that this happens "randomly" after some time -- can anyone think of something I need to be cleaning up in onHide/onShow that I'm not?

I can't seem to post formatted code (it's telling me I'm blocked). And I know I can't post images, either. But here's the code (unformatted) inside the onUpdate function. I'll try to post it more clearly in a comment below:

function onUpdate(dc as Dc) as Void {

var timeFormat = "$1$:$2$";

var clockTime = System.getClockTime();

var hours = clockTime.hour;

var timeString = Lang.format(timeFormat, [hours, clockTime.min.format("%02d")]);

if (getApp().getProperty("UseBB")){

if(bbDisplay == null){

//We need a BB but it's null - init it

bbDisplay = Graphics.createBufferedBitmap({

:width=>dc.getWidth(),

:height=>dc.getHeight(),

}).get();

}

else if(bbDisplay != null){

bbDisplay = null;

}

//Will be either main DC or BufferedBitmaps's DC

var drawContext;

//Will print whether we're using BB or not

var textPrint;

if(bbDisplay != null){

drawContext = bbDisplay.getDc();

textPrint = "Using BB";

}

else{

drawContext = dc;

textPrint = "NOT Using BB";

}

drawContext.setColor(Graphics.COLOR_WHITE, Graphics.COLOR_BLACK);

drawContext.clear();

var font = Graphics.FONT_SYSTEM_SMALL;

//Draw time

drawContext.drawText(

drawContext.getWidth() / 2.0,

drawContext.getHeight() / 2.0 - Graphics.getFontHeight(font),

font,

timeString,

Graphics.TEXT_JUSTIFY_CENTER

);

//Draw whether using BB

drawContext.drawText(

drawContext.getWidth() / 2.0,

drawContext.getHeight() / 2.0,

font,

textPrint,

Graphics.TEXT_JUSTIFY_CENTER

);

if(bbDisplay != null){

dc.drawBitmap(0, 0, bbDisplay);

}
}

Top Replies

All Replies

0 myaro over 2 years ago in reply to jim_m_58

When I went from making WF 1 to making WF 2, I got really excited about the opportunity of using Barrels to factor out all the parts of the project that I find myself repeating. It's been a bit less smooth than I hoped, though. I wouldn't be surprised if this bug comes from a Barrel-related corner case.

Some issues I encountered:

Rez strings from barrels can't be accessed in Settings.xml of the main app, as implied by documentation (reported here)
When importing barrels raw (not the compiled versions), making changes in the barrel code and then re-running the main app code won't recompile the changed barrel code. I have to "touch" an .mc file in the main project before re-running for it to show the new changes. (not yet reported - low priority)
Now this issue seems related. I'll report back tomorrow.

It's making me wonder whether barrels may be a bit more of an advanced feature used by a smaller subset of devs, which therefore has less polish because it's not used as much. I've started considering some other route of shared code & resources.

0 peterdedecker over 2 years ago in reply to myaro

myaro said:
It's making me wonder whether barrels may be a bit more of an advanced feature used by a smaller subset of devs, which therefore has less polish because it's not used as much.

I think it was the wrong call from the dev team to not support older devices. Barrels could have been implemented as modules then no Firmware changes would have been necessary

0 jim_m_58 over 2 years ago in reply to myaro

I have about a dozen barrls and I built them so the mc files could also be directly included, for devices that don't support barrels, or where I don't want the extra overhead. The result is I still have common code, that I can use two ways - with and without barrels.

Here's part of one of my jungle files where I use things directly: I have different view files for the semioctagons devices in this case

project.manifest = manifest.xml

project.typecheck = 0

#barrels clocks.
barClkGoals=..\ClockGoals
barClkFields=..\ClockFields
barClkIcons=..\ClockIcons
barClkIconsNew=..\ClockIconsNew
barClkMisc=..\ClockMisc
barClkSettings=..\ClockSettings
barClkSun=..\ClockSun

###############################
#watchfaces, etc
###############################
#using goals and sun
base.sourcePath=source;source-basic;$(barClkGoals);$(barClkIconsNew);$(barClkMisc);$(barClkSettings);$(barClkSun)
semioctagon.sourcePath=source;source-so;$(barClkSettings)

0 myaro over 2 years ago in reply to jim_m_58

The updated WF crashed again last night, this time more "on-demand" when I toggled Sleep mode on the device before bed.

This was the WF version where the BufferedBitmap var was contained in the View.mc file (rather than the barrel), but the barrel continued to provide the superclass for my View.mc file, and onUpdate() is implemented ONLY in the superclass. The subclass gets a call to a custom paint() function with some additional parameters calculated by its parent's onUpdate class.

Interestingly, rather than crashing upon waking up (me and the device) in the morning, this crash happened while manually toggling sleep mode. The watch went to sleep before I did, so I toggled Sleep mode off. Then, before actually sleeping, I toggled Sleep mode back on and the WF crashed.

The CIQ team reached out to me for more info, so I'll continue exploring and debugging and report back. I'm next going to see if I can reliably cause this crash via toggling Sleep mode at any time. Then, if so, I'll work backwards and try to identify whether it's the existence of the Barrel that's causing the crash.

0 _psx_ over 2 years ago in reply to myaro

my WF PSX-1 uses buffered bitmaps, no barrel and only my code (no any copy of code, especially from analog example that your barrel is based). So you can test it if crash too (I have only f6p and of course no crash).

0 myaro over 2 years ago in reply to _psx_

Great! I'll explore it once I'm done with my testing with my app as the only one on-device.

+1 myaro over 2 years ago in reply to myaro

Quick update to share:

After a few iterations of trial-and-error, I think I may have found the cause & workaround.

In the onUpdate() code, there was a code path through which certain bitmaps (packed from PNGs) got loaded during each update. Even if it was already loaded, it would load the bitmap again. This happened in every update - every second in awake-mode or every minute in AMOLED always-on mode.

The performance impact seemed negligible, so it wasn't obvious to me this was happening unnecessarily. Since these loaded assets don't seem to impact the memory counter, it wasn't getting detected there either. And there was no indication on-device of excessive battery use. This was silent except for when the watch face would go to create a new BufferedBitmap, which would fail.

I created a new utility class for use across my ecosystem, which has resource slots for each icon that could get displayed on the screen. That way, for a given slot, the resource that's loaded can either be reused if it matches the requested resource in that update. Otherwise, it nulls the old one and loads the new one.

This cut down on unnecessary loading while keeping most of the code the same. So far, I've gone 24 hours without a crash.

What my code was doing was definitely an anti-pattern. But it's definitely an issue that one watch face's bug was able to bring down all the others using graphics pool / buffered bitmaps.

For contribution back to the community, here's the cache code I wrote. Make a single ResourceCache and then just call getResource() with a slot number (you manage) and the resource. If the slot gets called with the same resource, no extra loading happens. If it's a new slot or the resource changes, it loads the new resource and caches it.

0 flocsy🤠 over 2 years ago in reply to myaro

Very good news! Though as you wrote, still sounds like the real solutions is in Garmin's hand: they need to protect the same from happening somehow.

0 flocsy🤠 over 2 years ago in reply to myaro

Some advice (general to all languages BTW): the lookup in a map is expensive, and you do it twice here: hasKey, get. Only use hasKey if you don't care about the value. But in your case you do, so better:
var rezItem = resourceSlots.get(slotNumber);
if (rezItem != null) {...}

0 zoouuu over 1 year ago

Hello,

I had the same behavior with same outcome: all watch faces crashing at the same time.

I believed that was fixed now, but in fact I still have same crash on fenix 7 version 15.76.

The watch face runs a command

var affineMatrix = new Graphics.AffineTransform();

every second onPartialUpdate and crash after around 4 days.