Multiple watch faces crashing simultaneously with "out of memory" error (minimal repro app included)

For the last 1-2 weeks, a few watch faces have been crashing on my Epix 2 after it's been running for more than one day. They all crash at the same time, and the only way to get them to load again is to restart the device. I've generally ruled out a memory leak.

The source of the problem seems to be use of Graphics.createBufferedBitmap. This call is "randomly" working fine and randomly causing an out-of-memory issue. ("Randomly" is in quotes because it works fine when the device is freshly started, but after some unknown time, all watch faces making this call fail - including ones I made and one from the store).

Here's what I know:

The 3 watch faces I made (all still in private beta):
- Generally use 70% - 80% memory (on device) - I added a gauge to see it realtime on all my faces
- Memory usage remains constant according to this gauge - there's no memory leak / change in memory over multiple days of consistent use of the same watch face
- 1 of the watch faces worked consistently well with no crashes (memory-related or otherwise) from Dec 1, 2022 through the end of December. The other 2 faces are brand new as of the last 1-2 weeks since the new year.
- All my faces use BufferedBitmap to enable 1hz updating (with a second indicator). For each minute, I draw to the BufferedBitmap and then draw that to the screen. Then, between minutes, if the device is awake, I'll draw the background minute via the BufferedBitmap. Then I'll also draw the second indicator directly to the DC over the top. This allows me to keep the onUpdate call for each second to about 6 milliseconds of execution time.
- All my watch faces use exactly 1 BufferedBitmap set to the full dimensions of the DC. I use anti-aliasing, so I cannot use a pallet. I also do not specify a color depth. This BufferedBitmap is stored in a variable inside the class that extends WatchUi.WatchFace.
All watch faces (ones I made + one I didn't):
- Work fine for the first day or so
- They all crash together at the same time / they all work fine when the device is restarted
- The crash generally happens when the watch face goes from not being displayed to being displayed again (e.g. after the Morning Report or after exiting an activity)

The watch face that's crashing that I didn't make is "Analog Switch"

About a week ago, I began some research into this issue by looking at the error logs on the device and tracking it back to the function where the Graphics.createBufferedBitmap call happens.

To test my hypothesis that this is a system-level issue (and not something related to the design of the watch faces), I created a minimum reproducible watch face (I call it MemoryX for "Memory Explorer"). To create it, I took the new Watch Face project and made very slight modifications to make it use a BufferedBitmap to draw the time to the screen (based on a setting). Once the watch goes into the mode where all the watch faces crash, MemoryX loads fine if I turn off the use of the BufferedBitmap. But if I toggle the setting to make it use the BufferedBitmap, it crashes, too. When using the BufferedBitmap, MemoryX only used 7.3/123.9kB on the simulator. But this crashes on the device.

Questions:

Is anyone else experiencing this or something related? I noticed Gavriel reported another memory issue recently.
Why would they all crash at the same time? Even if one of the faces had an out-of-memory issue, shouldn't that be isolated?
It's odd that this happens "randomly" after some time -- can anyone think of something I need to be cleaning up in onHide/onShow that I'm not?

I can't seem to post formatted code (it's telling me I'm blocked). And I know I can't post images, either. But here's the code (unformatted) inside the onUpdate function. I'll try to post it more clearly in a comment below:

function onUpdate(dc as Dc) as Void {

var timeFormat = "$1$:$2$";

var clockTime = System.getClockTime();

var hours = clockTime.hour;

var timeString = Lang.format(timeFormat, [hours, clockTime.min.format("%02d")]);

if (getApp().getProperty("UseBB")){

if(bbDisplay == null){

//We need a BB but it's null - init it

bbDisplay = Graphics.createBufferedBitmap({

:width=>dc.getWidth(),

:height=>dc.getHeight(),

}).get();

}

else if(bbDisplay != null){

bbDisplay = null;

}

//Will be either main DC or BufferedBitmaps's DC

var drawContext;

//Will print whether we're using BB or not

var textPrint;

if(bbDisplay != null){

drawContext = bbDisplay.getDc();

textPrint = "Using BB";

}

else{

drawContext = dc;

textPrint = "NOT Using BB";

}

drawContext.setColor(Graphics.COLOR_WHITE, Graphics.COLOR_BLACK);

drawContext.clear();

var font = Graphics.FONT_SYSTEM_SMALL;

//Draw time

drawContext.drawText(

drawContext.getWidth() / 2.0,

drawContext.getHeight() / 2.0 - Graphics.getFontHeight(font),

font,

timeString,

Graphics.TEXT_JUSTIFY_CENTER

);

//Draw whether using BB

drawContext.drawText(

drawContext.getWidth() / 2.0,

drawContext.getHeight() / 2.0,

font,

textPrint,

Graphics.TEXT_JUSTIFY_CENTER

);

if(bbDisplay != null){

dc.drawBitmap(0, 0, bbDisplay);

}
}

Top Replies

All Replies

0 myaro over 2 years ago

Here's the code for MemoryX's onUpdate function:

Here's what MemoryX looks like on the SIM. Notice memory usage when using BB.

Should be using BB

0 flocsy🤠 over 2 years ago

Can you log the time or timestamp and the total/free/used memory just before the usage of the buffered bitmap, so we could see on a real device what's going on?

Though if the bug (as it seems) is a memory leak in the device firmware then probably we won't even see it, because the app will think it only uses 4k, still has 120k free, but then trying to allocate memory the firmware realizes it doesn't have enough.

You can also try this: add another setting: a number. And before (or instead?) the buffered bitmap code allocate that amount of memory, then free it. The goal is to see how much you can allocate successfully.

0 jim_m_58 over 2 years ago in reply to flocsy🤠

Don't forget that with CIQ 4 devices, things like the buffered bitmap are not in the apps memory, but in the graphics pool. That's why the createBufferedBitmap call is specific to CIQ 4 devices.

0 flocsy🤠 over 2 years ago in reply to jim_m_58

So it looks like the memory leak is when the watchface switches modes. Probably the app is killed but the buffer isn't released.

While it's probably something Garmin should fix, maybe there's something a developer can do: release it in the onStop or onWhatever callback. Though unless Garmin fix it the other developer's apps that don't do this can still eat up all the memory and still crash our apps.

myaro can you try this? If I'm right then you should open a bug report, this sounds like a serious bug - though Garmin won't have any choice but to fix it, it will break probably some apps that might somehow use this.

0 myaro over 2 years ago in reply to flocsy🤠

Hi Gavriel and jim_m_58, thanks for the comments.

I think jim_m_58's context explains why I'm not seeing any memory impact of the BufferedBitmap in the sim. I have two clarification questions, if you know the answers:

Does this mean that the BufferedBitmap doesn't count against our apps' memory? I suspect they don't, but I couldn't find clarification.
And do you know how to find out what size it's taking up? It doesn't seem visible inside the sim's memory profiler. And whenever I tried to calculate the size, I came up with outrageous numbers (relative to the size allowed for our apps) . For example: the Epix 2 has a 416px round screen with 5 bits per pixel. That's 416px*416px*5bits/px = 865,280 bits or 108,160 bytes. That would be nearly all the memory allowance of my app, so I assume either the calc is wrong or it doesn't count against the memory limit.

To Gavriel's point about there being a Garmin memory leak, that sounds like the best answer. That would explain why this issue seems to be impacting multiple watch faces. It also explains why it seems to happen after the watch face is hidden (e.g. for "Morning Report" or an activity).

The question is: what's the correct way to work around this? Do I attempt to free up that resource in:

WatchUi.WatchFace.onHide() and WatchUi.WatchFace.onShow()?

Application.AppBase.onStop() and Application.AppBase.onStart()?

Does anyone have good example code showing the right way to use WatchUi.WatchFace.onHide?

0 jim_m_58 over 2 years ago in reply to myaro

For your questions, start with

https://forums.garmin.com/developer/connect-iq/b/news-announcements/posts/a-whole-new-world-of-graphics-with-connect-iq-4

The graphics pool is shared between apps. Things there are freed up as needed.

There have been a few cases in the past where one app can "step on" memory in such a way it causes other apps to crash until you reboot, Not really a leak, but corruption, Try the watch faces one after another after a watch reboot, starting with a different one each time and see if you can narrow down which might be the one causing the issue.

0 flocsy🤠 over 2 years ago in reply to jim_m_58

Why bother? He knows that it happens to his own WF.

So IMHO, add log to onHide, onStop so you see when they are called.

Then add the freeing code to onStop (my guess, but use common sense after the logs). (maybe read more about the lifecycle to understand if it's a good idea to add it to onHide)

Then test the same scenario that until now caused the Out of memory error after 1 day (maybe try to avoid the other suspicious CIQ apps for this test, but that also means that you need to do the "bad" test also without them, and without the freeing code so you know after how much time it would happen)

See if it worked. If it did, open a bug to Garmin, and keep the freeing code. If it didn't.... open a bug to Garmin, but I don't know what are the chances it'll get any attention unless you come up with some new findings.

0 _psx_ over 2 years ago in reply to jim_m_58

So is it possible such scenario

- wf starts and uses graphic pool - takes 5kb of it's memory and reserve it using bmp.get()

- user exits to widget

- in graphic pool bmp still occupies 5kb even wf is unloaded?

The only case of sharing can be possible in data fields (2 of more ciq apps on screen) or on widget list. But no in case of watch face, I think.

0 jim_m_58 over 2 years ago in reply to flocsy🤠

The reason is to id which of his 4 watch faces is causing the issue for the others. Sounds like the issue started with the newest two. As I've said this has happened in the past, but in that case WF's from other devs were impacted. It's rare, but might happen

0 flocsy🤠 over 2 years ago in reply to jim_m_58

It doesn't matter IMHO. If the workaround works, then it's a good idea to find out, contact their devs and send them the workaround. But if it doesn't work, then only Garmin could fix it.