Memory consumed by Storage.setValue

My guess is that something like this happens:

The storage (at least values already read) are cashed in the memory. I think this is either documented or at least I remember it from the forum, that the new Storage and Properties classes hit less on the battery because they do less I/O than the AppBase.get/setProperty functions.

Which would explain why this:

var x = Storage.getValue("x");
Storage.setValue("x", x);

uses more memory than:

var x = Storage.getValue("x");
Storage.deleteValue("x");
Storage.setValue("x", x);

because at the time you call setValue, the object needs to be serialized. That takes memory. More or less the same amount that the supposedly cashed value of x in the memory. So during the setValue it uses twice as much memory (1x in the cashe, 1x for serialization)

If you 1st delete it then I guess it's also deleted from the cache. Maybe it's worth to look at freeMemory also between deleteValue and setValue, you can also check if what I remember about the values being cached and only flushed to the "disk" on app exit is correct, by also testing this with and without Storage.getValue("x") before the deleteValue

flocsy🤠 said:

The storage (at least values already read) are cashed in the memory. I think this is either documented or at least I remember it from the forum, that the new Storage and Properties classes hit less on the battery because they do less I/O than the AppBase.get/setProperty functions.

I think it's the other way around. 

The old (and deprecated) AppBase.getProperty() / setProperty() cached all values in memory. Any value set using setProperty() would only be written when the app exits.

And we can say for sure that the newer Application.Storage API writes data immediately when you call setValue(), which means that it must be equally or more I/O intensive than AppBase.get/setProperty. (Ofc that doesn't rule out the possibility that Application.Storage is also cached in memory somehow.)

[1/3]

[https://developer.garmin.com/connect-iq/core-topics/persisting-data/]

Accessing Properties and Settings: Object Store

This doesn't confirm that Application.Storage *isn't* cached, but it's implied that newer methods of persisting data cost less memory than AppBase.get/setProperty.

Interestingly, the docs also say this:

Accessing Properties and Settings: Application.Properties
Since API level 2.4.0

The Application.Properties module provides an interface for accessing the values and properties of settings. Information is automatically saved on disk when AppBase.onStop() is called. To get or set a property value use the Properties.getValue() or methods, respectively:

Hmmm, sounds just like AppBase.getProperty/setProperty....

The newer properties/storage docs have been known to be wrong in the past (*some* of those errors have been fixed [*]), but I have no reason to doubt the stuff I quoted.

It does seem to me that Garmin really wants to say that if you were using AppBase.getProperty/setProperty for all persistence in the past, you should switch to Application.Storage for non-settings / non-properties related persistence and Application.Properties for settings-related / properties-related persistence.

The biggest difference - afaik - between the old AppBase.get/setProperty and and the new Application.Properties, is that the latter API will throw an exception if you try to get or set a key which is not defined as a property.

[*] basically the Application.Properties api docs are inconsistent about whether the key to be get or set needs to exist in properties or settings. It's actually properties.

[3/3]

So I did quick test:

- created a test data field for fr955

- created an array of about 4500 bytes. With just this addition, the peak memory was 18.3 kb

- added one line to save the array to Storage (setValue). With this additional line of code, peak memory was 25.4 kb

- added an additional line to load the array (getValue) from Storage again (without actually assigning the result to any variable. With this code, peak memory was 29.8 kb

So far so good. (nothing out of the ordinary).

- Finally I added Storage.deleteValue between setValue and getValue. Peak memory was still 29.8 kb after this addition.

For me, it didn't seem like Storage.deleteValue had any effect on the peak memory. I also did not see any indication in the memory viewer that Storage was being cached. (Unlike application properties/settings, which can be seen in the memory viewer, along with the resource table - indices, not contents - if you load at least one resource.)

Maybe I'm doing something wrong, like not testing in a glance or something. But I don't think what you're seeing is a general situation.

EDIT: oops this test doesn't even make any sense haha.

Ofc it would make a difference if I set a value, delete a value, then get the same value, compared to the same thing without deleting it. Indeed, I tested with a 9000 byte array and there's a huge difference.

I obviously did the steps in the wrong order

SylvainGa said:

Weird that a setValue which is supposed to overwrite what's there in the first place takes that much free memory...

Assuming that the data needs to be deserialized/serialized when reading/writing from disk, and that Garmin does not do so incrementally, but all at once, I would expect reading or writing any given data structure to take at least twice the size of the data structure. Having said that, it doesn't make sense that would have enough memory to read a given array, but the same amount of free memory isn't enough to write the same array.

e.g.

Let's say I want to write a 7 KB array to storage.

Assuming:

- that the serialized form is substantially different than the in-memory form

- that Garmin doesn't serialize chunk-by-chunk while it writes, but instead serializes all the data then writes the entire serialized form to storage

- that serialized form of the data is at least as big as the data (in the case of inefficient serialization formats like JSON, it could be much bigger, although I'm fairly sure Garmin uses some binary format)

Then I would expect the to need at least an additional 7 KB (or more) of memory just to write that 7 KB array to storage.

Having said that, I would expect to only need the same amount of overhead to read that same array from storage (by the exact same reasoning).

Meaning that if I had enough extra memory to read an array, then that memory should be sufficient to write the array.

I don't really get how deleting the existing value from storage would help. 

Me neither. But it was consistent between the Simulator and my real device. Both kept crashing with out of memory error until I added the DeleteValue before the SetValue. In the simulator, it was easy to copy over the .DAT/.IDX file and try and try again to figure out why it crashed and try to fix it. Once I got it working, I pushed the app on my real device (which, like I said was crashing too) and the crash stopped. Very weird.

Yeah, that's weird. Also, as I said above, my test was completely backwards and nonsensical. I'll try a proper test and get back to you.

I did a proper test and got the same results as you.

First I create a 9015 byte array in memory (code) and saved it to storage. Then I deleted the array from the code. Note that the DAT file for the array was about 9 KB.

I tried several tests:

- Peak memory with no hardcoded array and no additional code was 9.5 KB

- I added code to read the array using Storage.getValue. Peak memory was 25.3 KB.

25.3 - 9.5 KB  = 15.8 KB which is a little less than double the size of the array in memory

- I added code to write the array back using Storage.setValue. Peak memory was 43.0 KB!

- Finally I added code to call Storage.deleteValue in between the get and the set. Peak memory was 34.2 KB

It doesn't make sense to me that in both cases - delete or no delete - the peak memory for writing the array is higher than the peak memory for just reading the array.

Since the array was both read and written from the same place, I would expect the peak memory to be the same. Even though the setValue() call occurs once the array has already been read, the same can be said of the peak that is measured when only getting the array and not setting it.

So I think there's at least 2 "problems" (or inexplicable situations) here, unless I am missing something:

- writing a value to storage apparently has more memory overhead than reading the same value (unless my reasoning is bad, which is entirely possible haha)

- writing a value to storage takes less overhead if the value doesn't already exist in storage??? (like you said)

Also, I printed out the amount of free memory at each step. Ofc this is not as informative as the *peak* memory, but maybe it can give us a hint if Storage is somehow cached in a way we can't see in the memory viewer and/or whether deleting an item from Storage actually affects *application* memory.

    function printMemory() {
        System.println(System.getSystemStats().freeMemory);
    }

    function test() {
        System.println("----");
        System.println("before reading");
        printMemory(); // 241216
        var bytes = Application.Storage.getValue("test");
        System.println("after reading"); 
        printMemory(); // 232192
        Application.Storage.deleteValue("test");
        System.println("after deleting"); 
        printMemory(); // 232208
        Application.Storage.setValue("test", bytes); 
        System.println("after setting"); 
        printMemory(); // 232192
    }
    
// full output:
/*
before reading
241216
after reading
232192
after deleting
232208
after setting
232192
*/