Migrating from Properties to Storage

I am still using properties, but thinking about switching. What I experience with the properties file is unwanted extra memory usage for devices with less storage. I have a datafield that supports the watches with 32kB available for datafields (FR935 etc) and "128kB watches" (Fenix 5x etc). If I add extra options for the 128kB watches, with extra parameters for usage only for those watches in the properties file, it also eats up memory from the 32kB watches. Even though the code is modular and the 32kB watches don't get those data.
Would switching to storage avoid that?

For App settings (things defined as a property in the xml file), switching from Application.getApp().getProperty() to Application.Properties.getValue() won't do much, as the settings are still in the ,set file.

What's confusing is Application.getApp().getProperty() is also used for the old style object store (the .str file), while now it's Application.Storage.getValue() (the new object store)

So if you have a problem due to the size of app settings, changing from the old way to Application.Properties won't do anything. Moving some app settings into the object store will, but then they can't be changed with GCM or GE.

There are things that should have been in the object store all along - things that only your apps saves and uses. Based on the app type, you can have a menu in the app that allows setting things in the object store, but these can only be changed on the device itself.

I'd like Brian.ConnectIQ to chime in here, but I'm pretty sure switching to Application.Storage will reduce memory usage in a running app.

Yes Travis, maybe Brian can chime in. My understanding (though I might be wrong!) is that with Application.Storage it will keep a chunk of it in memory, as it can be up to 128k on disk, and key/values are limited to 8kb each to fit in memory. I guess I've been thinking it will keep things cached to avoid file system reads, so if you had a 4kb object store, I'm thinking much or all of that would be cached with Application.Storage (maybe up to the 8kb limit of the old object store which is also the max key/value in Application.Storage).

There is an index table that is kept in memory, but the key and value data is not.

It should be very easy to verify this.. Just create a simple app with 1000 values stored via AppBase.setProperty() and does the same with Application.Storage.setValue().

using Toybox.Application.Storage;
using Toybox.Lang;
using Toybox.System;
using Toybox.WatchUi;


class Delegate extends WatchUi.BehaviorDelegate {

hidden var _app;

function initialize(app) {
BehaviorDelegate.initialize();
_app = app;
}

function onNextPage() {
var usedMemory = getUsedMemory();
var timeStamp = System.getTimer();

for (var i = 0; i < 1000; ++i) {
Storage.setValue(i, i);
}

System.println(Lang.format("Storage.setValue(..) => $1$ bytes in $2$ ms", [ getUsedMemory() - usedMemory, System.getTimer() - timeStamp ]));
Storage.clearValues();

return true;
}

function onPreviousPage() {
var usedMemory = getUsedMemory();
var timeStamp = System.getTimer();

for (var i = 0; i < 1000; ++i) {
_app.setProperty(i, i);
}

System.println(Lang.format("AppBase.setProperty(..) => $1$ bytes in $2$ ms", [ getUsedMemory() - usedMemory, System.getTimer() - timeStamp ]));
_app.clearProperties();

return true;
}

hidden function getUsedMemory() {
var systemStats = System.getSystemStats();
return systemStats.usedMemory;
}
}


Here is the output after pressing up/down several times in the sim.

AppBase.setProperty(..) => 73920 bytes in 15 ms
Storage.setValue(..) => 6112 bytes in 2075 ms
Storage.setValue(..) => 6112 bytes in 2402 ms
AppBase.setProperty(..) => 73920 bytes in 62 ms
Storage.setValue(..) => 6112 bytes in 1857 ms
AppBase.setProperty(..) => 73920 bytes in 62 ms
AppBase.setProperty(..) => 73920 bytes in 62 ms
AppBase.setProperty(..) => 73920 bytes in 46 ms
Storage.setValue(..) => 6112 bytes in 2090 ms
Storage.setValue(..) => 6112 bytes in 1997 ms


If you change the value to be a String (Storage.setValue(i, i); and _app.setProperty(i, i.toString())), the savings is even more significant.

AppBase.setProperty(..) => 100048 bytes in 63 ms
Storage.setValue(..) => 6112 bytes in 2137 ms


I'm looking for a way to clear Storage values from my development app on my vahr.
I see entries in both APPS/DATA/<appname>.DAT and .IDX
and in APPS/SETTINGS/ <appname>.SET

I delete all these files and run the app, but the Storage values persist!
And when I run the app, both sets fo files reappear.

I suspect the answer is above, but I don't see it.

(I also tried deleting the app from the device, but failed)

Travis, I took a bit different approach to look at this.
function initialize() {
View.initialize();
var st=System.getTimer();
for(var i=0;i<500;i++) {
//Application.getApp().setProperty(i,i);
Application.Storage.setValue(i,i);
}
var wrt=System.getTimer();
var t1=wrt-st;
for(var i=0;i<500;i++) {
//Application.getApp().getProperty(i);
Application.Storage.getValue(i);
}
var end=System.getTimer();
var t2=end-wrt;
var t3=end-st;

time=""+t1+"/"+t2+"/"+t3;
}

function onUpdate(dc) {
dc.setColor(Graphics.COLOR_BLACK,Graphics.COLOR_BLACK);
dc.clear();
dc.setColor(Graphics.COLOR_WHITE,Graphics.COLOR_TRANSPARENT);
dc.drawText(120,100,Graphics.FONT_SMALL,time,Graphics.TEXT_JUSTIFY_CENTER);
dc.drawText(120,130,Graphics.FONT_SMALL,""+System.getSystemStats().usedMemory,Graphics.TEXT_JUSTIFY_CENTER);
}

With two versions of the app, one using the set/getProperty and the other with the set/getValue for Application.Storage, and tested on a fenix 5.
In onUpdate() I display time and used memory

Wow! You're right Travis - a HUGE difference in memory!

set/getProperty, the first run memory was about 24k, second run (as it had the data to load) about 43k. Application,Storage, about 10k every time.

But the interesting thing was the times. setProperty 238 ms, getProperty 195 ms (total 433 ms) while for setValue 29439 ms, getValue 16787 ms (total 46226 ms). I was surprised the watchdog didn't throw me out with the times I see for set/getValue, with this running for 46 seconds!

So there's definitely a cost for what's saved in memory in time, That's why I thought there might be some kind of caching, as it looks like an average of about 30ms each time getValue is called.

Application.Properties (and the older API AppBase.[get|set]Properties) load the dictionary of keys/values into memory when application loads, and save it out when the application shuts down. This means application memory is consumed by this dictionary at all times.


Application.Storage reads/writes values directly from/to disk. This means memory is not actively consumed by keys/values, but access is significantly slower. For values that require frequent reference, it is recommended to cache these values manually in your application and avoid frequent getValue/setValue calls. There is a tracking table required for values stored in Application.Storage. This is a sorted list of key hashes, and costs ~4 bytes per entry.

As you said Brian, Application.Storage is much slower, and while my test of 500 key/value pairs might not be realistic, 50 might be. Maybe a note in the Programmer's Guide/Api doc about the speed, as even with a smaller number of keys, there could be a noticeable delay if things are all loaded at startup? It's one of those things some users might notice, and isn't really shown in the sim.
And maybe something about the app caching things a bit for both reads and writes?

If you have a large number of values that need loaded at startup, it might be a good idea to put them in a container (Array/Dictionary) to reduce getValue/setValue calls also.