How to store a database?

Interesting. I have some data I want to load ONLY once, whenever a user starts the app for the very first time. I now have this hard coded in the code itself. I might want to poke this bear of JSON resources.

Funny Flow, but in another thread from just a few minutes back you said:

"(Arrays are expensive, but dictionaries are even more expensive). "

When you use JSON data, you always start with a dictionary, which can contains arrays.  How expensive is really a matter of your code and the structure of the data.

jim_m_58 said:

Funny Flow, but in another thread from just a few minutes back you said:

"(Arrays are expensive, but dictionaries are even more expensive). "

Not sure what your point is? I agree that dictionaries (with lots of keys and/or nested arrays/dictionaries) are expensive. I disagree with your statement that JSON data has to be a dictionary. Even if it did, you could simply construct your JSON data resource as a single dictionary with a single key whose value is an array, so you'd have a *fixed* overhead for each JSON resource.

Therefore, I also disagree with the implication that hardcoded data is better than JSON resources because "hardcoded data can be anything" and "JSON data must be a dictionary." If you followed my argument, the reason I think JSON data is better is that hardcoded data has a huge overhead in terms of the code to initialize the data (this code is not present for JSON data).

The documentation literally says JSON data can be: a dictionary, array, primitive, or "mixed data" (e.g. any of those things nested within each other.)

So maybe I'm not understanding what you're saying, but it sounds highly misleading to me.

https://developer.garmin.com/connect-iq/core-topics/resources/#jsondata

<resources>
    <jsonData id="jsonDictionary">{"key":"value", "3":"three", "three":3}</jsonData>
    <jsonData id="jsonArray">[1,2,3,4,5,6]</jsonData>
    <jsonData id="jsonMix">[1,{"1":"one"},["a","b","c"]]</jsonData>
    <jsonData id="jsonPrimitive">5</jsonData>
    <jsonData id="jsonFile" filename="data.json"/>
</resources>

jim_m_58 said:

When you use JSON data, you always start with a dictionary, which can contains arrays.  How expensive is really a matter of your code and the structure of the data.

Are you sure about that? Maybe I'm not understanding what you're trying to say. It sounds like you're trying to say that JSON data must *always* be in the form of a dictionary (at the top-level), whereas when you hard code data, that data can obviously be in the form of an array.

I have JSON data where the *top-level* data structure is an array. When I call loadResource(Rez.JsonData.Foo), I get an *array* back.

As I said in my comment, yes, for every resource that you add, one entry has to be added to resource table which will reside in RAM. That's a *fixed* overhead per resource (maybe 24-ish bytes per resource), not the kind of massive overhead that you would get by structuring data (JSON or hardcoded) as an dictionary with lots of keys, as opposed to a flat array full of elements.

I even gave an example above of how hardcoding an array has a huge overhead (proportional to the size of the data) in terms of the code to initialize the array:

Array with 100 numbers (hardcoded):

- 515 bytes for the object in memory

- 1314 bytes for code to initialize the object (this would not exist for a JSON resource)

The same array as a JSON resource would still take 515 bytes for the object in memory, and a handful of bytes for the loadResource() code. Once again, the only question here is whether your app uses other resources. If your app loads other resources, then using JSON resources is a no-brainer, because the resource table has to reside in RAM anyway. If you are not using other resources, then as soon as you call loadResource() even once, you incur the permanent RAM penalty of loading the resource table, which is fixed penalty for each resource in the table. Again, the issue here is for people who have lots of strings for app settings, as the IDs for those strings take up the bulk of the resource table (assuming you don't have a lot of other resources.)

So please point out to me where I said anything that was factually incorrect?

e.g. I have an app where the JSON data xml looks like this:

<resources>
    <jsonData id="jsonLayoutRound240_2" filename="gen/round240.2.layout.json"/>
...

And the corresponding file for that resource looks like this:

[
  4294967295,
  0,
  4294967295,
  0,
...


I'd really love to hear your explanation for how that's not an array? When I call loadResource(Rez.JsonData...) I literally get an array back.

Again, I'm aware that for each JSON resource, an entry has to be added to the resource table which must reside in memory. That's not my understanding of the meaning of the statement "JSON resources have the overhead of dictionaries".

And even if that's what you meant, so what?

It's one dictionary, with one entry per JSON resource. That's a fixed overhead for each resource. The real issue with using dictionaries is when you have lots of keys, or when the values are also dictionaries (or arrays.) For example, a flat array with 10 numbers will be a lot smaller than the equivalent dictionary with 10 keys/values.

Given that the code overhead for a 515 byte array is *1314* bytes when you hardcode the data, I'd prefer to have constant overhead for *one* dictionary entry, which is surely smaller than 1314 bytes.

Usually in Monkey C when we talk about the overhead of dictionaries, we're talking about a complex dictionary with a ton of keys and/or nested array and dictionaries, not the overhead of defining and loading a resource.

Like I never see anyone say "don't use string resources, because of the overhead of dictionaries". (EDIT: well, maybe it’s been said that string resources are expensive, but not in those words. The difference with strings compared to other resources is that apps will typically have lots of strings, especially for app settings where the use of string resources is mandatory)

Nobody ever says "don't use fonts or bitmaps, because of the overhead of dictionaries".

And nobody ever says "don't use layouts, because of the overhead of dictionaries".

All of those things incur the same per-resource overhead as JSON data, in the resource table.

So I don't get why this would be said for JSON resource data, where there's literally no downside as long as your app already loads other resources at run time. And even if it doesn't, if your data is large enough, then moving it to a resource still saves more memory (compared to hardcoding the data.)

TL;DR the statement "dictionaries are more expensive than arrays" has subtext and nuance.

Like, if I define a dictionary with a single key and set the value to a flat array, yeah that's "more expensive" than the flat array on its own, but it's not the end of the world. It's a small fixed overhead.

The usual interpretation of the original statement is more like "if I have a flat array with X values, then the corresponding dictionary with X values and keys is horrendously more expensive." Even a flat array with 2 * X values is probably better than the same dictionary with X keys/values.

If I were to refactor a flat array with X elements into a somewhat equivalent dictionary with X keys/values, that would incur a memory penalty that's proportional to X (as opposed to a fixed penalty). That's *my* understanding of the statement "dictionaries are more expensive (in terms of space) than arrays."

Circling back to resources:

If we're saying that "JSON data is bad because each JSON resource adds an extra entry to the JSON resource table (which is a dictionary I guess)", then I call BS, because that's a FIXED overhead per resource. For each new JSON resource, you're not creating a whole new complex dictionary with lots of keys and nested data, you're just adding one entry to an existing resource table.

It's especially a disingenuous argument because the JSON data itself can be a primitive, array, dictionary or "mix" (e.g. array or dictionary with nested arrays or dictionaries.)

Going back to my example of a flat array with 100 numbers, if I have the following two choices...

A) Hardcode the array, so the total memory usage is: 515 bytes for the object and 1314 for the initialization code

B) Use a JSON resource, so the total memory usage is: 515 bytes for the object, handful of bytes for loadResource() code, plus the size of the JSON resource table (remember it's a *fixed cost* for each resource in the table)

...then my first question is:

1) Do I already call loadResource()?

If the answer is yes, then it's a no-brainer. The resource table is already in memory, and adding one JSON resource will cost me a small fixed amount of memory, regardless of resource data size, which will surely be less than the size of the initialization code, unless my JSON data is something stupid like a single number. (But in this example it's an array of 100 numbers)

If the answer is no, then it's more complicated. Now that I have to call loadResource(), I incur the penalty of having the resource table reside in memory. Then the next question is:

2) How big is the resource table compared to the initialization code for my data?

If SIZE(resource table) >= SIZE(initialization code), then obviously I hardcode.

If SIZE(resource table) < SIZE(initialization code), then I will use JSON data.

--

Also, regarding the comment that "it matters how you structure your data" - of course it does. But if we're comparing apples to apples, then I would compare the same data structure whether it's hardcoded or it's in a JSON resource. So that's a non sequitur - you obviously want to choose the best data structure for your purposes (to maximize speed of access, minimize memory usage, or balance the two), but it doesn't play a role in whether you should use a JSON resource or hardcode your data.