Communications web request memory leak.

Mike 26 days ago

I am using a simple Communication.webRequest() to a REST-server to retrieve a json. Unfortunately the server response is relatively large.

So I get the -402 (Serialized response was too large) and -403 (Ran out of memory processing network response) errors from time to time.

I have observed, that after the -402 error, the memory goes back to where it was before, but when getting the -403 error, the memory goes up >2Kb every time. So it looks to me as if the memory does not get released properly when the internal json->dict conversion fails.

Mike 1 day ago

I have an idea for a feature request that could serve as a workaround:
Would it be possible to pass a dummy json / dictonary with the webrequest and use this to exclude certain parts of the received json? The internal code that turns the received json into the data dictionary could just ignore those fields defined in the dictionary and save a lot of memory!

example:

ignore = {
"field1" => null,
"field2" => {"field21" => null},
}
Communications.makeWebRequest(url, params, options, ignore, responseCallback);

This would ignore field1 and field21, but not other fields in the response, e.g. field 3 or field22, ...

flocsy🤠 1 day ago in reply to Mike

I'm not sure how the JSON parser Garmin uses works, but what you ask for is not that trivial as it sounds. My bet is that if it would be implemented it would still parse the whole response body and only then it would throw away the unnecessary parts. That would not change the peak memory use.

It might be possible to do what you ask for, but my guess is that it would add complexity and possible bugs as it most probably would involve serious tweaking of the parser code.

However I would definitely support such feature request, as this is part of the reason I am not able to use some public apis I'd like to in datafields.

flowstate 17 hours ago in reply to flocsy🤠

Not directly related, but there was a recent-ish discussion about how writing an object that already exists to storage results in a 300% overhead (meaning if the object is ~100 KB, then an additional ~300 KB of memory is required just to write the object), while writing an object that doesn't exist has a 200% overhead (a ~100 KB object would require ~200 KB of additional memory to be written to storage).

https://forums.garmin.com/developer/connect-iq/i/bug-reports/application-storage-unreasonable-high-overhead-for-deletevalue-200-of-object-size-and-setvalue-200--300-of-object-size

You'd think that ideally, there would be a 100% overhead at most, in both cases.

So this isn't the only case where CIQ code for serialization/deserialization seems to have some serious memory issues.

Mike said:
feature request that could serve as a workaround

Seems that it would be better (and easier) to just fix the memory leak. I guess that wouldn't really fix your original problem where the response is too big to be converted :/, unless the conversion process could be optimized somehow.

Mike 12 hours ago in reply to flowstate

Thanks for the replies. I am sure it is not easy or it would have been done already.

On the other hand json is a tree structure and I think some kine of cut off could be implemented. I think the memory leak comes from some recursive method walking through the json. Maybe a recursion depth of the json would also be a good feature that can at least help in some cases, like Communications.makeWebRequest(url, params, options, 2, responseCallback) would cut off any field in the received json deeper than 2 levels.

As flocsy (237419224e814eb7be09ba868d70607f) said, the current memory restriction and leak prevents the use of many APIs.

flocsy🤠 12 hours ago in reply to Mike

Anyways, I don't give this high chance, but maybe it would be slightly higher if you post this to the bugs forum (since we don't have a dedicated feature request forum) and this is mainly a user forum.

Mike 8 hours ago in reply to flocsy🤠

good point