Background Memory Limits for EXIT() and SETVALUE()

I'm writing a field that requires a significant data set I get from a WebRequest.

I've reduced the code in my background processes so I no longer get the dreaded -403 HTTP error.

But now I've tried to pass the dictionary of the retrieved JSON data in Background.exit(data) and I get an error that the size being passed to exit() is too large. I see from the API docs that EXIT() has an 8KB limit. I'm not sure how to check the storage size of a dictionary. Any ideas? View Memory in the sim doesn't show it.

I've also tried Storage.setValue("data",data), but that also runs out of memory. Any way to check how much space is available in the Object Store?

I might have to resort to sending some of the data using EXIT() and the rest using SETVALUE(). To split up the payload to get all the data to my foreground.

0 flocsy🤠 over 1 year ago

Or you might need to split it to a couple of smaller http requests, then at least the result of each can be saved to the storage.

If it's your api then you can also add some kind of "pagination" to it so that if it gets over 7k it doesn't put more data in it but instead indicates that you need to call to the api again to get the next page.

0 DaveBrillhart over 1 year ago in reply to flocsy🤠

The pagination idea will work. I'm drawing a blank... how can I check the byte size of my dictionary object?

0 flocsy🤠 over 1 year ago in reply to DaveBrillhart

I guess you'll need to try. You probably return some "list" of data, hopefully with some upper limit on the size of one item. Try it with 1, then increase the size of the "page" and when it's saved you should be able to see it in the memory view. Once the backend knows the way to calculate the DF could send the amount of free (I'm not sure what should be here memory? storage?) to the api and the api can decide how to paginate. Or the DF can do the calculation and request the pages accordingly.

0 _psx_ over 1 year ago

Is this dictionary dynamic (different number of object, strings in values) or static (e.g. 5000 numbers)?

Do you generate this dictionary on server?