VAHR: makeWebRequest with valid JSON ends up with code -400

Hello, my watchface is getting JSON from my server, but what started happening recently is that only the first request after the app is installed succeeds (code 200), and all subsequent requests (even though they are the same) fail with error code -400 (INVALID_HTTP_BODY_IN_NETWORK_RESPONSE).

The url is https://pwys.tmskfk.com/?lat=38.8914...jor=0&minor=35 - you can check that it returns valid JSON.

The only idea I have is that the response has "Content-Type: application/json; charset=utf-8" - normally it should be just "Content-Type: application/json", but this apparently allows for security vulnerability and thus express.js (the framework I use on server) has disallowed it and automatically adds charset. See https://github.com/expressjs/express/issues/3490.

Can this be the cause?

tomaskafka over 6 years ago

Or, one more idea, is it possible that there is a fixed maximum possible size of incoming JSON? And anything that is over this limit is cut off, resulting in invalid response body (as truncated JSON is no longer valid)?

jim_m_58 over 6 years ago

Yes, there is a max as it's not just the json data, but that gets converted to a dictionary before the app sees it. How large/complex is your response and how much memory is available in your app? It's even more limited on a WF with backgrounding, as the max size for a background process on a vahr is 16k (code and data). With backgrounding, you want the response as small as possible.

A suggestion. Try doing your makeWebRequest in a widget or device app in the main process first and make sure that works, and then look at getting it to work in a background process. It's easier to see what's going on in the main widget/devices app with more memory and to see with "View Memory" in the sim than in a WF.

tomaskafka over 6 years ago

Hi Jim, thank you! Is there any chance you work for Garmin and could tell a bit about implementation details?

Yes, I'm low on RAM and it's a WF background service, but it worked quite reliably for about a year (although with a tight gaps - with about 1 kB of RAM left). I know that when I had extra features, the request would crash the watchface because of OOM, but this time I just consistently get -400 response. Can this also happen because of running out of RAM?

Another thing I'm thinking of is, I recently hardened HTTPS on my server, which makes it send some extra headers. Do you know if these headers are also seen and processed by the app? (or, alternatively, maybe the watch app just asks Connect on the phone to send the request, and only pass the response content?)

jim_m_58 over 6 years ago

The folks that work for Garmin all have forum names that end in ".ConnectIQ". It's just I've been doing CIQ since 1.0.0 and one of the first things I wrote was a widget that got data from the internet.

tomaskafka over 6 years ago

Thank you! Garmin devs, could I please have your eyes on this?

I have now narrowed down what's happening:

I have exact same code getting JSON from exact same url, but on VAHR (latest firmware):

1. In simulator, everything works - background requests fetch data every time (and there is enough free RAM - about 7 kB in background service and 4 kB in watchface, for about 800B JSON)

2. On real VAHR device, with an app built through "Build for device" (both release and debug builds) everything works too.

3. BUT: On a same real VAHR device, same code, but the app was built for Garmin store, and installed from a store (https://apps.garmin.com/en-US/apps/2de2ee03-0f81-4668-b885-120def9f07c4): Only a first request after installing a watchface ends well (code 200, and weather is fetched), all subsequent requests return error code -400.

This happens no matter if I set SDK 2.4.9 or 3.0.2 as active (I supposed that switching this could mean that the app will be built with different SDKs, but the result is the same).

This seems pretty unreal, but I already tried this twice. And always, app built directly for watch is OK, app built for Garmin Connect IQ store and downloaded from there will fail (and users with VAHR are writing me that the watchface stopped updating :(( ).

Any idea what;s happening please?

tomaskafka over 6 years ago

Sooo, here's the solution:

makeWebRequest params cannot have NULLs as values. When this happens (params = { "email" => null }), it works OK in simulator, works OK when built for device, and fails with -400 code when released in a real app. Sigh. Really ugly bug, documented nowhere. At least I made my website get A+ in SSL Labs test, suspecting the HTTPS certificate.

Here's a minimal sample - a widget, tap it to run a random request, the one with null values fails - but only in production on a real device.

https://github.com/tkafka/garmin-makeWebRequest-bad

tomaskafka over 6 years ago

Also documentation of makeWebRequest parameters here is ambiguous: https://developer.garmin.com/downloads/connect-iq/monkey-c/doc/Toybox/Communications.html#makeWebRequest-instance_method

These values should not be URL encoded.
Can be null. <- ONLY THE WHOLE PARAMETERS CAN BE NULL. INDIVIDUAL VALUES IN DICT CAN NOT!

Coleman.Apps over 6 years ago

tomaskafka,

Thank you for all the hard work tracking down what was causing your JSON to be rejected. I'm going to have to test to see what the scope is here and where this issue lies in the ecosystem exactly, but this is the perfect set of information for me. I've got a ticket started and we will get to work on it!

Thanks,
- Coleman

Update: It looks like this is an older bug rearing its face again. I've reached out to get some movement on a fix.

Tag: WERETECH-5208

tomaskafka over 6 years ago

Coleman: Thanks! :)