Webrequest response issue

I had a watch face that I have been testing for a while and recently stopped working. I am looking at the new stuff now, but I cannot figure out what changed.
It is a background webrequest.

Looking at the http traffic logs on the simulator, it seems to work fine in communicating and receiving data.
But in eclipse (and on the watch face) I do not receive the data.
I get this error: Background: Communication Error Code: -400 but in the http logs it is 200.

So I assume is something in the way data are parsed.

It worked with just {:responseType => Comm.HTTP_RESPONSE_CONTENT_TYPE_JSON }
Now it doesn't anymore.

This is the response I get from the http logs:
In the header:
HTTP/1.1 200 OK\r\n

In the body:

37c\r\n{"status":200,"callCount":1,"copyright":"Tidal data retrieved from www.worldtide.info. Copyright (c) 2014-2017 Brainware LLC. Licensed for use of individual spatial coordinates on behalf of\\/by an end-user. Copyright (c) 2010-2016 Oregon State University. Licensed for individual spatial coordinates via ModEM-Geophysics Inc. NO GUARANTEES ARE MADE ABOUT THE CORRECTNESS OF THIS DATA. You may not use it if anyone or anything could come to harm as a result of using it (e.g. for navigational purposes).","requestLat":9.98,"requestLon":-85.64,"responseLat":10,"responseLon":-85.6667,"atlas":"TPXO_8_v1","requestDatum":"MLLW","responseDatum":"MLLW","extremes":[{"dt":1544482804,"date":"2018-12-10T23:00+0000","height":1.863,"type":"High"},{"dt":1544504290,"date":"2018-12-11T04:58+0000","height":0.17,"type":"Low"},{"dt":1544526199,"date":"2018-12-11T11:03+0000","height":1.951,"type":"High"}]}\r\n0\r\n\r\n

Any help it is greatly appreciated.

agsurf5 over 6 years ago

Ps in the header I also get this:
Content-Type: text/json; charset=utf-8\r\n

Travis.ConnectIQ over 6 years ago

If you look at the docs, the error code -400 maps to INVALID_HTTP_BODY_IN_NETWORK_RESPONSE[/url]. This error can occur in a few cases, but the one you're seeing is where you declare that the response body will be application/json (by specifying :responseType => Comm.HTTP_RESPONSE_CONTENT_TYPE_JSON in the options dict) and the web server sends back a response with a header that indicates the body is not application/json.

To debug things like this, one of the best things you can do is to reproduce the web request and then inspect the response (including the HTTP headers). I don't know the exact query you're making, but I can make up a request using some information from a similar question from the past:

C:\Users\vitek\Desktop>curl -v -k -L "www.worldtides.info/api * Trying 69.64.65.73... * Connected to www.worldtides.info (69.64.65.73) port 443 (#0) * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * TLSv1.2 (OUT), TLS Unknown, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * NPN, negotiated HTTP1.1 * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Unknown (67): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * Server certificate: * subject: CN=worldtides.info * start date: 2018-10-23 12:03:06 GMT * expire date: 2019-01-21 12:03:06 GMT * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway. > GET /api?extremes&lat=33.768321&lon=-118.195617&start=1535576400&length=86400&key=0 HTTP/1.1 > Host: www.worldtides.info > User-Agent: curl/7.44.0 > Accept: */* > < HTTP/1.1 400 Bad Request < Server: nginx < Date: Mon, 10 Dec 2018 23:47:39 GMT < Content-Type: text/json; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET, POST < Access-Control-Allow-Headers: Origin, Content-Type, Accept, Accept-Language, Content-Language < {"status":400,"error":"API key is invalid"}* Connection #0 to host www.worldtides.info left intact C:\Users\vitek\Desktop>

Note that the response header indicates the Content-Type is text/json? I'm not sure if the folks over at worldtides.info have recently made a change, but this seems wrong.

To make matters more confusing, the simulator has support for making HTTP requests over WiFi or over Bluetooth. When the simulator uses the WiFi code path, it validates the content type as described above. The Bluetooth code is a little more relaxed and allows it through. If the simulated device has music support, the simulator will prefer to send web requests over WiFi, which exposes you to this problem.

If you want to simulate a music-enabled device but avoid this error, you can use the Settings > Connection Type menu to disable the WiFi code path and its more strict checking. I wouldn't suggest this as a long term solution though, it is just something you can do to get unstuck.

Travis

agsurf5 over 6 years ago

Thanks Travis.
This helps a bit, but I am still lost.

Nothing changed on my side. And I see other apps that use the same Worldtides API still working.
I just tried the wifi path you mention, and I still get the same error.

I think the error is in the content type, but I am not sure why it is triggered now.

agsurf5 over 6 years ago

Turns out it was some small modification in the grammar and it worked. I am not sure what changed, but I think it is some stricter enforcement. This one works:

Comm.makeWebRequest(
             "www.worldtides.info/api",
            {"extremes"=>"","lat"=>lat,"lon"=>lon,"length"=>"86400", "datum"=>"MLLW", "key"=>"XXXXXmyKEY"},
            {:responseType => Comm.HTTP_RESPONSE_CONTENT_TYPE_JSON},
            method(:onReceive)
            );

Travis.ConnectIQ over 6 years ago

I can't tell what you've changed if you just show the 'after', but I'm pretty sure the above is still subject to the error.

If you are simulating with a music-based device, you will see errors like this. If you simulate the fenix5 and run your app, I'd expect it to work just fine. If you switch to a fenix5plus, I'd expect it to fail... unless you disable the WiFi connection as I mentioned above.

Travis

Alan.raceQs over 6 years ago

While we're looking at HTTP responses, there's a typo in the doco (both the web version and the SDK package) which led me up the garden path for a while, between HTTP_RESPONSE_TYPE and HTTP_RESPONSE_CONTENT_TYPE. When I went searching for valid HTTP_RESPONSE_TYPEs, I failed to find HTTP_RESPONSE_CONTENT_TYPEs!

agsurf5 over 6 years ago

While we're looking at HTTP responses, there's a typo in the doco (both the web version and the SDK package) which led me up the garden path for a while, between HTTP_RESPONSE_TYPE and HTTP_RESPONSE_CONTENT_TYPE. When I went searching for valid HTTP_RESPONSE_TYPEs, I failed to find HTTP_RESPONSE_CONTENT_TYPEs!

Can you clarify more? It is unclear to me. And I am back to error code 400 in the simulator....

FlowState over 6 years ago

And I am back to error code 400 in the simulator....

As Travis explained, I believe this is because:
1) The worldtides.info API is returning "text/json" instead of the proper "application/json" content-type
text/json is wrong, although I can sorta see why some sites would use it (although there's no reason to.)
2) The simulator is using the Wi-Fi code path which has stricter checking for the content type

If you turn off Wi-Fi network requests (as Travis mentioned), it should work.

I think the proper solution for this is one of the following:
- Ask worldtides.info to fix their API
- Write a proxy service to return the worldtides info, but with the correct content type

jim_m_58 over 6 years ago

If I recall, HermoT was getting a -400 back from with iOS when he had parameters as part of the URL and not as the separate params to the call. He uses worldtides in his TidalFace watch face, so he'd also know if something changed on that end.

Turning off Wifi in the sim may help narrow down the problem, but in the longer term might not be the best, as with 3.0.8, you also have a new option for LTE on the va3 Music LTE, and if the BLE is disabled, it will use LTE, and I'd expect LTE to be connected much of the time on that watch.

HermoT over 6 years ago

agsurf5 You are welcome to send me your code snippet through PM. I have my own key, so make sure to remove your API key before sending me the code.
I have been using worldtides for more than a year in TidalFace with iOS as my main development and testing tool, so I might be able to suggest changes to your code that will work.

I have a proxy, so I am already catering for the content type issue. If there is nothing wrong in your code, the issue might probably only be solvable with a proxy.