GCM Android failing to decode GZIP-encoded packets.

I haven't checked the headers sent out on iPhone, but I'm fairly certain that the system (both Garmin Connect Mobile on the mobile and the ConnectIQ framework on the device) expect to receive uncompressed data.

You should have been able to use the Accept header to force the server to send back data that the framework can handle. I can see that you are sending an Accept header, and that should have been enough. Unfortunately, it looks like that header isn't being sent through from Garmin Connect Mobile. It seems to me that is the actual bug.

That said, you might want to try to force the server (assuming you can't control the server settings) to use no compression by sending the additional Content-Encoding header...

var headers = {
"Content-Type" => Comm.REQUEST_CONTENT_TYPE_URL_ENCODED,
"Accept" => "application/json",
"Accept-Encoding" => "identity"
};


Note that you most likely do not need to use wireshark or packet capture. You should be able to setup a simple proxy (like tcptunnel) to do what you need. You'd have to update your code to connect to the address of the machine running the tunnel and the phone and pc would need to be on the same network, but it should be sufficient.

Travis

Thanks for that, but curiously, adding "Accept-Encoding" => "identity" not only didn't fix the problem, it introduced another:
I now get 0 from the makeWebRequest callback on Android.
(it still works on Sim)
GET /api/gpx.php?pebbleGuid=13a9bf459c69b278bf5a7afc348f1b7d? HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: identity
User-Agent: Mozilla/5.0 ( compatible )
Accept: */*
Host: raceqs.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Mon, 30 Jan 2017 01:01:37 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=df80f9485f690881524f959931b9f76c61485738097; expires=Tue, 30-Jan-18 01:01:37 GMT; path=/; domain=.raceqs.com; HttpOnly
Vary: Accept-Encoding
Expires: -1
Access-Control-Allow-Origin: *
Cache-Control: no-cache
CF-Cache-Status: MISS
Server: cloudflare-nginx
CF-RAY: 3290fe63d7702264-LAX

0


I've tried to reproduce the problem using the information you provided in the google document, but I'm getting an empty response...

C:\Users\Travis\Desktop\connectiq\bin>curl -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -v raceqs.com/.../gpx.php
* Trying 2400:cb00:2048:1::6818:7564...
* Connected to raceqs.com (2400:cb00:2048:1::6818:7564) port 80 (#0)
> GET /api/gpx.php?pebbleGuid=13a9bf459c69b278bf5a7afc348f1b7d? HTTP/1.1
> Host: raceqs.com
> User-Agent: curl/7.46.0
> Content-Type: application/x-www-form-urlencoded
> Accept: application/json
>
< HTTP/1.1 200 OK
< Date: Mon, 30 Jan 2017 05:26:20 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d926aa9e67520c35f76fa712d230467a91485753980; expires=Tue, 30-Jan-18 05:26:20 GMT; path=/; domain=.raceqs.com; HttpOnly
< Vary: Accept-Encoding
< Expires: -1
< Access-Control-Allow-Origin: *
< Cache-Control: no-cache
< CF-Cache-Status: MISS
< Server: cloudflare-nginx
< CF-RAY: 3292822925c529dd-SEA
<
* Connection #0 to host raceqs.com left intact


This appears similar to what you're seeing now, so I wonder if something is up with the service... or maybe the GUID I've lifted is invalid. I believe that ConnectIQ doesn't like it if there is no response body, so this could be the problem.

Regardless, this seems like a bug (the fact that the Accept header is being tossed out). I don't personally think it is a bug that GCM doesn't accept encoded response data, but it could easily be considered an enhancement request.

If you are able, you should post something in the Bug Reports forum. If you do so, be sure to provide a complete working test case for the Garmin guys... it makes it a lot easier to reproduce the problem on their end.

Travis

Actually, I made a mistake in my test, and that points out a problem with your code. I'm not sure if this has an effect or not, but I'll throw it out there. You are encoding the parameters in the url you're providing to the makeWebRequest() method. You should not do this. You should put the parameters into the params map that you pass to the call.

If I properly build the request with curl I get a valid response.

C:\Users\Travis\Desktop\connectiq\bin>curl -G -d "pebbleGuid=13a9bf459c69b278bf5a7afc348f1b7d" -H "Accept: application/json" -v http://raceqs.com/api/gpx.php
* Trying 104.24.117.100...
* Connected to raceqs.com (104.24.117.100) port 80 (#0)
> GET /api/gpx.php?pebbleGuid=13a9bf459c69b278bf5a7afc348f1b7d HTTP/1.1
> Host: raceqs.com
> User-Agent: curl/7.46.0
> Accept: application/json
>
< HTTP/1.1 200 OK
< Date: Mon, 30 Jan 2017 05:54:30 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=dcbbef542beb0bcabf25727422f538f271485755670; expires=Tue, 30-Jan-18 05:54:30 GMT; path=/; domain=.raceqs.com; HttpOnly
< Vary: Accept-Encoding
< Expires: -1
< Access-Control-Allow-Origin: *
< Cache-Control: no-cache
< CF-Cache-Status: MISS
< Server: cloudflare-nginx
< CF-RAY: 3292ab6c113f1ba3-SEA
<
[{"name":"SOPStart","lat":-33.57125183,"lon":151.32454367},{"name":"04-70","lat":-33.56217321,"lon":151.41855438},{"name":"04-90","lat":-33.58694846,"lon":151.41845128},{"name":"08-70","lat":-33.55237732
,"lon":151.49815999},{"name":"1PT","lat":-33.49333333,"lon":151.48666667},{"name":"2PT","lat":-33.51333333,"lon":151.46033333},{"name":"BJ","lat":-33.57333333,"lon":151.34000000},{"name":"LR","lat":-33.7
5000000,"lon":151.36666667},{"name":"TGL","lat":-33.44333333,"lon":151.50000000},{"name":"THRD PT","lat":-33.52458718,"lon":151.42002339},{"name":"SOPS FINISH","lat":-33.58773429,"lon":151.30869221}]* Co
nnection #0 to host raceqs.com left intact

C:\Users\Travis\Desktop\connectiq\bin>


I believe the MonkeyC code to make that same request is as follows...

function loadMarks() {
var url="raceqs.com/.../gpx.php";

var params = {
// Specify the request parameters here. This fixes a problem where the request
// you are sending was getting an extra '?' appended to the end, which may have
// caused some problems.
"pebbleGuid" => "13a9bf459c69b278bf5a7afc348f1b7d"
};

var headers = {
// The docs indicate you don't need this header if you really want a get request
// as the request parameters can only be added to the url.
// "Content-Type" => Comm.REQUEST_CONTENT_TYPE_URL_ENCODED,

// This header should not be necessary, but could possibly be necessary for
// GCM on Android.
// "Accept-Encoding" => "identity;q=1, *;q=0",

"Accept" => "application/json"
};

var options = {
:headers => headers,
:method => Comm.HTTP_REQUEST_METHOD_GET,
:responseType => Comm.HTTP_RESPONSE_CONTENT_TYPE_JSON
};

Comm.makeWebRequest(url, params, options, method(:receiveMarks));
}


I'll try the code in a minute...

Travis

I've tested the above code (by pasting it into the testcase here) on the simulator on on a fr920xt connected to an iPhone and it worked correctly. Of course that won't help you if there is a bug in Garmin Connect Mobile on Android, but it is the best I can do.

Travis

Thanks Travis, you've given me lots to think about. Unfortunately I can't test for the moment as the admin at raceqs.com is tweaking the gpx.php code which is now broken. And there's a 9 hr time zone difference! (He's in eastern Europe and I'm in Sydney.) I'll get on to it as soon as I get it back.

I wonder if that trailing "?" on the pebbleGuid is more significant than we think? I didn't spot it earlier. It definitely wasn't in my MC code but it reliably shows up in the packet capture when I run the code on Android GCM.
It doesn't show up when I run the code in the Sim.
Late post:
When I create the parameters using, as you suggested:
var params = {
"pebbleGuid" => "13a9bf459c69b278bf5a7afc348f1b7d"
};
the trailing "?" disappears. So it's being added by GCM Android! So, the message is: Don't attach parameters to the url!


But so many questions remain...
Why can't I add parameters to the URL on a GET? It's a perfectly normal thing to do.
Why doesn't GCM accept encoded data? Is it a new issue?
How many more incompatibilities are there between GCM Android, GCM iPhone and the emulator?
Am I too close to the envelope, expecting this to be plain sailing?

So, I've run my test case against Garmin Connect Mobile on iOS, and I see this in the headers...

GET /api/gpx.php?scenario=0
Host: 192.168.2.144:8080
Accept: application/json
Accept-Language: en-us
Connection: keep-alive
Accept-Encoding: gzip, deflate
User-Agent: ConnectMobile/2 CFNetwork/808.2.16 Darwin/16.3.0


I think this is pretty good evidence that gzip and deflate encodings are supported. I would be very surprised that the Accept header would be getting sent if the platform didn't support those encodings.

My test case generates requests in several different ways trying to reproduce the problem with the question mark getting appended to the url. I'm not able to reproduce the problem at all. Unfortunately, I don't have access to an Android device to test with, so I leave this to you if you're interested in taking it that far.

Here is the source for my python test program.

import BaseHTTPServer
import json

class MockRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

def init(self):
pass

def do_GET(self):
print self.command, self.path
print self.headers

message_body = json.dumps({ "result" : "success" })

self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", len(message_body))
self.end_headers()
self.wfile.write(message_body)


def _main():
server_address = ('', 8080)

httpd = BaseHTTPServer.HTTPServer(server_address, MockRequestHandler)
httpd.serve_forever()


if __name__=='__main__':
_main()


Here is the source code for my client program.

using Toybox.Application as App;
using Toybox.Lang as Lang;
using Toybox.System as Sys;
using Toybox.WatchUi as Ui;
using Toybox.Graphics as Gfx;
using Toybox.Communications as Comm;

var _G_code;
var _G_data;
var _G_pending = false;


class XBehaviorDelegate extends Ui.BehaviorDelegate
{
function initialize() {
BehaviorDelegate.initialize();
}

const _scenarios = [
{
:url => "http://192.168.2.144:8080/api/gpx.php?scenario=0",
:params => null,
:headers => {
}
},

{
:url => "http://192.168.2.144:8080/api/gpx.php?scenario=1",
:params => {
},
:headers => {
}
},

{
:url => "http://192.168.2.144:8080/api/gpx.php",
:params => {
"scenario" => "2"
},
:headers => {
}
},

{
:url => "http://192.168.2.144:8080/api/gpx.php?scenario=3",
:params => null,
:headers => {
"Content-Type" => Comm.REQUEST_CONTENT_TYPE_URL_ENCODED
}
},

{
:url => "http://192.168.2.144:8080/api/gpx.php?scenario=4",
:params => {
},
:headers => {
"Content-Type" => Comm.REQUEST_CONTENT_TYPE_URL_ENCODED
}
},

{
:url => "http://192.168.2.144:8080/api/gpx.php",
:params => {
"scenario" => "5"
},
:headers => {
"Content-Type" => Comm.REQUEST_CONTENT_TYPE_URL_ENCODED
}
}
];

hidden var _M_mode = 0;

function onSelect() {

if (!_G_pending) {
_G_pending = true;
_G_code = "";

var scenario = _scenarios[_M_mode];
_M_mode = (_M_mode + 1) % _scenarios.size();

var url = scenario[:url];
var params = scenario[:params];
var headers = scenario[:headers];

headers.put("Accept", "application/json");

var options = {
:headers => headers,
:method => Comm.HTTP_REQUEST_METHOD_GET,
:responseType => Comm.HTTP_RESPONSE_CONTENT_TYPE_JSON
};

Comm.makeWebRequest(url, params, options, method(:onWebResponse));

Ui.requestUpdate();
}

return true;
}

function onWebResponse(code, data) {
_G_pending = false;

_G_code = code;
_G_data = data;

Ui.requestUpdate();
}
}

class XView extends Ui.View
{
function initialize() {
View.initialize();
}

function onUpdate(dc) {
dc.setColor(Gfx.COLOR_WHITE, Gfx.COLOR_BLACK);
dc.clear();

var cy = dc.getHeight() / 2;
var cx = dc.getWidth() / 2;

var color = Gfx.COLOR_GREEN;
if (_G_pending) {
color = Gfx.COLOR_RED;
}

dc.setColor(color, color);
dc.fillCircle(cx, 10, 7);
dc.setColor(Gfx.COLOR_WHITE, Gfx.COLOR_BLACK);

if (_G_code != null) {


var fy = dc.getFontHeight(Gfx.FONT_LARGE);

dc.drawText(cx, cy - fy, Gfx.FONT_LARGE, _G_code.toString(), Gfx.TEXT_JUSTIFY_CENTER);

if (_G_data != null) {
dc.drawText(cx, cy, Gfx.FONT_XTINY, _G_data.toString(), Gfx.TEXT_JUSTIFY_CENTER);
}
}
}
}

class XApp extends App.AppBase
{
function initialize() {
AppBase.initialize();
}

function getInitialView() {
return [ new XView(), new XBehaviorDelegate() ];
}
}


In looking at the output you posted previously, I'm wondering if the 0 that appears at the end of the trace output was the response body. If it is, that is most likely why you were getting an error response code. If the response can't be interpreted as a JSON object, you'll get an error.

Travis

Just briefly, we're monitoring this thread and are actively investigating this problem. I don't have any answers just yet, but I didn't want o give the impression this is being ignored. Travis, I appreciate the work you've been doing here to isolate a potential cause, too.

We found a few issues in the 3.15 GCM/A release that may be related to this, and we've got someone looking more specifically at this Accept header/GZIP issue today. We'll be validating the fixes that have been made so far this afternoon, and they should make it into the next GCM release. I'm not sure whether an Accept header fix (assuming it a bug) will make it in the next release or not at this point, but I want to provide some assurance that we will have it addressed.