Share real device testing

Former Member over 9 years ago

It's clear that the simulator does not represent the true behaviour of any devices. It's great for getting going, but testing on the actual device is essential to ensure the app works correctly.

The problem with this is that you either need to buy lots of devices or only support one. The former is not possible for lone developers and the latter is bad for general adoption of the app. It would be a shame to not mark apps as compatible with a device that they would actually work on, if they could only be tested.

I suggest that a crowd testing approach is adopted, whereby we offer the device we have as a testbed for other people's apps. A developer can post a request for testing and provide the compiled app to be side-loaded for testing. The developer also details exactly what needs testing and what feedback they require. The tester will then run the app and give feedback appropriately. In exchange, the tester can make similar requests for their own apps, which will be tested by other users with different hardware to them.

Anyone up for this idea?

jim_m_58 over 9 years ago

What some people do today, is only put stuff in the app store for devices that have been tested, and then when someone wants it for a different device, they provide a .prg for that user that the user can sideload (not get it from the app store) to check it out. That way there is someone that wants the app and is willing to provide feedback, without releasing it to the world.

The topic of a "beta" mode in the app store has also been brought up here, so that if it's just a new version for known devices, or a version for new devices, it can be checked out in a limited environment. With "user settings" in 1.2.x, the settings part of a app can only be checked out for stuff that comes from the app store (at least initially), so I guess "beta mode" in the app store will come up again.

julienvm over 9 years ago

Sounds like a good idea to me!

Brandon.ConnectIQ over 9 years ago

I think having a beta feature on the app store is the ideal solution here (aside from improving the simulator). I know a lot of developers are really after an emulator which is not the intent behind the simulator. There are device emulators that exist internally, but we prefer to test on hardware ourselves and use the sim for Garmin app development. We'll continue to refine the simulator so that we can at least get device behaviors matched as closely as possible.

Our current concept of a beta version allows developers to have a published version A available on the store to the public, and then have a beta version B also published that is only available to the author. I've added a comment requesting the adding a feature to allow the author to selectively assign access to the beta version by developer screen name.

Former Member over 9 years ago

A beta feature would be great. It would be good if beta versions could show in search results and optionally be hidden (by the end user). They could then request access to the beta version from the author. My issue with not publishing for devices for which you haven't tested on the real hardware, is that users may never find the app at all if they only search for apps that support their device (which is the default process). They will then never request support for their device or participate in testing.

I still think we could tie in communal real hardware testing with the beta feature. Obviously commercial developers should just buy all the hardware - it's only the cost of a few hours dev time, so it's a no brainer (do you do a dev kit? you should!). Those of us who only develop for Garmin devices in our spare time, will still be unlikely to buy more than one device.

jim_m_58 over 9 years ago

As far as finding stuff that doesn't support your device, on the front page there is the "new and updated" section that covers stuff for all the devices. Right now, it only shows the newest 8. If that section was given a "more" button, you could scroll through stuff in the order it was added/updated for all devices.

Right now you can do that for a given device as an example (the va has about 25 pages for "more" on new and updated), so on the front page, the same thing but for all devices and not just one.

What might be nice for the beta stuff Brandon mentioned, is a "two level" definition of "beta". Say I have "public 1.0" version and "beta 1.1" version, with the beta version marked "restricted" or "open" during submission to the store. If it's "restricted", the developer has to grant access. If it's "open", anyone can download it, but "reviews" are for the beta version and don't impact the reviews for the public version.

With CIQ 1.2 and user settings, the only way I know of to use "user settings" on a real device is if the app comes from the app store. In the past, people could just email/dropbox/etc .prg files around for sideloading, but with user settings, that might not really work any more.

Former Member over 9 years ago

As far as finding stuff that doesn't support your device, on the front page there is the "new and updated" section that covers stuff for all the devices. Right now, it only shows the newest 8. If that section was given a "more" button, you could scroll through stuff in the order it was added/updated for all devices.

Right now you can do that for a given device as an example (the va has about 25 pages for "more" on new and updated), so on the front page, the same thing but for all devices and not just one.

That's a good idea, but when you enter a search term, the results are separated by device. I think it's pretty certain that most users will only look in the list for their own device, thus missing apps which might be made available to their device if the developer had the opportunity.

What might be nice for the beta stuff Brandon mentioned, is a "two level" definition of "beta". Say I have "public 1.0" version and "beta 1.1" version, with the beta version marked "restricted" or "open" during submission to the store. If it's "restricted", the developer has to grant access. If it's "open", anyone can download it, but "reviews" are for the beta version and don't impact the reviews for the public version.

With CIQ 1.2 and user settings, the only way I know of to use "user settings" on a real device is if the app comes from the app store. In the past, people could just email/dropbox/etc .prg files around for sideloading, but with user settings, that might not really work any more.

Agreed, that would be really good. Personally I'd be in favour of binning reviews from the beta version and requiring developers to provide a link to an issue tracker if they are publishing a beta version.

Nikeow over 9 years ago

I think having a beta feature on the app store is the ideal solution here (aside from improving the simulator). I know a lot of developers are really after an emulator which is not the intent behind the simulator. There are device emulators that exist internally, but we prefer to test on hardware ourselves and use the sim for Garmin app development. We'll continue to refine the simulator so that we can at least get device behaviors matched as closely as possible.

Our current concept of a beta version allows developers to have a published version A available on the store to the public, and then have a beta version B also published that is only available to the author. I've added a comment requesting the adding a feature to allow the author to selectively assign access to the beta version by developer screen name.

The best part of this woud be not needing to plug in and out the watch to the PC and wait etc.