Just delete the *.prg files in the <usb drive>\GARMIN\APPS folder.
Also make sure you delete the corresponding *.SET and *.STR files in the SETIINGS and DATA folders.
If using Connect and the app you are trying to delete is a watch face, make sure the watch face is not actively displaying while trying to delete, otherwise you will get a communication error.