Acknowledged

ByteArray addAll() not Memory Efficient

When using the addAll() method of Lang.ByteArray, peak memory usage implies that the ConnectIQ framework is doubling the size of the array being appended to instead of extending it by the size of the appended array.

For example, let's say we create a 1K byteArray called myArray and run the following code starting with an empty byte array, longByteArray:

while(longByteArray.size() < 32768) {
    longByteArray.addAll(myArray);
}

An efficient implementation would grow peak memory usage by ~1K more than longByteArray.size(). Instead, we see peak memory usage growing with longByteArray.size() * 2 + 1024.

It may also be worth noting that the Monkey C extension for Visual Studio code will time out as the array grows if a developer sets a breakpoint on the addAll function above while the watch window is open to longByteArray.

Sforzando
Parents

  • It's not allocating temporary memory:

    this is the memory before you all addAll:

    ---b----n----[longByteArray]---[myArray]---x---y------------------------------------------------------------------------------------------------------ 

    and this is just after it copied the last byte of myArray for the 1st time:

    ---b----n----[longByteArray]---[myArray]---x---y-[longByteArraymyArray]---------------------------------------------------------------------- 

    this is when it freed the old longByteArray:

    ---b----n----------------------------[myArray]---x---y-[longByteArraymyArray]---------------------------------------------------------------------- 

    this is after it added myArray for the 2nd time:

    ---b----n----------------------------[myArray]---x---y-[longByteArraymyArray]-[longByteArraymyArraymyArray]--------------------------

    etc...

    at the end of the last time it adds myArray (depending on how many times it freed the old longByteArray and whether there's enough continous space "before" or "after" the actual "old" longByteArray) it could be something like:

    -b-n----------[myArray]-x-y-[longByteArraymyArraymyArraymyArraymyArray]---------------[longByteArraymyArraymyArraymyArray]-----

    or worse:

    -b-n----------[myArray]-x-y----------------------------------------------------------[longByteArraymyArraymyArraymyArray]-[longByteArraymyArraymyArraymyArraymyArray]

    But from both of these you can see that the minimum amount of memory it needs is:

    1* myArray (the original)

    1* original longByteArray + (n-1) * myArray (the one it adds the last time myArray to)

    1* original longByteArray + (n) * myArray (the result)

    SUM: 2 * originalLongByteArray + (1 + (n - 1) + n) * myArray = 2 * originalLongByteArray + 2n * myArray

    so if the origial longArray was empty then this is exactly what you saw

Comment

  • It's not allocating temporary memory:

    this is the memory before you all addAll:

    ---b----n----[longByteArray]---[myArray]---x---y------------------------------------------------------------------------------------------------------ 

    and this is just after it copied the last byte of myArray for the 1st time:

    ---b----n----[longByteArray]---[myArray]---x---y-[longByteArraymyArray]---------------------------------------------------------------------- 

    this is when it freed the old longByteArray:

    ---b----n----------------------------[myArray]---x---y-[longByteArraymyArray]---------------------------------------------------------------------- 

    this is after it added myArray for the 2nd time:

    ---b----n----------------------------[myArray]---x---y-[longByteArraymyArray]-[longByteArraymyArraymyArray]--------------------------

    etc...

    at the end of the last time it adds myArray (depending on how many times it freed the old longByteArray and whether there's enough continous space "before" or "after" the actual "old" longByteArray) it could be something like:

    -b-n----------[myArray]-x-y-[longByteArraymyArraymyArraymyArraymyArray]---------------[longByteArraymyArraymyArraymyArray]-----

    or worse:

    -b-n----------[myArray]-x-y----------------------------------------------------------[longByteArraymyArraymyArraymyArray]-[longByteArraymyArraymyArraymyArraymyArray]

    But from both of these you can see that the minimum amount of memory it needs is:

    1* myArray (the original)

    1* original longByteArray + (n-1) * myArray (the one it adds the last time myArray to)

    1* original longByteArray + (n) * myArray (the result)

    SUM: 2 * originalLongByteArray + (1 + (n - 1) + n) * myArray = 2 * originalLongByteArray + 2n * myArray

    so if the origial longArray was empty then this is exactly what you saw

Children
  • in reply to flocsy🤠

    Let's assume the original longArray were preallocated using:

    var longArray =new [32768]b;

    Why could there not be a native CIQ function that takes a 1K myArray and adds it to longArray 32 times without using more than 33K? Basically, it could do in native code what this loop does if the watchdog didn't time out:

    for(var j = 0; j < 32; j++) {
    for(var i = 0; i < 1024; i++) {
        longByteArray[(j << 10) + i] = myArray[i];
    }
}

    The above loop is memory efficient but not time efficient, but CIQ native code could improve this.