JavaScript for impatient programmers (beta)
Please support this book: buy it or donate
(Ad, please don’t block.)

30. Typed Arrays: handling binary data (Advanced)



30.1. The basics of the API

Much data on the web is text: JSON files, HTML files, CSS files, JavaScript code, etc. JavaScript handles such data well, via its built-in strings.

However, before 2011, it did not handle binary data well. The Typed Array Specification 1.0 was introduced on 8 February 2011 and provides tools for working with binary data. With ECMAScript 6, Typed Arrays were added to the core language and gained methods that were previously only available for normal Arrays (.map(), .filter(), etc.).

30.1.1. Use cases for Typed Arrays

The main uses cases for Typed Arrays are:

30.1.2. Browser APIs that support Typed Arrays

The following browser APIs support Typed Arrays:

30.1.3. The core classes: ArrayBuffer, Typed Arrays, DataView

The Typed Array API stores binary data in instances of ArrayBuffer:

const buf = new ArrayBuffer(4); // length in bytes
  // buf is initialized with zeros

An ArrayBuffer itself is opaque. If you want to access its data, you must wrap it in another object – a view object. Two kinds of view objects are available:

Fig. 28 shows a class diagram of the API.

Figure 28: The classes of the Typed Array API.
Figure 28: The classes of the Typed Array API.

30.1.4. Using Typed Arrays

Typed Arrays are used much like normal Arrays, with a few notable differences:

This is an example of using a Typed Array:

const typedArray = new Uint8Array(2); // 2 elements
assert.equal(typedArray.length, 2);

// The wrapped ArrayBuffer
assert.deepEqual(
  typedArray.buffer, new ArrayBuffer(2)); // 2 bytes

// Getting and setting elements:
assert.equal(typedArray[1], 0); // initialized with 0
typedArray[1] = 72;
assert.equal(typedArray[1], 72);

Other ways of creating Typed Arrays:

const ta1 = new Uint8Array([5, 6]);
const ta2 = Uint8Array.of(5, 6);
assert.deepEqual(ta1, ta2);

30.1.5. Using DataViews

This is how DataViews are used:

const dataView = new DataView(new ArrayBuffer(4));
assert.equal(dataView.getUint16(0), 0);
assert.equal(dataView.getUint8(0), 0);
dataView.setUint8(0, 5);

30.2. Foundations of the Typed Array API

30.2.1. Element types

The following element types are supported by the API:

Element type Bytes Description C type
Int8 1 8-bit signed integer signed char
Uint8 1 8-bit unsigned integer unsigned char
Uint8C 1 8-bit unsigned integer (clamped conversion) unsigned char
Int16 2 16-bit signed integer short
Uint16 2 16-bit unsigned integer unsigned short
Int32 4 32-bit signed integer int
Uint32 4 32-bit unsigned integer unsigned int
Float32 4 32-bit floating point float
Float64 8 64-bit floating point double

The element type Uint8C is special: it is not supported by DataView and only exists to enable Uint8ClampedArray. This Typed Array is used by the canvas element (where it replaces CanvasPixelArray). The only difference between Uint8C and Uint8 is how overflow and underflow are handled (as explained in the next section). It is recommended to avoid the former – quoting Brendan Eich:

Just to be super-clear (and I was around when it was born), Uint8ClampedArray is totally a historical artifact (of the HTML5 canvas element). Avoid unless you really are doing canvas-y things.

30.2.2. Handling overflow and underflow

Normally, when a value is out of the range of the element type, modulo arithmetic is used to convert it to a value within range. For signed and unsigned integers that means that:

The following functions helps illustrate how conversion works:

function setAndGet(typedArray, value) {
  typedArray[0] = value;
  return typedArray[0];
}

Modulo conversion for unsigned 8-bit integers:

const uint8 = new Uint8Array(1);

// Highest value of range
assert.equal(setAndGet(uint8, 255), 255);
// Overflow
assert.equal(setAndGet(uint8, 256), 0);

// Lowest value of range
assert.equal(setAndGet(uint8, 0), 0);
// Underflow
assert.equal(setAndGet(uint8, -1), 255);

Modulo conversion for signed 8-bit integers:

const int8 = new Int8Array(1);

// Highest value of range
assert.equal(setAndGet(int8, 127), 127);
// Overflow
assert.equal(setAndGet(int8, 128), -128);

// Lowest value of range
assert.equal(setAndGet(int8, -128), -128);
// Underflow
assert.equal(setAndGet(int8, -129), 127);

Clamped conversion is different:

const uint8c = new Uint8ClampedArray(1);

// Highest value of range
assert.equal(setAndGet(uint8c, 255), 255);
// Overflow
assert.equal(setAndGet(uint8c, 256), 255);

// Lowest value of range
assert.equal(setAndGet(uint8c, 0), 0);
// Underflow
assert.equal(setAndGet(uint8c, -1), 0);

30.2.3. Endianness

Whenever a type (such as Uint16) is stored as a sequence of multiple bytes, endianness matters:

Endianness tends to be fixed per CPU architecture and consistent across native APIs. Typed Arrays are used to communicate with those APIs, which is why their endianness follows the endianness of the platform and can’t be changed.

On the other hand, the endianness of protocols and binary files varies and is fixed across platforms. Therefore, we must be able to access data with either endianness. DataViews serve this use case and let you specify endianness when you get or set a value.

Quoting Wikipedia on Endianness:

You can use the following function to determine the endianness of a platform.

const BIG_ENDIAN = Symbol('BIG_ENDIAN');
const LITTLE_ENDIAN = Symbol('LITTLE_ENDIAN');
function getPlatformEndianness() {
  const arr32 = Uint32Array.of(0x87654321);
  const arr8 = new Uint8Array(arr32.buffer);
  if (compare(arr8, [0x87, 0x65, 0x43, 0x21])) {
    return BIG_ENDIAN;
  } else if (compare(arr8, [0x21, 0x43, 0x65, 0x87])) {
    return LITTLE_ENDIAN;
  } else {
    throw new Error('Unknown endianness');
  }
}
function compare(arr1, arr2) {
  if (arr1.length !== arr2.length) {
    return false;
  }
  for (let i=0; i<arr1.length; i++) {
    if (arr1[i] !== arr2[i]) {
      return false;
    }
  }
  return true;
}

Other orderings are also possible. Those are generically called middle-endian or mixed-endian.

30.2.4. Indices and offsets

For Typed Arrays, we distinguish:

30.3. ArrayBuffers

ArrayBuffers store binary data, which is meant to be accessed via Typed Arrays and DataViews.

30.3.1. new ArrayBuffer()

The type signature of the constructor is:

new ArrayBuffer(length: number)

Invoking this constructor via new creates an instance whose capacity is length bytes. Each of those bytes is initially 0.

You can’t change the length of an ArrayBuffer, you can only create a new one with a different length.

30.3.2. Static methods of ArrayBuffer

30.3.3. Properties of ArrayBuffer.prototype

30.4. Typed Arrays

The various kinds of Typed Arrays are only different w.r.t. the types of their elements:

30.4.1. Typed Arrays versus normal Arrays

Typed Arrays are much like normal Arrays: they have a .length, elements can be accessed via the bracket operator [ ] and they have most of the standard Array methods. They differ from normal Arrays in the following ways:

30.4.2. Typed Arrays are iterable

Typed Arrays are iterable. That means that you can use the for-of loop and similar mechanisms:

const ui8 = Uint8Array.of(0,1,2);
for (const byte of ui8) {
  console.log(byte);
}
// Output:
// 0
// 1
// 2

ArrayBuffers and DataViews are not iterable.

30.4.3. Converting Typed Arrays to and from normal Arrays

To convert a normal Array to a Typed Array, you pass it to a Typed Array constructor. For example:

const tarr = new Uint8Array([0,1,2]);

To convert a Typed Array to a normal Array, you can use spreading or Array.from() (because Typed Arrays are iterable):

assert.deepEqual([...tarr], [0,1,2]);
assert.deepEqual(Array.from(tarr), [0,1,2]);

30.4.4. The class hierarchy of Typed Arrays

The properties of the various Typed Array objects are introduced in two steps:

  1. TypedArray: First, we look at the common superclass of all Typed Array classes (which was shown in the class diagram at the beginning of this chapter). I’m calling that superclass TypedArray, but it is not directly accessible from JavaScript. TypedArray.prototype houses all methods of Typed Arrays.
  2. «ElementType»Array: The actual Typed Array classes are called Uint8Array, Int16Array, Float32Array, etc.

30.4.5. Static methods of TypedArray

Both static TypedArray methods are inherited by its subclasses (Uint8Array etc.).

30.4.5.1. TypedArray.of()

This method has the type signature:

.of(...items: number[]): instanceof this

The notation of the return type is my invention: .of() returns an instance of this (the class on which of() was invoked). The elements of the instance are the arguments of of().

You can think of of() as a custom literal for Typed Arrays:

const float32Arr = Float32Array.of(0.151, -8, 3.7);
const int16Arr = Int32Array.of(-10, 5, 7);
30.4.5.2. TypedArray.from()

This method has the type signature:

TypedArray<T>.from<S>(
  source: Iterable<S>|ArrayLike<S>, mapfn?: S => T, thisArg?: any)
  : instanceof this

It converts source into an instance of this (a Typed Array). Once again, the syntax instanceof this is my invention.

For example, normal Arrays are iterable and can be converted with this method:

assert.deepEqual(
  Uint16Array.from([0, 1, 2]),
  Uint16Array.of(0, 1, 2));

Typed Arrays are also iterable:

assert.deepEqual(
  Uint16Array.from(Uint8Array.of(0, 1, 2)),
  Uint16Array.of(0, 1, 2));

The source can also be an Array-like object:

assert.deepEqual(
  Uint16Array.from({0:0, 1:1, 2:2, length: 3}),
  Uint16Array.of(0, 1, 2));

The optional mapfn lets you transform the elements of source before they become elements of the result. Why perform the two steps mapping and conversion in one go? Compared to mapping separately via .map(), there are two advantages:

  1. No intermediate Array or Typed Array is needed.
  2. When converting between Typed Arrays with different precisions, less can go wrong.

To illustrate the second advantage, let’s first convert a Typed Array to a Typed Array with a higher precision. If we use .from() to map, the result is automatically correct. Otherwise, you must first convert and then map.

const typedArray = Int8Array.of(127, 126, 125);
assert.deepEqual(
  Int16Array.from(typedArray, x => x * 2),
  Int16Array.of(254, 252, 250));

assert.deepEqual(
  Int16Array.from(typedArray).map(x => x * 2),
  Int16Array.of(254, 252, 250)); // OK
assert.deepEqual(
  Int16Array.from(typedArray.map(x => x * 2)),
  Int16Array.of(-2, -4, -6)); // wrong

If we go from a Typed Array to a Typed Array with a lower precision, mapping via .from() produces the correct result. Otherwise, we must first map and then convert.

assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250), x => x / 2),
  Int8Array.of(127, 126, 125));

assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250).map(x => x / 2)),
  Int8Array.of(127, 126, 125)); // OK
assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250)).map(x => x / 2),
  Int8Array.of(-1, -2, -3)); // wrong

The problem is that, if we map via .map(), then input type and output type are the same (if we work with Typed Arrays). In contrast, .from() goes from an arbitrary input type to an output type that you specify via its receiver.

According to Allen Wirfs-Brock, mapping between Typed Arrays was what motivated the mapfn parameter of .from().

30.4.6. Properties of TypedArray<T>.prototype

Indices accepted by Typed Array methods can be negative (they work like traditional Array methods that way). Offsets must be non-negative. For details, see Sect. “Indices and offsets”.

30.4.6.1. Properties specific to Typed Arrays

The following properties are specific to Typed Arrays; normal Arrays don’t have them:

30.4.6.2. Array methods

The following methods are basically the same as the methods of normal Arrays:

For details on how these methods work, please consult the chapter on normal Arrays.

30.4.7. new «ElementType»Array()

Each Typed Array constructor has a name that follows the pattern «ElementType»Array, where «ElementType» is one of the element types in the table at the beginning. That means that there are 9 constructors for Typed Arrays: Int8Array, Uint8Array, Uint8ClampedArray (element type Uint8C), Int16Array, Uint16Array, Int32Array, Uint32Array, Float32Array, Float64Array.

Each constructor has four overloaded versions – it behaves differently depending on how many arguments it receives and what their types are:

The following code shows three different ways of creating the same Typed Array:

const ta1 = new Uint8Array([0, 1, 2]);

const ta2 = Uint8Array.of(0, 1, 2);

const ta3 = new Uint8Array(3);
ta3[0] = 0;
ta3[1] = 1;
ta3[2] = 2;

assert.deepEqual(ta1, ta2);
assert.deepEqual(ta1, ta3);

30.4.8. Static properties of «ElementType»Array

30.4.9. Properties of «ElementType»Array.prototype

30.4.10. Concatenating Typed Arrays

Typed Arrays don’t have a method .concat(), like normal Arrays do. The work-around is to use the method

.set(typedArray: TypedArray, offset=0): void

That method copies an existing Typed Array into typedArray at index offset. Then you only have to make sure that typedArray is big enough to hold all (Typed) Arrays you want to concatenate:

function concatenate(resultConstructor, ...arrays) {
  let totalLength = 0;
  for (const arr of arrays) {
    totalLength += arr.length;
  }
  const result = new resultConstructor(totalLength);
  let offset = 0;
  for (const arr of arrays) {
    result.set(arr, offset);
    offset += arr.length;
  }
  return result;
}
assert.deepEqual(
  concatenate(Uint8Array,
    Uint8Array.of(1, 2), [3, 4]),
    Uint8Array.of(1, 2, 3, 4)
);

30.5. DataViews

30.5.1. new DataView()

30.5.2. Properties of DataView.prototype

«ElementType» can be: Float32, Float64, Int8, Int16, Int32, Uint8, Uint16, Uint32.

30.6. Further reading

The chapter on Typed Arrays in “Exploring ES6” has some additional content: