JavaScript for impatient programmers
Please support this book: buy it or donate
(Ad, please don’t block.)

29 Typed Arrays: handling binary data (Advanced)



29.1 The basics of the API

Much data on the web is text: JSON files, HTML files, CSS files, JavaScript code, etc. JavaScript handles such data well via its built-in strings.

However, before 2011, it did not handle binary data well. The Typed Array Specification 1.0 was introduced on February 8, 2011 and provides tools for working with binary data. With ECMAScript 6, Typed Arrays were added to the core language and gained methods that were previously only available for normal Arrays (.map(), .filter(), etc.).

29.1.1 Use cases for Typed Arrays

The main uses cases for Typed Arrays, are:

29.1.2 The core classes: ArrayBuffer, Typed Arrays, DataView

The Typed Array API stores binary data in instances of ArrayBuffer:

const buf = new ArrayBuffer(4); // length in bytes
  // buf is initialized with zeros

An ArrayBuffer itself is a black box: if you want to access its data, you must wrap it in another object – a view object. Two kinds of view objects are available:

Fig. 20 shows a class diagram of the API.

Figure 20: The classes of the Typed Array API.
Figure 20: The classes of the Typed Array API.

29.1.3 Using Typed Arrays

Typed Arrays are used much like normal Arrays with a few notable differences:

29.1.3.1 Creating Typed Arrays

The following code shows three different ways of creating the same Typed Array:

// Argument: Typed Array or Array-like object
const ta1 = new Uint8Array([0, 1, 2]);

const ta2 = Uint8Array.of(0, 1, 2);

const ta3 = new Uint8Array(3); // length of Typed Array
ta3[0] = 0;
ta3[1] = 1;
ta3[2] = 2;

assert.deepEqual(ta1, ta2);
assert.deepEqual(ta1, ta3);
29.1.3.2 The wrapped ArrayBuffer
const typedArray = new Int16Array(2); // 2 elements
assert.equal(typedArray.length, 2);

assert.deepEqual(
  typedArray.buffer, new ArrayBuffer(4)); // 4 bytes
29.1.3.3 Getting and setting elements
const typedArray = new Int16Array(2);

assert.equal(typedArray[1], 0); // initialized with 0
typedArray[1] = 72;
assert.equal(typedArray[1], 72);

29.1.4 Using DataViews

This is how DataViews are used:

const dataView = new DataView(new ArrayBuffer(4));
assert.equal(dataView.getInt16(0), 0);
assert.equal(dataView.getUint8(0), 0);
dataView.setUint8(0, 5);

29.2 Element types

Table 19: Element types supported by the Typed Array API.
Element Typed Array Bytes Description
Int8 Int8Array 1 8-bit signed integer ES6
Uint8 Uint8Array 1 8-bit unsigned integer ES6
Uint8C Uint8ClampedArray 1 8-bit unsigned integer ES6
(clamped conversion) ES6
Int16 Int16Array 2 16-bit signed integer ES6
Uint16 Uint16Array 2 16-bit unsigned integer ES6
Int32 Int32Array 4 32-bit signed integer ES6
Uint32 Uint32Array 4 32-bit unsigned integer ES6
Float32 Float32Array 4 32-bit floating point ES6
Float64 Float64Array 8 64-bit floating point ES6

Tbl. 19 lists the available element types. These types (e.g., Int32) show up in two locations:

The element type Uint8C is special: it is not supported by DataView and only exists to enable Uint8ClampedArray. This Typed Array is used by the canvas element (where it replaces CanvasPixelArray) and should otherwise be avoided. The only difference between Uint8C and Uint8 is how overflow and underflow are handled (as explained in the next subsection).

29.2.1 Handling overflow and underflow

Normally, when a value is out of the range of the element type, modulo arithmetic is used to convert it to a value within range. For signed and unsigned integers that means that:

The following function helps illustrate how conversion works:

function setAndGet(typedArray, value) {
  typedArray[0] = value;
  return typedArray[0];
}

Modulo conversion for unsigned 8-bit integers:

const uint8 = new Uint8Array(1);

// Highest value of range
assert.equal(setAndGet(uint8, 255), 255);
// Overflow
assert.equal(setAndGet(uint8, 256), 0);

// Lowest value of range
assert.equal(setAndGet(uint8, 0), 0);
// Underflow
assert.equal(setAndGet(uint8, -1), 255);

Modulo conversion for signed 8-bit integers:

const int8 = new Int8Array(1);

// Highest value of range
assert.equal(setAndGet(int8, 127), 127);
// Overflow
assert.equal(setAndGet(int8, 128), -128);

// Lowest value of range
assert.equal(setAndGet(int8, -128), -128);
// Underflow
assert.equal(setAndGet(int8, -129), 127);

Clamped conversion is different:

const uint8c = new Uint8ClampedArray(1);

// Highest value of range
assert.equal(setAndGet(uint8c, 255), 255);
// Overflow
assert.equal(setAndGet(uint8c, 256), 255);

// Lowest value of range
assert.equal(setAndGet(uint8c, 0), 0);
// Underflow
assert.equal(setAndGet(uint8c, -1), 0);

29.2.2 Endianness

Whenever a type (such as Uint16) is stored as a sequence of multiple bytes, endianness matters:

Endianness tends to be fixed per CPU architecture and consistent across native APIs. Typed Arrays are used to communicate with those APIs, which is why their endianness follows the endianness of the platform and can’t be changed.

On the other hand, the endianness of protocols and binary files varies, but is fixed per format, across platforms. Therefore, we must be able to access data with either endianness. DataViews serve this use case and let you specify endianness when you get or set a value.

Quoting Wikipedia on Endianness:

Other orderings are also possible. Those are generically called middle-endian or mixed-endian.

29.3 More information on Typed Arrays

In this section, «ElementType»Array stands for Int8Array, Uint8Array, etc. ElementType is Int8, Uint8, etc.

29.3.1 The static method «ElementType»Array.from()

This method has the type signature:

.from<S>(
  source: Iterable<S>|ArrayLike<S>,
  mapfn?: S => ElementType, thisArg?: any)
  : «ElementType»Array

.from() converts source into an instance of this (a Typed Array).

For example, normal Arrays are iterable and can be converted with this method:

assert.deepEqual(
  Uint16Array.from([0, 1, 2]),
  Uint16Array.of(0, 1, 2));

Typed Arrays are also iterable:

assert.deepEqual(
  Uint16Array.from(Uint8Array.of(0, 1, 2)),
  Uint16Array.of(0, 1, 2));

source can also be an Array-like object:

assert.deepEqual(
  Uint16Array.from({0:0, 1:1, 2:2, length: 3}),
  Uint16Array.of(0, 1, 2));

The optional mapfn lets you transform the elements of source before they become elements of the result. Why perform the two steps mapping and conversion in one go? Compared to mapping separately via .map(), there are two advantages:

  1. No intermediate Array or Typed Array is needed.
  2. When converting between Typed Arrays with different precisions, less can go wrong.

Read on for an explanation of the second advantage.

29.3.1.1 Pitfall: mapping while converting between Typed Array types

The static method .from() can optionally both map and convert between Typed Array types. Less can go wrong if you use that method.

To see why that is, let us first convert a Typed Array to a Typed Array with a higher precision. If we use .from() to map, the result is automatically correct. Otherwise, you must first convert and then map.

const typedArray = Int8Array.of(127, 126, 125);
assert.deepEqual(
  Int16Array.from(typedArray, x => x * 2),
  Int16Array.of(254, 252, 250));

assert.deepEqual(
  Int16Array.from(typedArray).map(x => x * 2),
  Int16Array.of(254, 252, 250)); // OK
assert.deepEqual(
  Int16Array.from(typedArray.map(x => x * 2)),
  Int16Array.of(-2, -4, -6)); // wrong

If we go from a Typed Array to a Typed Array with a lower precision, mapping via .from() produces the correct result. Otherwise, we must first map and then convert.

assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250), x => x / 2),
  Int8Array.of(127, 126, 125));

assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250).map(x => x / 2)),
  Int8Array.of(127, 126, 125)); // OK
assert.deepEqual(
  Int8Array.from(Int16Array.of(254, 252, 250)).map(x => x / 2),
  Int8Array.of(-1, -2, -3)); // wrong

The problem is that if we map via .map(), then input type and output type are the same. In contrast, .from() goes from an arbitrary input type to an output type that you specify via its receiver.

29.3.2 Typed Arrays are iterable

Typed Arrays are iterable. That means that you can use the for-of loop and other iteration-based mechanisms:

const ui8 = Uint8Array.of(0, 1, 2);
for (const byte of ui8) {
  console.log(byte);
}
// Output:
// 0
// 1
// 2

ArrayBuffers and DataViews are not iterable.

29.3.3 Typed Arrays vs. normal Arrays

Typed Arrays are much like normal Arrays: they have a .length, elements can be accessed via the bracket operator [], and they have most of the standard Array methods. They differ from normal Arrays in the following ways:

29.3.4 Converting Typed Arrays to and from normal Arrays

To convert a normal Array to a Typed Array, you pass it to a Typed Array constructor (which accepts Array-like objects and Typed Arrays) or to «ElementType»Array.from() (which accepts iterables and Array-like objects). For example:

const ta1 = new Uint8Array([0, 1, 2]);
const ta2 = Uint8Array.from([0, 1, 2]);
assert.deepEqual(ta1, ta2);

To convert a Typed Array to a normal Array, you can use spreading or Array.from() (because Typed Arrays are iterable):

assert.deepEqual(
  [...Uint8Array.of(0, 1, 2)], [0, 1, 2] );
assert.deepEqual(
  Array.from(Uint8Array.of(0, 1, 2)), [0, 1, 2] );

29.3.5 Concatenating Typed Arrays

Typed Arrays don’t have a method .concat(), like normal Arrays do. The workaround is to use their overloaded method .set():

.set(typedArray: TypedArray, offset=0): void
.set(arrayLike: ArrayLike<number>, offset=0): void

It copies the existing typedArray or arrayLike into the receiver, at index offset. TypedArray is a fictitious abstract superclass of all concrete Typed Array classes.

The following function uses that method to copy zero or more Typed Arrays (or Array-like objects) into an instance of resultConstructor:

function concatenate(resultConstructor, ...arrays) {
  let totalLength = 0;
  for (const arr of arrays) {
    totalLength += arr.length;
  }
  const result = new resultConstructor(totalLength);
  let offset = 0;
  for (const arr of arrays) {
    result.set(arr, offset);
    offset += arr.length;
  }
  return result;
}
assert.deepEqual(
  concatenate(Uint8Array, Uint8Array.of(1, 2), [3, 4]),
  Uint8Array.of(1, 2, 3, 4));

29.4 Quick references: indices vs. offsets

In preparation for the quick references on ArrayBuffers, Typed Arrays, and DataViews, we need learn the differences between indices and offsets:

Whether a parameter is an index or an offset can only be determined by looking at documentation; there is no simple rule.

29.5 Quick reference: ArrayBuffers

ArrayBuffers store binary data, which is meant to be accessed via Typed Arrays and DataViews.

29.5.1 new ArrayBuffer()

The type signature of the constructor is:

new ArrayBuffer(length: number)

Invoking this constructor via new creates an instance whose capacity is length bytes. Each of those bytes is initially 0.

You can’t change the length of an ArrayBuffer; you can only create a new one with a different length.

29.5.2 Static methods of ArrayBuffer

29.5.3 Properties of ArrayBuffer.prototype

29.6 Quick reference: Typed Arrays

The properties of the various Typed Array objects are introduced in two steps:

  1. TypedArray: First, we look at the abstract superclass of all Typed Array classes (which was shown in the class diagram at the beginning of this chapter). I’m calling that superclass TypedArray, but it is not directly accessible from JavaScript. TypedArray.prototype houses all methods of Typed Arrays.
  2. «ElementType»Array: The concrete Typed Array classes are called Uint8Array, Int16Array, Float32Array, etc. These are the classes that you use via new, .of, and .from().

29.6.1 Static methods of TypedArray<T>

Both static TypedArray methods are inherited by its subclasses (Uint8Array, etc.). TypedArray is abstract. Therefore, you always use these methods via the subclasses, which are concrete and can have direct instances.

29.6.2 Properties of TypedArray<T>.prototype

Indices accepted by Typed Array methods can be negative (they work like traditional Array methods that way). Offsets must be non-negative. For details, see §29.4 “Quick references: indices vs. offsets”.

29.6.2.1 Properties specific to Typed Arrays

The following properties are specific to Typed Arrays; normal Arrays don’t have them:

29.6.2.2 Array methods

The following methods are basically the same as the methods of normal Arrays:

For details on how these methods work, please consult §28.12.3 “Methods of Array<T>.prototype.

29.6.3 new «ElementType»Array()

Each Typed Array constructor has a name that follows the pattern «ElementType»Array, where «ElementType» is one of the element types in the table at the beginning. That means that there are nine constructors for Typed Arrays:

Each constructor has four overloaded versions – it behaves differently depending on how many arguments it receives and what their types are:

29.6.4 Static properties of «ElementType»Array

29.6.5 Properties of «ElementType»Array.prototype

29.7 Quick reference: DataViews

29.7.1 new DataView()

29.7.2 Properties of DataView.prototype

In the remainder of this section, «ElementType» refers to either:

These are the properties of DataView.prototype: