Shell scripting with Node.js
You can buy the offline version of this book (HTML, PDF, EPUB, MOBI) and support the free online version.
(Ad, please don’t block.)

5 Packages: JavaScript’s units for software distribution

This chapter explains what npm packages are and how they interact with ESM modules.

Required knowledge: I’m assuming that you are loosely familiar with the syntax of ECMAScript modules. If you are not, you can read chapter “modules” in “JavaScript for impatient programmers”.

5.1 What is a package?

In the JavaScripte ecosystem, a package is a way of organizing software projects: It is a directory with a standardized layout. A package can contain all kinds of files - for example:

A package can depend on other packages (which are called its dependencies) which contain:

The dependencies of a package are installed inside that package (we’ll see how soon).

One common distinction between packages is:

The next subsection explains how packages can be published.

5.1.1 Publishing packages: package registries, package managers, package names

The main way of publishing a package is to upload it to a package registry – an online software repository. The de facto standard is the npm registry but it is not the only option. For example, companies can host their own internal registries.

A package manager is a command line tool that downloads packages from a registry (or other sources) and installs them locally or globally. If a package contains bin scripts, it also makes those available locally or globally.

The most popular package manager is called npm and comes bundled with Node.js. Its name originally stood for “Node Package Manager”. Later, when npm and the npm registry were used not only for Node.js packages, the definition was changed to “npm is not a package manager” (source).

There are other popular package managers such as yarn and pnpm. All of these package managers use the npm registry by default.

Each package in the npm registry has a name. There are two kinds of names:

5.2 The file system layout of a package

Once a package my-package is fully installed, it almost always looks like this:

my-package/
  package.json
  node_modules/
  [More files]

What are the purposes of these file system entries?

Some packages also have the file package-lock.json that sits next to package.json: It records the exact versions of the dependencies that were installed and is kept up to date if we add more dependencies via npm.

5.2.1 package.json

This is a starter package.json that can be created via npm:

{
  "name": "my-package",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

What are the purposes of these properties?

Other useful properties:

For more information on package.json, see the npm documentation.

5.2.2 Property "dependencies" of package.json

This is what the dependencies in a package.json file look like:

"dependencies": {
  "minimatch": "^5.1.0",
  "mocha": "^10.0.0"
}

The properties record both the names of packages and constraints for their versions.

Versions themselves follow the semantic versioning standard. They are up to three numbers (the second and third number are optional and zero by default) separated by dots:

  1. Major version: This number changes when a packages changes in incompatible ways.
  2. Minor version: This number changes when functionality is added in a backward compatible manner.
  3. Patch version: This number changes when backward compatible bug fixes are made.

Node’s version ranges are explained in the semver repository. Examples include:

5.2.3 Property "bin" of package.json

This is how we can tell npm to install modules as shell scripts:

"bin": {
  "my-shell-script": "./src/shell/my-shell-script.mjs",
  "another-script": "./src/shell/another-script.mjs"
}

If we install a package with this "bin" value globally, Node.js ensures that the commands my-shell-script and another-script become available at the command line.

If we install the package locally, we can use the two commands in package scripts or via the npx command.

A string is also allowed as the value of "bin":

{
  "name": "my-package",
  "bin": "./src/main.mjs"
}

This is an abbreviation for:

{
  "name": "my-package",
  "bin": {
    "my-package": "./src/main.mjs"
  }
}

5.2.4 Property "license" of package.json

The value of property "license" is always a string with a SPDX license ID. For example, the following value denies others the right to use a package under any terms (which is useful if a package is unpublished):

"license": "UNLICENSED"

The SPDX website lists all available license IDs. If you find it difficult to pick one, the website “Choose an open source license” can help – for example, this is the advice if you “want it simple and permissive”:

The MIT License is short and to the point. It lets people do almost anything they want with your project, like making and distributing closed source versions.

Babel, .NET, and Rails use the MIT License.

You can use that license like this:

"license": "MIT"

5.3 Archiving and installing packages

Packages in the npm registry are often archived in two different ways:

Either way, the package is archived without its dependencies – which we have to install before we can use it.

If a package is stored in a git repository:

If a package is published to the npm registry:

Dev dependencies (property devDependencies in package.json) are only installed during development but not when we install the package from the npm registry.

Note that unpublished packages in git repositories are handled similarly to published packages during development.

5.3.1 Installing a package from git

To install a package pkg from git, we clone its repository and:

cd pkg/
npm install

Then the following steps are performed:

If the root package doesn’t have a package-lock.json file, it is created during installation (as mentioned, dependencies don’t have this file).

In a dependency tree, the same dependency may exist multiple times, possibly in different versions. There a ways to minimize duplication, but that is beyond the scope of this chapter.

5.3.1.1 Reinstalling a package

This is a (slightly crude) way of fixing issues in a dependency tree:

cd pkg/
rm -rf node_modules/
rm package-lock.json
npm install

Note that that may result in different, newer, packages being installed. We can avoid that by not deleting package-lock.json.

5.3.2 Creating a new package and installing dependencies

There are many tools and technique for setting up new packages. This is one simple way:

mkdir my-package
cd my-package/
npm init --yes

Afterward, the directory looks like this:

my-package/
  package.json

This package.json has the starter content that we have already seen.

5.3.2.1 Installing dependencies

Right now, my-package doesn’t have any dependencies. Let’s say we want to use the library lodash-es. This is how we install it into our package:

npm install lodash-es

This command performs the following steps:

5.4 Referring to modules via specifiers

Code in other ECMAScript modules is accessed via import statements (line A and line B):

// Static import
import {namedExport} from 'https://example.com/some-module.js'; // (A)
console.log(namedExport);

// Dynamic import
import('https://example.com/some-module.js') // (B)
.then((moduleNamespace) => {
  console.log(moduleNamespace.namedExport);
});

Both static imports and dynamic imports use module specifiers to refer to modules:

There are three kinds of module specifiers:

5.4.1 Filename extensions in module specifiers

Caveat of style 3 bare specifiers: How the filename extension is interpreted depends on the dependency and may differ from the importing package. For example, the importing package may use .mjs for ESM modules and .js for CommonJS modules, while the ESM modules exported by the dependency may have bare paths with the filename extension .js.

5.5 Module specifiers in Node.js

Let’s see how module specifiers work in Node.js.

5.5.1 Resolving module specifiers in Node.js

The Node.js resolution algorithm works as follows:

This is the algorithm:

The result of the resolution algorithm must point to a file. That explains why absolute specifiers and relative specifiers always have filename extensions. Bare specifiers mostly don’t because they are abbreviations that are looked up in package exports.

Module files usually have these filename extensions:

If Node.js executes code provided via stdin, --eval or --print, we use the following command-line option so that it is interpreted as an ES module:

--input-type=module

5.5.2 Package exports: controlling what other packages see

In this subsection, we are working with a package that has the following file layout:

my-lib/
  dist/
    src/
      main.js
      util/
        errors.js
      internal/
        internal-module.js
    test/

Package exports are specified via property "exports" in package.json and support two important features:

Recall the three styles of bare specifiers:

Package exports help us with all three styles

5.5.2.1 Style 1: configuring which file represents (the bare specifier for) the package

package.json:

{
  "main": "./dist/src/main.js",
  "exports": {
    ".": "./dist/src/main.js"
  }
}

We only provide "main" for backward-compatibility (with older bundlers and Node.js 12 and older). Otherwise, the entry for "." is enough.

With these package exports, we can now import from my-lib as follows.

import {someFunction} from 'my-lib';

This imports someFunction() from this file:

my-lib/dist/src/main.js
5.5.2.2 Style 2: mapping extension-less subpaths to module files

package.json:

{
  "exports": {
    "./util/errors": "./dist/src/util/errors.js"
  }
}

We are mapping the specifier subpath 'util/errors' to a module file. That enables the following import:

import {UserError} from 'my-lib/util/errors';
5.5.2.3 Style 2: better subpaths without extensions for a subtree

The previous subsection explained how to create a single mapping for an extension-less subpath. There is also a way to create multiple such mappings via a single entry:

package.json:

{
  "exports": {
    "./lib/*": "./dist/src/*.js"
  }
}

Any file that is a descendant of ./dist/src/ can now be imported without a filename extension:

import {someFunction} from 'my-lib/lib/main';
import {UserError}    from 'my-lib/lib/util/errors';

Note the asterisks in this "exports" entry:

"./lib/*": "./dist/src/*.js"

These are more instructions for how to map subpaths to actual paths than wildcards that match fragments of file paths.

5.5.2.4 Style 3: mapping subpaths with extensions to module files

package.json:

{
  "exports": {
    "./util/errors.js": "./dist/src/util/errors.js"
  }
}

We are mapping the specifier subpath 'util/errors.js' to a module file. That enables the following import:

import {UserError} from 'my-lib/util/errors.js';
5.5.2.5 Style 3: better subpaths with extensions for a subtree

package.json:

{
  "exports": {
    "./*": "./dist/src/*"
  }
}

Here, we shorten the module specifiers of the whole subtree under my-package/dist/src:

import {InternalError} from 'my-package/util/errors.js';

Without the exports, the import statement would be:

import {InternalError} from 'my-package/dist/src/util/errors.js';

Note the asterisks in this "exports" entry:

"./*": "./dist/src/*"

These are not filesystem globs but instructions for how to map external module specifiers to internal ones.

5.5.2.6 Exposing a subtree while hiding parts of it

With the following trick, we expose everything in directory my-package/dist/src/ with the exception of my-package/dist/src/internal/

"exports": {
  "./*": "./dist/src/*",
  "./internal/*": null
}

Note that this trick also works when exporting subtrees without filename extensions.

5.5.2.7 Conditional package exports

We can also make exports conditional: Then a given path maps to different values depending on the context in which a package is used.

Node.js vs. browsers. For example, we could provide different implementations for Node.js and for browsers:

"exports": {
  ".": {
    "node": "./main-node.js",
    "browser": "./main-browser.js",
    "default": "./main-browser.js"
  }
}

The "default" condition matches when no other key matches and must come last. Having one is recommended whenever we are distinguishing between platforms because it takes care of new and/or unknown platforms.

Development vs. production. Another use case for conditional package exports is switching between “development” and “production” environments:

"exports": {
  ".": {
    "development": "./main-development.js",
    "production": "./main-production.js",
  }
}

In Node.js we can specify an environment like this:

node --conditions development app.mjs

5.5.3 Package imports

Package imports let a package define abbreviations for module specifiers that it can use itself, internally (where package exports define abbreviations for other packages). This is an example:

package.json:

{
  "imports": {
    "#some-pkg": {
      "node": "some-pkg-node-native",
      "default": "./polyfills/some-pkg-polyfill.js"
    }
  },
  "dependencies": {
    "some-pkg-node-native": "^1.2.3"
  }
}

The package import # is conditional (with the same features as conditional package exports):

(Only package imports can refer to external packages, package exports can’t do that.)

What are the use cases for package imports?

Be careful when using package imports with a bundler: This feature is relatively new and your bundler may not support it.

5.5.4 node: protocol imports

Node.js has many built-in modules such as 'path' and 'fs'. All of them are available as both ES modules and CommonJS modules. One issue with them is that they can be overridden by modules installed in node_modules which is both a security risk (if it happens accidentally) and a problem if Node.js wants to introduce new built-in modules in the future and their names are already taken by npm packages.

We can use the node: protocol to make it clear that we want to import a built-in module. For example, the following two import statements are mostly equivalent (if no npm module is installed that has the name 'fs'):

import * as fs from 'node:fs/promises';
import * as fs from 'fs/promises';

An additional benefit of using the node: protocol is that we immediately see that an imported module is built-in. Given how many built-in modules there are, that helps when reading code.

Due to node: specifiers having a protocol, they are considered absolute. That’s why they are not looked up in node_modules.

Comments
Next: 6 An overview of npm (a package manager for JavaScript)