Shell scripting with Node.js
You can buy the offline version of this book (HTML, PDF, EPUB, MOBI) and support the free online version.
(Ad, please don’t block.)

14 Creating cross-platform shell scripts

In this chapter, we learn how to implement shell scripts via Node.js ESM modules. There are two common ways of doing so:

14.1 Required knowledge

You should be loosely familiar with the following two topics:

14.1.1 What’s next in this chapter

Windows doesn’t really support standalone shell scripts written in JavaScript. Therefore, we’ll first look into how to write standalone scripts with filename extensions for Unix. That knowledge will help us with creating packages that contain shell scripts. Later, we’ll learn:

Installing shell scripts via packages is the topic of §13 “Installing npm packages and running bin scripts”.

14.2 Node.js ESM modules as standalone shell scripts on Unix

Let’s turn an ESM module into a Unix shell script that we can run without it being inside a package. In principle, we can choose between two filename extensions for ESM modules:

However, since we want to create a standalone script, we can’t rely on package.json being there. Therefore, we have to use the filename extension .mjs (we’ll get to workarounds later).

The following file has the name hello.mjs:

import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);

We can already run this file:

node hello.mjs

14.2.1 Node.js shell scripts on Unix

We need to do two things so that we can run hello.mjs like this:

./hello.mjs

These things are:

14.2.2 Hashbangs on Unix

In a Unix shell script, the first line is a hashbang – metadata that tells the shell how to execute the file. For example, this is the most common hashbang for Node.js scripts:

#!/usr/bin/env node

This line has the name “hashbang” because it starts with a hash symbol and an exclamation mark. It is also often called “shebang”.

If a line starts with a hash, it is a comment in most Unix shells (sh, bash, zsh, etc.). Therefore, the hashbang is ignored by those shells. Node.js also ignores it, but only if it is the first line.

Why don’t we use this hashbang?

#!/usr/bin/node

Not all Unixes install the Node.js binary at that path. How about this path then?

#!node

Alas, not all Unixes allow relative paths. That’s why we refer to env via an absolute path and use it to run node for us.

For more information on Unix hashbangs, see “Node.js shebang” by Alex Ewerlöf.

14.2.2.1 Passing arguments to the Node.js binary

What if we want to pass arguments such as command line options to the Node.js binary?

One solution that works on many Unixes is to use option -S for env which prevents it from interpreting all of its arguments as a single name of a binary:

#!/usr/bin/env -S node --disable-proto=throw

On macOS, the previous command works even without -S; on Linux it usually doesn’t.

14.2.2.2 Hashbang pitfall: creating hashbangs on Windows

If we use a text editor on Windows to create an ESM module that should run as a script on either Unix or Windows, we have to add a hashbang. If we do that, the first line will end with the Windows line terminator \r\n:

#!/usr/bin/env node\r\n

Running a file with such a hashbang on Unix produces the following error:

env: node\r: No such file or directory

That is, env thinks the name of the executable is node\r. There are two ways to fix this.

First, some editors automatically check which line terminators are already used in a file and keep using them. For example, Visual Studio Code, shows the current line terminator (it calls it “end of line sequence”) in the status bar at the bottom right:

We can switch pick a line terminator by clicking on that status information.

Second, we can create a minimal file my-script.mjs with only Unix line terminators that we never edit on Windows:

#!/usr/bin/env node
import './main.mjs';

14.2.3 Making files executable on Unix

In order to become a shell script, hello.mjs must also be executable (a permission of files), in addition to having a hashbang:

chmod u+x hello.mjs

Note that we made the file executable (x) for the user who created it (u), not for everyone.

14.2.4 Running hello.mjs directly

hello.mjs is now executable and looks like this:

#!/usr/bin/env node

import * as os from 'node:os';

const {username} = os.userInfo();
console.log(`Hello ${username}!`);

We can therefore run it like this:

./hello.mjs

Alas, there is no way to tell node to interpret a file with an arbitrary extension as an ESM module. That’s why we have to use the extension .mjs. Workarounds are possible but complicated, as we’ll see later.

14.3 Creating an npm package with shell scripts

In this section we create an npm package with shell scripts. We then examine how we can install such a package so that its scripts become available at the command line of your system (Unix or Windows).

The finished package is available here:

14.3.1 Setting up the package’s directory

These commands work on both Unix and Windows:

mkdir demo-shell-scripts
cd demo-shell-scripts
npm init --yes

Now there are the following files:

demo-shell-scripts/
  package.json
14.3.1.1 package.json for unpublished packages

One option is to create a package and not publish it to the npm registry. We can still install such a package on our system (as explained later). In that case, our package.json looks as follows:

{
  "private": true,
  "license": "UNLICENSED"
}

Explanations:

14.3.1.2 package.json for published packages

If we want to publish our package to the npm registry, our package.json looks like this:

{
  "name": "@rauschma/demo-shell-scripts",
  "version": "1.0.0",
  "license": "MIT"
}

For your own packages, you need to replace the value of "name" with a package name that works for you:

14.3.2 Adding dependencies

Next, we install a dependency that we want to use in one of our scripts – package lodash-es (the ESM version of Lodash):

npm install lodash-es

This command:

If we only use a package during development, we can add it to "devDependencies" instead of to "dependencies" and npm will only install it if we run npm install inside our package’s directory, but not if we install it as a dependency. A unit testing library is a typical dev dependency.

These are two ways in which we can install a dev dependency:

The second way means that we can easily postpone the decision whether a package is a dependency or a dev dependency.

14.3.3 Adding content to the package

Let’s add a readme file and two modules homedir.mjs and versions.mjs that are shell scripts:

demo-shell-scripts/
  package.json
  package-lock.json
  README.md
  src/
    homedir.mjs
    versions.mjs

We have to tell npm about the two shell scripts so that it can install them for us. That’s what property "bin" in package.json is for:

"bin": {
  "homedir": "./src/homedir.mjs",
  "versions": "./src/versions.mjs"
}

If we install this package, two shell scripts with the names homedir and versions will become available.

You may prefer the filename extension .js for the shell scripts. Then, instead of the previous property, you have to add the following two properties to package.json:

"type": "module",
"bin": {
  "homedir": "./src/homedir.js",
  "versions": "./src/versions.js"
}

The first property tells Node.js that it should interpret .js files as ESM modules (and not as CommonJS modules – which is the default).

This is what homedir.mjs looks like:

#!/usr/bin/env node
import {homedir} from 'node:os';

console.log('Homedir: ' + homedir());

This module starts with the aforementioned hashbang which is required if we want to use it on Unix. It imports function homedir() from the built-in module node:os, calls it and logs the result to the console (i.e., standard output).

Note that homedir.mjs does not have to be executable; npm ensure executability of "bin" scripts when it installs them (we’ll see how soon).

versions.mjs has the following content:

#!/usr/bin/env node

import {pick} from 'lodash-es';

console.log(
  pick(process.versions, ['node', 'v8', 'unicode'])
);

We import function pick() from Lodash and use it to display three properties of the object process.versions.

14.3.4 Running the shell scripts without installing them

We can run, e.g., homedir.mjs like this:

cd demo-shell-scripts/
node src/homedir.mjs

14.4 How npm installs shell scripts

14.4.1 Installation on Unix

A script such as homedir.mjs does not need to be executable on Unix because npm installs it via an executable symbolic link:

14.4.2 Installation on Windows

To install homedir.mjs on Windows, npm creates three files:

npm adds these files to a directory:

14.5 Publishing the example package to the npm registry

Let’s publish package @rauschma/demo-shell-scripts (which we have created previously) to npm. Before we use npm publish to upload the package, we should check that everything is configured properly.

14.5.1 Which files are published? Which files are ignored?

The following mechanisms are used to exclude and include files when publishing:

The npm documentation has more details on what’s included and whats excluded when publishing.

14.5.2 Checking if a package is properly configured

There are several things we can check before we upload a package.

14.5.2.1 Checking which files will be uploaded

A dry run of npm install runs the command without uploading anything:

npm publish --dry-run

This displays which files would be uploaded and several statistics about the package.

We can also create an archive of the package as it would exist on the npm registry:

npm pack

This command creates the file rauschma-demo-shell-scripts-1.0.0.tgz in the current directory.

14.5.2.2 Installing the package globally – without uploading it

We can use either of the following two commands to install our package globally without publishing it to the npm registry:

npm link
npm install . -g

To see if that worked, we can open a new shell and check if the two commands are available. We can also list all globally installed packages:

npm ls -g
14.5.2.3 Installing the package locally (as a depencency) – without uploading it

To install our package as a dependency, we have to execute the following commands (while we are in directory demo-shell-scripts):

cd ..
mkdir sibling-directory
cd sibling-directory
npm init --yes
npm install ../demo-shell-scripts

We can now run, e.g., homedir with either one of the following two commands:

npx homedir
./node_modules/.bin/homedir

14.5.3 npm publish: uploading packages to the npm registry

Before we can upload our package, we need to create an npm user account. The npm documentation describes how to do that.

Then we can finally publish our package:

npm publish --access public

We have to specify public access because the defaults are:

Option --access only has an effect the first time we publish. Afterward, we can omit it and need to use npm access to change the access level.

We can change the default for the initial npm publish via publishConfig.access in package.json:

"publishConfig": {
  "access": "public"
}
14.5.3.1 A new version is required for every upload

Once we have uploaded a package with a specific version, we can’t use that version again, we have to increase either of the three components of the version:

major.minor.patch

14.5.4 Automatically performing tasks every time before publishing

There may be steps that we want to perform every time before we upload a package – e.g.:

That can be done automatically via the package.json property `“scripts”. That property can look like this:

"scripts": {
  "build": "tsc",
  "test": "mocha --ui qunit",
  "dry": "npm publish --dry-run",
  "prepublishOnly": "npm run test && npm run build"
}

mocha is a unit testing library. tsc is the TypeScript compiler.

The following package scripts are run before npm publish:

For more information on this topic, see §15 “Running cross-platform tasks via npm package scripts”.

14.6 Standalone Node.js shell scripts with arbitrary extensions on Unix

14.6.1 Unix: arbitrary filename extension via a custom executable

The Node.js binary node uses the filename extension to detect which kind of module a file is. There currently is no command line option to override that. And the default is CommonJS, which is not what we want.

However, we can create our own executable for running Node.js and, e.g., call it node-esm. Then we can rename our previous standalone script hello.mjs to hello (without any extension) if we change the first line to:

#!/usr/bin/env node-esm

Previously, the argument of env was node.

This is an implementation of node-esm proposed by Andrea Giammarchi:

#!/usr/bin/env sh
input_file=$1
shift
exec node --input-type=module - $@ < $input_file

This executable sends the content of a script to node via standard input. The command line option --input-type=module tells Node.js that the text it receives is an ESM module.

We also use the following Unix shell features:

Before we can use node-esm, we have to make sure that it is executable and can be found via the $PATH. How to do that is explained later.

14.6.2 Unix: arbitrary filename extension via a shell prolog

We have seen that we can’t specify the module type for a file, only for standard input. Therefore, we can write a Unix shell script hello that uses Node.js to run itself as an ESM module (based on work by sambal.org):

#!/bin/sh
':' // ; cat "$0" | node --input-type=module - $@ ; exit $?

import * as os from 'node:os';

const {username} = os.userInfo();
console.log(`Hello ${username}!`);

Most of the shell features that we are using here are described at the beginning of this chapter. $? contains the exit code of the last shell command that was executed. That enables hello to exit with the same code as node.

The key trick used by this script is that the second line is both Unix shell script code and JavaScript code:

An additional benefit of hiding the shell code from JavaScript is that JavaScript editors won’t be confused when it comes to processing and displaying the syntax.

14.7 Standalone Node.js shell scripts on Windows

14.7.1 Windows: configuring the filename extension .mjs

One option for creating standalone Node.js shell scripts on Windows is to the filename extension .mjs and configure it so that files that have it are run via node. Alas that only works for the Command shell, not for PowerShell.

Another downside is that we can’t pass arguments to a script that way:

>more args.mjs
console.log(process.argv);

>.\args.mjs one two
[
  'C:\\Program Files\\nodejs\\node.exe',
  'C:\\Users\\jane\\args.mjs'
]

>node args.mjs one two
[
  'C:\\Program Files\\nodejs\\node.exe',
  'C:\\Users\\jane\\args.mjs',
  'one',
  'two'
]

How do we configure Windows so that the Command shell directly runs files such as args.mjs?

File associations specify which app a file is opened with when we enter its name in a shell. If we associate the filename extension .mjs with the Node.js binary, we can run ESM modules in shells. One way to do that is via the Settings app, as explained in “How to Change File Associations in Windows” by Tim Fisher.

If we additionally add .MJS to the variable %PATHEXT%, we can even omit the filename extension when referring to an ESM module. This environment variable can be changed permanently via the Settings app – search for “variables”.

14.7.2 Windows Command shell: Node.js scripts via a shell prolog

On Windows, we are facing the challenge that there is no mechanism like hashbangs. Therefore, we have to use a workaround that is similar to the one we used for extensionless files on Unix: We create a script that runs the JavaScript code inside itself via Node.js.

Command shell scripts have the filename extension .bat. We can run a script named script.bat via either script.bat or script.

This is what hello.mjs looks like if we turn it into a Command shell script hello.bat:

:: /*
@echo off
more +5 %~f0 | node --input-type=module - %*
exit /b %errorlevel%
*/

import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);

Running this code as a file via node would require two features that don’t exist:

Therefore, we have no choice but to pipe the file’s content into node. We also use the following command shell features:

14.7.3 Windows PowerShell: Node.js scripts via a shell prolog

We can use a trick similar to the one used in the previous section and turn hello.mjs into a PowerShell script hello.ps1 as follows:

Get-Content $PSCommandPath | Select-Object -Skip 3 | node --input-type=module - $args
exit $LastExitCode
<#
import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);
// #>

We can run this script via either:

.\hello.ps1
.\hello

However, before we can do that, we need to set an execution policy that allows us to run PowerShell scripts (more information on execution policies):

The following command lets us run local scripts:

Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

14.8 Creating native binaries for Linux, macOS, and Windows

The npm package pkg turns a Node.js package into a native binary that even runs on systems where Node.js isn’t installed. It supports the following platforms: Linux, macOS, and Windows.

14.9 Shell paths: making sure shells find scripts

In most shells, we can type in a filename without directly referring to a file and they search several directories for a file with that name and run it. Those directories are usually listed in a special shell variable:

We need the PATH variable for two purposes:

14.9.1 Unix: $PATH

Most Unix shells have the variable $PATH that lists all paths where a shell looks for executables when we type in a command. Its value may look like this:

$ echo $PATH
/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin

The following command works on most shells (source) and changes the $PATH until we leave the current shell:

export PATH="$PATH:$HOME/bin"

The quotes are needed in case one of the two shell variables contains spaces.

14.9.1.1 Permanently changing the $PATH

On Unix, how the $PATH is configured depends on the shell. You can find out which shell you are running via:

echo $0

MacOS uses Zsh where the best place to permanently configure $PATH is the startup script $HOME/.zprofile – like this:

path+=('/Library/TeX/texbin')
export PATH

14.9.2 Changing the PATH variable on Windows (Command shell, PowerShell)

On Windows, the default environment variables of the Command shell and PowerShell can be configured (permanently) via the Settings app – search for “variables”.

Comments
Next: 15 Running cross-platform tasks via npm package scripts