Shell scripting with Node.js
You can buy the offline version of this book (HTML, PDF, EPUB, MOBI) and support the free online version.
(Ad, please don’t block.)

15 Running cross-platform tasks via npm package scripts

package.json has the property "scripts" which lets us define package scripts, small shell scripts that perform package-related tasks such as compiling artifacts or running tests. This chapter explains them and how we can write them so that they work on both Windows and Unixes (macOS, Linux, etc.).

15.1 npm package scripts

npm package scripts are defined via property "scripts" of package.json:

{
  ···
  "scripts": {
    "tsc": "tsc",
    "tscwatch": "tsc --watch",
    "tscclean": "shx rm -rf ./dist/*"
  },
  ···
}

The value of "scripts" is an object where each property defines a package script:

If we type:

npm run <script-name>

then npm executes the script whose name is script-name in a shell. For example, we can use:

npm run tscwatch

to run the following command in a shell:

tsc --watch

In this chapter, we will occasionally use the npm option -s, which is an abbreviation for --silent and tells npm run to produce less output:

npm -s run <script-name>

This option is covered in more detail in the section on logging.

15.1.1 Shorter npm commands for running package scripts

Some package scripts can be run via shorter npm commands:

Commands Equivalent
npm test, npm t npm run test
npm start npm run start
npm stop npm run stop
npm restart npm run restart

15.1.2 Which shell is used to run package scripts?

By default, npm runs package scripts via cmd.exe on Windows and via /bin/sh on Unix. We can change that via the npm configuration setting script-shell.

However, doing so is rarely a good idea: Many existing cross-platform scripts are written for sh and cmd.exe and will stop working.

15.1.3 Preventing package scripts from being run automatically

Some script names are reserved for life cycle scripts which npm runs whenever we execute certain npm commands.

For example, npm runs the script "postinstall" whenever we execute npm install (without arguments). Life cycle scripts are covered in more detail later.

If the configuration setting ignore-scripts is true, npm will never run scripts automatically, only if we invoke them directly.

15.1.4 Getting tab completion for package scripts on Unix

On Unix, npm supports tab completion for commands and package script names via npm completion. We can install it by adding this line to our .profile / .zprofile / .bash_profile / etc.:

. <(npm completion)

If you need tab completion for non-Unix platforms, do a web search such as “npm tab completion PowerShell”.

15.1.5 Listing and organizing package scripts

npm run without a name lists the available scripts. If the following scripts exist:

"scripts": {
  "tsc": "tsc",
  "tscwatch": "tsc --watch",
  "serve": "serve ./site/"
}

Then they are listed like this:

% npm run
Scripts available via `npm run-script`:
  tsc
    tsc
  tscwatch
    tsc --watch
  serve
    serve ./site/
15.1.5.1 Adding separators

If there are many package scripts, we can misuse script names as separators (script "help" will be explained in the next subsection):

  "scripts": {
    "help": "scripts-help -w 40",
    "\n========== Building ==========": "",
    "tsc": "tsc",
    "tscwatch": "tsc --watch",
    "\n========== Serving ==========": "",
    "serve": "serve ./site/"
  },

Now the scripts are listed as follows:

% npm run
Scripts available via `npm run-script`:
  help
    scripts-help -w 40

========== Building ==========

  tsc
    tsc
  tscwatch
    tsc --watch
  
========== Serving ==========

  serve
    serve ./site/

Note that the trick of prepending newlines (\n) works on Unix and on Windows.

15.1.5.2 Printing help information

The package script "help" prints help information via the bin script scripts-help from package @rauschma/scripts-help. We provide descriptions via the package.json property "scripts-help" (the value of "tscwatch" is abbreviated so that it fits into a single line):

"scripts-help": {
  "tsc": "Compile the TypeScript to JavaScript.",
  "tscwatch": "Watch the TypeScript source code [...]",
  "serve": "Serve the generated website via a local server."
}

This is what the help information looks like:

% npm -s run help
Package “demo”

╔══════╤══════════════════════════╗
║ help │ scripts-help -w 40       ║
╚══════╧══════════════════════════╝

Building

╔══════════╤══════════════════════════════════════════╗
║ tsc      │ Compile the TypeScript to JavaScript.    ║
╟──────────┼──────────────────────────────────────────╢
║ tscwatch │ Watch the TypeScript source code and     ║
║          │ compile it incrementally when and if     ║
║          │ there are changes.                       ║
╚══════════╧══════════════════════════════════════════╝

Serving

╔═══════╤══════════════════════════════════════════╗
║ serve │ Serve the generated website via a local  ║
║       │ server.                                  ║
╚═══════╧══════════════════════════════════════════╝

15.2 Kinds of package scripts

If certain names are used for scripts, they are run automatically in some situations:

All other scripts are called directly-run scripts.

15.2.1 Pre and post scripts

Whenever npm runs a package script PS, it automatically runs the following scripts – if they exist:

The following scripts contain the pre script prehello and the post script posthello:

"scripts": {
  "hello": "echo hello",
  "prehello": "echo BEFORE",
  "posthello": "echo AFTER"
},

This is what happens if we run hello:

% npm -s run hello
BEFORE
hello
AFTER

15.2.2 Life cycle scripts

npm runs life cycle scripts during npm commands such as:

If any of the life cycle scripts fail, the whole command stops immediately with an error.

What are use cases for life cycle scripts?

These are the most important life cycle scripts (for detailed information on all life cycle scripts, see the npm documentation):

The following table summarizes when these life cycle scripts are run:

prepublishOnly prepack prepare install
npm publish ✔ ✔ ✔
npm pack ✔ ✔
npm install ✔ ✔
global install ✔ ✔
install via git, path ✔

Caveat: Doing things automatically is always a bit tricky. I usually follow these rules:

15.3 The shell environment in which package scripts are run

In this section, we’ll occasionally use

node -p <expr>

which runs the JavaScript code in expr and prints the result to the terminal - for example:

% node -p "'hello everyone!'.toUpperCase()" 
HELLO EVERYONE!

15.3.1 The current directory

When a package script runs, the current directory is always the package directory, independently of where we are in the directory tree whose root it is. We can confirm that by adding the following script to package.json:

"cwd": "node -p \"process.cwd()\""

Let’s try out cwd on Unix:

% cd /Users/robin/new-package/src/util 
% npm -s run cwd
/Users/robin/new-package

Changing the current directory in this manner, helps with writing package scripts because we can use paths that are relative to the package directory.

15.3.2 The shell PATH

When a module M imports from a module whose specifier starts with the name of a package P, Node.js goes through node_modules directories until it finds the directory of P:

That is, M inherits the node_modules directories of its ancestor directories.

A similar kind of inheritance happens with bin scripts, which are stored in node_modules/.bin when we install a package. npm run temporarily adds entries to the shell PATH variable ($PATH on Unix, %Path% on Windows):

To see these additions, we can use the following package script:

"bin-dirs": "node -p \"JS\""

JS stands for a single line with this JavaScript code:

(process.env.PATH ?? process.env.Path)
.split(path.delimiter)
.filter(p => p.includes('.bin'))

On Unix, we get the following output if we run bin-dirs:

% npm -s run bin-dirs
[
  '/Users/robin/new-package/node_modules/.bin',
  '/Users/robin/node_modules/.bin',
  '/Users/node_modules/.bin',
  '/node_modules/.bin'
]

On Windows, we get:

>npm -s run bin-dirs
[
  'C:\\Users\\charlie\\new-package\\node_modules\\.bin',
  'C:\\Users\\charlie\\node_modules\\.bin',
  'C:\\Users\\node_modules\\.bin',
  'C:\\node_modules\\.bin'
]

15.4 Using environment variables in package scripts

In task runners such as Make, Grunt, and Gulp, variables are important because they help reduce redundancy. Alas, while package scripts don’t have their own variables, we can work around that deficiency by using environment variables (which are also called shell variables).

We can use the following commands to list platform-specific environment variables:

On macOS, the result looks like this:

TERM_PROGRAM=Apple_Terminal
SHELL=/bin/zsh
TMPDIR=/var/folders/ph/sz0384m11vxf5byk12fzjms40000gn/T/
USER=robin
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PWD=/Users/robin/new-package
HOME=/Users/robin
LOGNAME=robin
···

In the Windows Command shell, the result looks like this:

Path=C:\Windows;C:\Users\charlie\AppData\Roaming\npm;···
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROMPT=$P$G
TEMP=C:\Users\charlie\AppData\Local\Temp
TMP=C:\Users\charlie\AppData\Local\Temp
USERNAME=charlie
USERPROFILE=C:\Users\charlie
···

Additionally, npm temporarily adds more environment variables before it runs a package script. To see what the end result looks like, we can use the following command:

npm run env

This command invokes a built-in package script. Let’s try it out for this package.json:

{
  "name": "@my-scope/new-package",
  "version": "1.0.0",
  "bin": {
    "hello": "./hello.mjs"
  },
  "config": {
    "stringProp": "yes",
    "arrayProp": ["a", "b", "c"],
    "objectProp": {
      "one": 1,
      "two": 2
    }
  }
}

The names of all of npm’s temporary variables start with npm_. Let’s only print those, in alphabetical order:

npm run env | grep npm_ | sort

The npm_ variables have a hierarchical structure. Under npm_lifecycle_, we find the name and the definition of the currently running package script:

npm_lifecycle_event: 'env',
npm_lifecycle_script: 'env',

On Windows, npm_lifecycle_script would SET in this case.

Under prefix npm_config_, we can see some of npm’s configuration settings (which are described in the npm documentation). These are a few examples:

npm_config_cache: '/Users/robin/.npm',
npm_config_global_prefix: '/usr/local',
npm_config_globalconfig: '/usr/local/etc/npmrc',
npm_config_local_prefix: '/Users/robin/new-package',
npm_config_prefix: '/usr/local'
npm_config_user_agent: 'npm/8.15.0 node/v18.7.0 darwin arm64 workspaces/false',
npm_config_userconfig: '/Users/robin/.npmrc',

The prefix npm_package_ gives us access to the contents of package.json. Its top level looks like this:

npm_package_json: '/Users/robin/new-package/package.json',
npm_package_name: '@my-scope/new-package',
npm_package_version: '1.0.0',

Under npm_package_bin_, we can find the properties of the package.json property "bin":

npm_package_bin_hello: 'hello.mjs',

The npm_package_config_ entries give us access to the properties of "config":

npm_package_config_arrayProp: 'a\n\nb\n\nc',
npm_package_config_objectProp_one: '1',
npm_package_config_objectProp_two: '2',
npm_package_config_stringProp: 'yes',

That means that "config" lets us set up variables that we can use in package scripts. The next subsection explores that further.

Note the object was converted to “nested” entries (line 2 and line 3), while the Array (line 1) and the numbers (line 2 and line 3) were converted to strings.

These are the remaining npm_ environment variables:

npm_command: 'run-script',
npm_execpath: '/usr/local/lib/node_modules/npm/bin/npm-cli.js',
npm_node_execpath: '/usr/local/bin/node',

15.4.1 Getting and setting environment variables

The following package.json demonstrates how we can access variables defined via "config" in package scripts:

{
  "scripts": {
    "hi:unix": "echo $​npm_package_config_hi",
    "hi:windows": "echo %​npm_package_config_hi%"
  },
  "config": {
    "hi": "HELLO"
  }
}

Alas, there is no built-in cross-platform way of accessing environment variables from package scripts.

There are, however, packages with bin scripts that can help us.

Package env-var lets us get environment variables:

"scripts": {
  "hi": "env-var echo {{npm_package_config_hi}}"
}

Package cross-env lets us set environment variables:

"scripts": {
  "build": "cross-env FIRST=one SECOND=two node ./build.mjs"
}

15.4.2 Setting up environment variables via .env files

There are also packages that let us set up environment variables via .env files. These files have the following format:

# Comment
SECRET_HOST="https://example.com"
SECRET_KEY="123456789" # another comment

Using a file that is separate from package.json enables us to keep that data out of version control.

These are packages that support .env files:

15.5 Arguments for package scripts

Let’s explore how arguments are passed on to shell commands that we invoke via package scripts. We’ll use the following package.json:

{
  ···
  "scripts": {
    "args": "log-args"
  },
  "dependencies": {
    "log-args": "^1.0.0"
  }
}

The bin script log-args looks like this:

for (const [key,value] of Object.entries(process.env)) {
  if (key.startsWith('npm_config_arg')) {
    console.log(`${key}=${JSON.stringify(value)}`);
  }
}
console.log(process.argv.slice(2));

Positional arguments work as expected:

% npm -s run args three positional arguments
[ 'three', 'positional', 'arguments' ]

npm run consumes options and creates environment variables for them. They are not added to process.argv:

% npm -s run args --arg1='first arg' --arg2='second arg'
npm_config_arg2="second arg"
npm_config_arg1="first arg"
[]

If we want options to show up in process.argv, we have to use the option terminator --. That terminator is usually inserted after the name of the package script:

% npm -s run args -- --arg1='first arg' --arg2='second arg' 
[ '--arg1=first arg', '--arg2=second arg' ]

But we can also insert it before that name:

% npm -s run -- args --arg1='first arg' --arg2='second arg' 
[ '--arg1=first arg', '--arg2=second arg' ]

15.6 The npm log level (how much output is produced)

npm supports the following log levels:

Log level npm option Aliases
silent --loglevel silent -s --silent
error --loglevel error
warn --loglevel warn -q --quiet
notice --loglevel notice
http --loglevel http
timing --loglevel timing
info --loglevel info -d
verbose --loglevel verbose -dd --verbose
silly --loglevel silly -ddd

Logging refers to two kinds of activities:

The following subsections describe:

15.6.1 Log levels and information printed to the terminal

By default, package scripts are relatively verbose when it comes to terminal output. Take, for example, the following package.json file:

{
  "name": "@my-scope/new-package",
  "version": "1.0.0",
  "scripts": {
    "hello": "echo Hello",
    "err": "more does-not-exist.txt"
  },
  ···
}

This is what happens if the log level is higher than silent and the package script exits without errors:

% npm run hello

> @my-scope/new-package@1.0.0 hello
> echo Hello

Hello

This is what happens if the log level is higher than silent and the package script fails:

% npm run err      

> @my-scope/new-package@1.0.0 err
> more does-not-exist.txt

does-not-exist.txt: No such file or directory

With log level silent, the output becomes less cluttered:

% npm -s run hello
Hello

% npm -s run err
does-not-exist.txt: No such file or directory

Some errors are swallowed by -s:

% npm -s run abc
%

We need at least log level error to see them:

% npm --loglevel error run abc
npm ERR! Missing script: "abc"
npm ERR! 
npm ERR! To see a list of scripts, run:
npm ERR!   npm run

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/robin/.npm/_logs/2072-08-30T14_59_40_474Z-debug-0.log

Unfortunately, log level silent also suppresses the output of npm run (without arguments):

% npm -s run
%

15.6.2 Log levels and information written to the npm log

By default, the logs are written to the npm cache directory, whose path we can get via npm config:

% npm config get cache
/Users/robin/.npm

The contents of the log directory look like this:

% ls -1 /Users/robin/.npm/_logs
2072-08-28T11_44_38_499Z-debug-0.log
2072-08-28T11_45_45_703Z-debug-0.log
2072-08-28T11_52_04_345Z-debug-0.log

Each line in a log starts with a line index and a log level. This is an example of a log that was written with log level notice. Interestingly, even log levels that are “more verbose” than notice (such as silly) show up in it:

0 verbose cli /usr/local/bin/node /usr/local/bin/npm
1 info using npm@8.15.0
···
33 silly logfile done cleaning log files
34 timing command:run Completed in 9ms
···

If npm run returns with an error, the corresponding log ends like this:

34 timing command:run Completed in 7ms
35 verbose exit 1
36 timing npm Completed in 28ms
37 verbose code 1

If there is no error, the corresponding log ends like this:

34 timing command:run Completed in 7ms
35 verbose exit 0
36 timing npm Completed in 26ms
37 info ok

15.6.3 Configuring logging

npm config list --long prints default values for various settings. These are the default values for logging-related settings:

% npm config list --long | grep log
loglevel = "notice"
logs-dir = null
logs-max = 10

If the value of logs-dir is null, npm uses directory _logs inside the npm cache directory (as mentioned previously).

To permanently change these settings, we also use npm config – for example:

We can also temporarily change settings via command line options – for example:

npm --loglevel silent run build

Other ways of changing settings (such as using environment variables) are explained by the npm documentation.

15.6.4 Output of life cycle scripts that run during npm install

The output of life cycle scripts than run during npm install (without arguments) is hidden. We can change that by (temporarily or permanently) setting foreground-scripts to true.

15.6.5 Observations of how npm logging works

15.7 Cross-platform shell scripting

The two shells that are most commonly used for package scripts are:

In this section, we examine constructs that work in both shells.

15.7.1 Paths and quoting

Tips:

15.7.2 Chaining commands

There are two ways in which we can chain commands that work on both platforms:

Chaining while ignoring the exit code differs between platforms:

The following interaction demonstrates how && and || work on Unix (on Windows, we’d use dir instead of ls):

% ls unknown && echo "SUCCESS" || echo "FAILURE"
ls: unknown: No such file or directory
FAILURE

% ls package.json && echo "SUCCESS" || echo "FAILURE"
package.json
SUCCESS

15.7.3 The exit codes of package scripts

The exit code can be accessed via a shell variable:

npm run returns with the same exit code as the last shell script that was executed:

{
  ···
  "scripts": {
    "hello": "echo Hello",
    "err": "more does-not-exist.txt"
  }
}

The following interaction happens on Unix:

% npm -s run hello ; echo $?
Hello
0
% npm -s run err ; echo $?
does-not-exist.txt: No such file or directory
1

15.7.4 Piping and redirecting input and output

15.7.5 Commands that work on both platforms

The following commands exist on both platforms (but differ when it comes to options):

15.7.6 Running bin scripts and package-internal modules

The following package.json demonstrates three ways of invoking bin scripts in dependencies:

{
  "scripts": {
    "hi1": "./node_modules/.bin/cowsay Hello",
    "hi2": "cowsay Hello",
    "hi3": "npx cowsay Hello"
  },
  "dependencies": {
    "cowsay": "^1.5.0"
  }
}

Explanations:

On Unix, we can invoke package-local scripts directly – if they have hashbangs and are executable. However that doesn’t work on Windows, which is why it is better to invoke them via node:

"build": "node ./build.mjs"

15.7.7 node --eval and node --print

When the functionality of a package script becomes too complex, it’s often a good idea to implement it via a Node.js module – which makes it easy to write cross-platform code.

However, we can also use the node command to run small JavaScript snippets, which is useful for performing small tasks in a cross-platform manner. The relevant options are:

The following commands work on both Unix and Windows (only the comments are Unix-specific):

# Print a string to the terminal (cross-platform echo)
node -p "'How are you?'"

# Print the value of an environment variable
# (Alas, we can’t change variables via `process.env`)
node -p process.env.USER # only Unix
node -p process.env.USERNAME # only Windows
node -p "process.env.USER ?? process.env.USERNAME"

# Print all environment variables
node -p process.env

# Print the current working directory
node -p "process.cwd()"

# Print the path of the current home directory
node -p "os.homedir()"

# Print the path of the current temporary directory
node -p "os.tmpdir()"

# Print the contents of a text file
node -p "fs.readFileSync('package.json', 'utf-8')"

# Write a string to a file
node -e "fs.writeFileSync('file.txt', 'Text content', 'utf-8')"

If we need platform-specific line terminators, we can use os.EOL – for example, we could replace 'Text content' in the previous command with:

`line 1${os.EOL}line2${os.EOL}`

Observations:

15.8 Helper packages for common operations

15.8.1 Running package scripts from a command line

npm-quick-run provides a bin script nr that lets us use abbreviations to run package scripts – for example:

15.8.2 Running multiple scripts concurrently or sequentially

Running shell scripts concurrently:

The following two packages give us cross-platform options for that and for related functionality:

15.8.3 File system operations

Package shx lets us use “Unix syntax” to run various file system operations. Everything it does, works on Unix and Windows.

Creating a directory:

"create-asset-dir": "shx mkdir ./assets"

Removing a directory:

"remove-asset-dir": "shx rm -rf ./assets"

Clearing a directory (double quotes to be safe w.r.t. the wildcard symbol *):

"tscclean": "shx rm -rf \"./dist/*\""

Copying a file:

"copy-index": "shx cp ./html/index.html ./out/index.html"

Removing a file:

"remove-index": "shx rm ./out/index.html"

shx is based on the JavaScript library ShellJS, whose repository lists all supported commands. In addition to the Unix commands we have already seen, it also emulates: cat, chmod, echo, find, grep, head, ln, ls, mv, pwd, sed, sort, tail, touch, uniq, and others.

15.8.4 Putting files or directories into the trash

Package trash-cli works on macOS (10.12+), Linux, and Windows (8+). It puts files and directories into the trash and supports paths and glob patterns. These are examples of using it:

trash tmp-file.txt
trash tmp-dir
trash "*.jpg"

15.8.5 Copying trees of files

Package copyfiles lets us copy trees of files.

The following is a use case for copyfiles: In TypeScript, we can import non-code assets such as CSS and images. The TypeScript compiler compiles the code to a “dist” (output) directory but ignores non-code assets. This cross-platform shell command copies them to the dist directory:

copyfiles --up 1 "./ts/**/*.{css,png,svg,gif}" ./dist

TypeScript compiles:

my-pkg/ts/client/picker.ts  -> my-pkg/dist/client/picker.js

copy-assets copies:

my-pkg/ts/client/picker.css -> my-pkg/dist/client/picker.css
my-pkg/ts/client/icon.svg   -> my-pkg/dist/client/icon.svg

15.8.6 Watching files

Package onchange watches files and runs a shell command every time they change – for example:

onchange 'app/**/*.js' 'test/**/*.js' -- npm test

One common alternative (among many others):

15.8.7 Miscellaneous functionality

15.8.8 HTTP servers

During development, it’s often useful to have an HTTP server. The following packages (among many others) can help:

15.9 Expanding the capabilities of package scripts

15.9.1 per-env: switching between scripts, depending on $NODE_ENV

The bin script per-env lets us run a package script SCRIPT and automatically switches between (e.g.) SCRIPT:development, SCRIPT:staging, and SCRIPT:production, depending on the value of the environment variable NODE_ENV:

{
  "scripts": {
    // If NODE_ENV is missing, the default is "development"
    "build": "per-env",

    "build:development": "webpack -d --watch",
    "build:staging": "webpack -p",
    "build:production": "webpack -p"
  },
  // Processes spawned by `per-env` inherit environment-specific
  // variables, if defined.
  "per-env": {
    "production": {
      "DOCKER_USER": "my",
      "DOCKER_REPO": "project"
    }
  }
}

15.9.2 Defining operating-system-specific scripts

The bin script cross-os switches between scripts depending on the current operating system.

{
  "scripts": {
    "user": "cross-os user"
  },
  "cross-os": {
    "user": {
      "darwin": "echo $USER",
      "win32": "echo %USERNAME%",
      "linux": "echo $USER"
    }
  },
  ···
}

Supported property values are: darwin, freebsd, linux, sunos, win32.

15.10 Sources of this chapter

Comments
Next: V Handling common tasks in scripting