Distributed Caching in Nx
In Nx 8.11, you can configure Nx to use a local cache for builder command outputs. In specific scenarios, this makes your builder commands nearly instantaneous.
A builder command is anything that can be run with
nx run
(orng run
). This includesbuild
,test
andlint
.
Enabling Caching
Step 1: The caching will turned on by default in Nx 10. For now you have to add the following code to nx.json
{
// normal nx.json content...
"tasksRunnerOptions": {
"default": {
"runner": "@nrwl/workspace/src/tasks-runner/tasks-runner-v2",
"options": {
"cacheableOperations": ["build", "test", "lint"]
}
}
}
}
Step 2: Use Nx normally
The cacheableOperations
can be any command that you can execute with nx run [someCommand]
(or ng run
if you use Nx with the Angular CLI). With this configuration in place, any build
, test
or lint
commands will be automatically cached and a previously cached output will be used if it exists.
Caching and Nx affected
nx affected
ensures that a builder command will be run only on the projects that have been affected (compared to the master git branch).
The new caching feature takes effect when Nx starts to run a builder command on a particular project. If the project files are the exact same as a previously cached output, that cache will be used instead of actually running the command.
Implementation Details: The cache’s hash key is calculated based on all the files in the project and any dependencies of that project. The cached value is stored by default in
node_modules/.cache/nx
.
In the above sample monorepo, nx affected --target=test
will run tests on myapp-e2e
, myapp
, and feat-home
. If you run that command a second time without making any file changes, all three test runs will use a cached value and the console will output the test results very quickly. Now let’s say you change a file in the myapp
project. This breaks the cache for myapp
(since one of its files was changed) and for myapp-e2e
since one of its dependencies was changed. Runningnx affected --target=test
will attempt to run tests on all three projects: feat-home
will use the cached value, but myapp
and myapp-e2e
will re-execute their tests.
Caching will speed up your process without you needing to even think about it.
When does this help?
Scenario 1: Testing (and Linting)
- You run tests on the projects affected by your PR
- You change some files to fix some tests
- You run the same test command again
Most of the projects will use the cached test output, speeding up your tests. Only the projects that had their cache invalidated by the file changes will actually execute their test suites.
Scenario 2: Switching Branches
- You build one branch of your codebase
- You switch to a different branch and build it (which deletes the previous build)
- You switch back to the first branch and re-build it
Then that build will hit the cache and appear in the dist
folder almost instantly.
Scenario 3: Portal/Micro-Frontend App
- You have two or more interdependent apps (therefore two entry points), like an
api
app and aclient
app or multiple client apps being served together in a portal or micro-frontend style. - One of the apps gets deleted from your
dist
folder. - Without making any changes to that app or its dependencies, you re-build it.
Then that build will hit the cache and appear in the dist
folder almost instantly.
The Problem with Building From Source
If you build your application in Java, Go and .NET, you are likely to build individual pieces of an application and then use the compiled output.
By contrast, most of the time Webpack builds the whole application from source. This makes caching difficult. (Bazel and other tools relying on caching run up against this same problem.)
So, in our example above, if we ran nx affected --target=build
, and then changed the myapp
project and ran the same command, the cached value for feat-home
could not be re-used by Webpack.
With Nx you could make feat-home
buildable and use its output to bundle the final application with Webpack. In this case, the cached value offeat-home
will be used, but the dev setup will become more complex. In other words, You could set up your projects to build them incrementally, which will take advantage of caching, but it will require more complex setups which negatively affect dev experience for small/mid-sized projects.
The Brighter Future — Distributed Caching
We’re privately testing a distributed caching configuration which brings us closer to the ideal of never repeating a command on the same set of files across an entire organization.
Developer Builds
With distributed caching enabled, once a command has been run in CI, anyone who checks out that branch can immediately use the cached output without actually executing the build
command on their system.
CI Builds
CI rebuild times can be dramatically improved, since any project that has no changed dependencies will just load from the cache instead of re-executing the command. The data for the chart below was taken from two production code bases. This is the power of caching.
Learn More
- Check out nx.dev to learn more about Nx.
- Watch our free Nx course on YouTube.
If you liked this, click the 👏 below. Follow Isaac Mann and @nrwl_io to read more about software development.