[Feature] Download and cache dependencies from yarn.lock lockfile

[GauBen]

• I'd be willing to implement this feature (contributing guide)
• This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

Creating a Docker image of a Yarn-managed monorepo is not really efficient regarding layer caching. As of now there seems to be two possibilities:

# Copy the whole monorepo and do everything at once
COPY . .
RUN yarn install && yarn build

This will download the dependencies every time a file is changed

# Copy only package files
COPY package.json yarn.lock .
COPY packages/a/package.json ./packages/a
COPY packages/b/package.json ./packages/b
COPY packages/c/package.json ./packages/c
COPY packages/d/package.json ./packages/d
# ... endless and hand-maintained list
RUN yarn install

# Build everything
COPY . .
RUN yarn build

This will properly leverage layer caching but requires hand-maintenance of the docker file

Describe the solution you'd like

# Copy only the lock file
COPY yarn.lock .
RUN yarn cache download

# Create node_modules and build everything
COPY . .
RUN YARN_ENABLE_OFFLINE_MODE=1 yarn install && yarn build

This will only re-download the dependencies if yarn.lock is changed, which seems to offer the best of the solutions above, without the hassle

Reference: https://pnpm.io/cli/fetch

Describe the drawbacks of your solution

This solution builds on the current yarn cache clean command, suggesting creating new cache management commands (e.g. download, diff, audit...)

Describe alternatives you've considered

Cache commands should probably all be in yarn core, but this could definitely be a plugin if it can be implemented as a plugin

[merceyz]

You can avoid downloading all the packages anew by mounting a cache for the install step

RUN \
  --mount=type=cache,target=/root/.yarn \
  yarn

With this cache adding a new dependency will only fetch that one dependency.

[GauBen]

Thanks @merceyz for the advice, I will take a look. Relying on docker build cache rather than the host machine cache would probably be easier for my use case, but both should be possible

[GauBen]

I'm back with a prototype!

https://gist.github.com/GauBen/8a9847abfbee0138ed2c5fa04812a500

It's currently very experimental but works for my use case

merceyz: The Project class has a fetchEverything function you might be able to use there.

berry/packages/yarnpkg-core/sources/Project.ts

Lines 1095 to 1208 in 2a47867

	async fetchEverything({cache, report, fetcher: userFetcher, mode, persistProject = true}: InstallOptions) {
	const cacheOptions = {
	mockedPackages: this.disabledLocators,
	unstablePackages: this.conditionalLocators,
	};
	const fetcher = userFetcher || this.configuration.makeFetcher();
	const fetcherOptions = {checksums: this.storedChecksums, project: this, cache, fetcher, report, cacheOptions};
	let locatorHashes = Array.from(
	new Set(
	miscUtils.sortMap(this.storedResolutions.values(), [
	(locatorHash: LocatorHash) => {
	const pkg = this.storedPackages.get(locatorHash);
	if (!pkg)
	throw new Error(`Assertion failed: The locator should have been registered`);
	return structUtils.stringifyLocator(pkg);
	},
	]),
	),
	);
	// In "dependency update" mode, we won't trigger the link step. As a
	// result, we only need to fetch the packages that are missing their
	// hashes (to add them to the lockfile).
	if (mode === InstallMode.UpdateLockfile)
	locatorHashes = locatorHashes.filter(locatorHash => !this.storedChecksums.has(locatorHash));
	let firstError = false;
	const progress = Report.progressViaCounter(locatorHashes.length);
	await report.reportProgress(progress);
	const limit = pLimit(FETCHER_CONCURRENCY);
	await miscUtils.allSettledSafe(locatorHashes.map(locatorHash => limit(async () => {
	const pkg = this.storedPackages.get(locatorHash);
	if (!pkg)
	throw new Error(`Assertion failed: The locator should have been registered`);
	if (structUtils.isVirtualLocator(pkg))
	return;
	let fetchResult;
	try {
	fetchResult = await fetcher.fetch(pkg, fetcherOptions);
	} catch (error) {
	error.message = `${structUtils.prettyLocator(this.configuration, pkg)}: ${error.message}`;
	report.reportExceptionOnce(error);
	firstError = error;
	return;
	}
	if (fetchResult.checksum != null)
	this.storedChecksums.set(pkg.locatorHash, fetchResult.checksum);
	else
	this.storedChecksums.delete(pkg.locatorHash);
	if (fetchResult.releaseFs) {
	fetchResult.releaseFs();
	}
	}).finally(() => {
	progress.tick();
	})));
	if (firstError)
	throw firstError;
	const cleanInfo = persistProject && mode !== InstallMode.UpdateLockfile
	? await this.cacheCleanup({cache, report})
	: null;
	if (report.cacheMisses.size > 0 || cleanInfo) {
	const addedSizes = await Promise.all([...report.cacheMisses].map(async locatorHash => {
	const locator = this.storedPackages.get(locatorHash);
	const checksum = this.storedChecksums.get(locatorHash) ?? null;
	const p = cache.getLocatorPath(locator!, checksum);
	const stat = await xfs.statPromise(p);
	return stat.size;
	}));
	const finalSizeChange = addedSizes.reduce((sum, size) => sum + size, 0) - (cleanInfo?.size ?? 0);
	const addedCount = report.cacheMisses.size;
	const removedCount = cleanInfo?.count ?? 0;
	const addedLine = `${miscUtils.plural(addedCount, {
	zero: `No new packages`,
	one: `A package was`,
	more: `${formatUtils.pretty(this.configuration, addedCount, formatUtils.Type.NUMBER)} packages were`,
	})} added to the project`;
	const removedLine = `${miscUtils.plural(removedCount, {
	zero: `none were`,
	one: `one was`,
	more: `${formatUtils.pretty(this.configuration, removedCount, formatUtils.Type.NUMBER)} were`,
	})} removed`;
	const sizeLine = finalSizeChange !== 0
	? ` (${formatUtils.pretty(this.configuration, finalSizeChange, formatUtils.Type.SIZE_DIFF)})`
	: ``;
	const message = removedCount > 0
	? addedCount > 0
	? `${addedLine}, and ${removedLine}${sizeLine}.`
	: `${addedLine}, but ${removedLine}${sizeLine}.`
	: `${addedLine}${sizeLine}.`;
	report.reportInfo(MessageName.FETCH_NOT_CACHED, message);
	}
	}