Error enospc no space left on device write

Description

When trying to deploy a yarn workspace monorepo I receive "ENOSPC: no space left on device, write" (free plan) when just installing dependencies:

11/15 10:53 AM (4h)
installing to /tmp/252defbc/user/packages/enum-server-api
11/15 10:53 AM (4h)
yarn install v1.12.3
11/15 10:53 AM (4h)
[1/5] Validating package.json...
11/15 10:53 AM (4h)
[2/5] Resolving packages...
11/15 10:53 AM (4h)
[3/5] Fetching packages...
11/15 10:53 AM (4h)
info fsevents@1.2.4: The platform "linux" is incompatible with this module.
11/15 10:53 AM (4h)
info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
11/15 10:53 AM (4h)
[4/5] Linking dependencies...
11/15 10:53 AM (4h)
error Could not write file "/tmp/252defbc/user/packages/enum-server-api/yarn-error.log": "ENOSPC: no space left on device, write"

Is there any limitation regarding dependencies size or working with monorepos?

How to reproduce

To reproduce you can simply try to deploy this branch:

https://github.com/nitzano/enum-server/tree/now_deploy

My now.json setup:

{
  "version": 2,
  "name": "enum-server",
  "builds": [
    {
      "src": "packages/enum-server-api/package.json",
      "use": "@now/node-server"
    }
  ]
}

Suggestions for better-than-ext4 choices for storing masses of small files:

If you’re using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. A quick Google search found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. I don’t know if it’s worth using on a single host, without taking advantage of replication.

Another object-storage system is OpenStack’s Swift. Its design docs say it stores each object as a separate file, with metadata in xattrs. Here’s an article about it. Their deployment guide says they found XFS gave the best performance for object storage. So even though the workload isn’t what XFS is best at, it was apparently better than the competitors when RackSpace was testing things. Possibly Swift favours XFS because XFS has good / fast support for extended attributes. It might be that ext3/ext4 would do ok on single disks as an object-store backend if extra metadata wasn’t needed (or if it was kept inside the binary file).

Swift does the replication / load-balancing for you, and suggests that you give it filesystems made on raw disks, not RAID. It points out that its workload is essentially worst-case for RAID5 (which makes sense if we’re talking about a workload with writes of small files. XFS typically doesn’t quite pack them head-to-tail, so you don’t get full-stripe writes, and RAID5 has to do some reads to update the parity stripe. Swift docs also talk about using 100 partitions per drive. I assume that’s a Swift term, and isn’t talking about making 100 different XFS filesystems on each SATA disk.

Running a separate XFS for every disk is actually a huge difference. Instead of one gigantic free-inode map, each disk will have a separate XFS with separate free-lists. Also, it avoids the RAID5 penalty for small writes.

If you already have your software built to use a filesystem directly as an object store, rather than going through something like Swift to handle the replication / load-balancing, then you can at least avoid having all your files in a single directory. (I didn’t see Swift docs say how they lay out their files into multiple directories, but I’m certain they do.)

With almost any normal filesystem, it will help to used a structure like

1234/5678   # nested medium-size directories instead of
./12345678   # one giant directory

Probably about 10k entries is reasonable, so taking a well-distributed 4 characters of your object names and using them as directories is an easy solution. It doesn’t have to be very well balanced. The odd 100k directory probably won’t be a noticeable issue, and neither will some empty directories.

XFS is not ideal for huge masses of small files. It does what it can, but it’s more optimized for streaming writes of larger files. It’s very good overall for general use, though. It doesn’t have ENOSPC on collisions in its directory indexing (AFAIK), and can handle having one directory with millions of entries. (But it’s still better to use at least a one-level tree.)

Dave Chinner had some comments on XFS performance with huge numbers of inodes allocated, leading to slow-ish touch performance. Finding a free inode to allocate starts taking more CPU time, as the free inode bitmap gets fragmented. Note that this is not an issue of one-big-directory vs. multiple directories, but rather an issue of many used inodes over the whole filesystem. Splitting your files into multiple directories helps with some problems, like the one that ext4 choked on in the OP, but not the whole-disk problem of keeping track of free space. Swift’s separate-filesystem-per-disk helps with this, compared to on giant XFS on a RAID5.

I don’t know if btrfs is good at this, but I think it may be. I think Facebook employs its lead developer for a reason. Some benchmarks I’ve seen, of stuff like untarring a Linux kernel source, show btrfs does well.

I know reiserfs was optimized for this case, but it’s barely, if at all, maintained anymore. I really can’t recommend going with reiser4. It might be interesting to experiment, though. But it’s by far the least future-proof choice. I’ve also seen reports of performance degradation on aged reiserFS, and there’s no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

I’m probably missing something, so final recommendation: ask about this on serverfault! If I had to pick something right now, I’d say give BTRFS a try, but make sure you have backups. (esp. if you use BTRFS’s build-in multiple disk redundancy, instead of running it on top of RAID. The performance advantages could be big, since small files are bad news for RAID5, unless it’s a read-mostly workload.)