Speeding up Statamic tests

The problem

At InnoBrain GmbH we’re building a lot of large Statamic sites full of content. We also like to run the sites file-based, even if they get a little bigger. We also like tests. Our own Website ran the full suite in 103.6s for 171 tests. The 10 slowest tests were only ~16% of that. The cost comes per test, and that’s because of the Stache.

Laravel builds a fresh application per test we run. phpunit.xml by default sets CACHE_STORE=array. Every boot has to build the Stache again if we run anything that touches it. It makes a big difference. Rendering a page fresh (cold Stache warm) = 1727ms, every render after = ~150ms.

All we have to do is persist

So, we create a custom Stache store for the tests. We don’t want to use the normal one here so we don’t corrupt it with test data. Keep the watcher on, so content mutations still invalidate correctly.

// config/statamic/stache.php
'watcher' => env('STATAMIC_STACHE_WATCHER', true),
'cache_store' => env('STATAMIC_STACHE_CACHE_STORE'),

// config/cache.php — new store (path keyed by TEST_TOKEN, see below)
'stache_test' => [
    'driver' => 'file',
    'path' => storage_path('framework/testing/stache-cache-'.env('TEST_TOKEN', '0')),
    'lock_path' => storage_path('framework/testing/stache-cache-'.env('TEST_TOKEN', '0')),
],

<!-- phpunit.xml -->
<env name="STATAMIC_STACHE_CACHE_STORE" value="stache_test"/>

Parallel persistence

Our CI runs pest --parallel. We already parallelize our content folder with a content-{TEST_TOKEN} dir, so we also have to divide up the Staches like that. We just key the cache path by TEST_TOKEN (shown above), matching the per-worker content directory.

Results

Running the tests serial got us from 103.6s -> 23.6s (4.4x improvement). Parallel: we’re now at ~9.7s. This has big impact on CI, where the runners are slower. Tests ran for ~7min before.