Continuous Integration
Formulize runs its continuous integration on GitHub Actions. The workflow files are in .github/workflows/ in the repository, and there is nothing to sign up for or configure outside it — no external CI service, no credentials, no per-fork setup. A fork gets the same builds its parent does.
There are four workflows.
e2e test suite
e2e-test.yml — runs on every push to master. It is a thin trigger; the work is in e2e-test-manual.yml, which it calls.
e2e-test-manual.yml — the actual suite. It can also be run by hand from the Actions tab (Manual E2E Test Suite), against any branch, which is how you check a branch before merging it.
What a run does, in order:
- Checks out the repository and sets up Node 20.
- Sets the file and directory permissions Formulize needs to be writable.
- Brings up the application with
docker compose up -d --build— a real Apache, PHP and MariaDB, exactly the same Docker setup used for local development. The database starts genuinely empty; nothing is seeded. - Runs the standalone PHP logic tests — password hashing,
{element_handle}reference binding, normalize-then-escape form sinks,EMPTYSETsearches and search-term prefixes, partial date searches. These need nothing but PHP, so they run first and fail fast. - Installs the Playwright dependencies and the Chromium browser.
- Runs the setup specs —
npx playwright test formulize-core/01-setup --workers=1. These install Formulize from scratch and build the museum sample system. One worker, in file-name order, because each depends on the last. - Runs the validate specs —
npx playwright test formulize-core/02-validate --workers=4 --fully-parallel. These interrogate the system the setup specs built. This step is why a spec whose tests depend on each other has to saytest.describe.configure({ mode: 'serial' })— see Creating Tests. - Runs the sequential validate specs -
npx playwright test formulize-core/03-validate-sequential --workers=1. These change things the museum system shares, such as element handles, check that what depends on them still works, and put them back, so they run after the validate specs, on one worker. - Runs the appearance specs —
npx playwright test formulize-core/04-appearance --workers=1. These test the Lyris theme’s looks and the appearance editor, switching the whole site to Lyris and back, so they run after the validate specs, on one worker, not alongside anything else. - Runs the PHP integrity tests — element reference scan, saved views round trip, element code files. These need the finished system, so they come last.
- Dumps the database and collects the files the run produced, then tears the containers down.
Steps 7 to 9 are skipped when the manual run is given a test filter, because a partly built system is not the system they are asking about.
Skipping a run
Include [SKIP TEST] (or [SKIP TESTS]) in a commit message to skip the suite for that push. It is checked against every commit message in the push, so it is for small merges to master that genuinely cannot affect behaviour — documentation, comments — not for getting a red build through.
Reading the results
Every run uploads two artifacts, kept for seven days:
- playwright-report — the HTML report. Download it and open
index.html, or runnpx playwright show-reportagainst it. Failures carry a trace and a video, becauseplaywright.config.jssetstrace: 'retain-on-failure'andvideo: 'retain-on-failure'. The trace viewer shows the DOM, the network and the console at each step of the failing test. - final-state — the state of the system when the run ended: a database dump,
mainfile.php,install.lock, and the contents oftrust/,logs/,modules/formulize/code/andmodules/formulize/templates/screens/. This is collectedif: always(), so it is there for a failed run too, which is when it is worth having: you can restore the dump locally and look at the system the test was actually standing in.
The run also uses a PHP error reporter (tests/e2e/php-error-reporter.js), so PHP notices and warnings raised during the tests surface in the report rather than only in the container log.
Because the suite runs on master, a red build means master is broken, and that is what the badge on the repository readme is showing.
Measuring coverage
Run the workflow by hand with coverage ticked to find out which lines of PHP the suite runs. It counts every PHP test and every request the Playwright specs make, so the numbers are for the whole suite. The option is in the workflow on branches that have tests/coverage/, so choose one of those under Use workflow from, as well as the branch to test.
The run’s summary page shows:
- the share of the lines that can run that the suite ran, by folder
- what each set of tests ran, and what no other set ran
- the files with the most lines that didn’t run
- the lines that differ from where the branch left coverage_base,
masterunless you change it, that can run, and that no test ran. This is the list to check before merging a branch: a changed line on it isn’t tested by anything.
The coverage-report artifact has those as Markdown, along with files.md, every file and its share, coverage.json, which has the lines for each file and set of tests, and lcov.info. Editors can show lcov.info beside the code. In VS Code, put it in the root of the repository and use the Coverage Gutters extension.
A coverage run is slower, because PHP runs more slowly while it’s measured, so a spec close to its time limit can fail in a coverage run when it passes otherwise. Use the ordinary run to decide whether a branch passes.
A line counts as run if any request ran it. That doesn’t mean a test checked what the line did.
How it works is in tests/coverage/. prepend.php runs before every request and records the lines it ran, and merge.php makes the report.
Deploy Jekyll site to Pages
jekyll.yml — builds this documentation site and deploys it to GitHub Pages. It runs on pushes to master, can be run by hand, and is also fired by Formulize itself: the News form on formulize.net POSTs a repository_dispatch with the type formulize-news-updated when a story is saved, because news is baked into the site at build time and only reaches it on a rebuild.
Two parts of the build are not just Jekyll:
- The roadmap is fetched live from the GitHub API. Every open milestone whose title is a plain version number becomes a section on the roadmap, ordered by version, with its Marquee Feature issues under it. Closing a milestone on GitHub drops it from the roadmap on the next deploy; opening one adds it. Nothing about the roadmap is hardcoded in the workflow.
- The MCP tool reference is dumped from a real running copy of Formulize. The workflow brings up Docker, runs the install spec from the e2e suite to get an installed system, and then runs
mcp/dump_tools_for_docs.phpagainst it, so the tool reference shows actual resolved schemas — including the tools whose schemas are built entirely at runtime and cannot be read as static text.
If any part of the site can’t be built, such as the news, the roadmap or the tool reference, the build fails and nothing is deployed, so the site stays as it was. See An incomplete site is a failed build.
See Writing and previewing documentation for how to work on the site, and GitHub Pages config for how it is hosted.
Publish PHP Development Docker image
build-docker-dev-image.yaml — rebuilds and pushes the formulize/php-dev image to Docker Hub. It runs only when docker/php/Dockerfile or the workflow itself changes, and builds for both linux/amd64 and linux/arm64 so the development environment works on Apple Silicon as well as Intel and AMD machines.
This is the image docker-compose.yaml pulls, so a change to it reaches every developer’s environment the next time they rebuild.
Related
- Testing — what the test suites are and how to run them locally
- Creating Tests — writing a new Playwright spec
- Local development environment — the Docker setup CI uses