Skip to content

CI standards

Commands

Package commands have no prefix: plan, build, publish, manifest, test, update, revise-base, review-lines. Other concerns use dev-*, builder-*, registry-* and docs-*. make help groups them. Actions supplies runners, scheduling and credentials; Make/Go owns execution locally and in CI.

make dev-check runs actionlint, formatting, vet, Go tests, coverage floors, inventory/source validation and generated-file checks. Artifact builder coverage must be at least 70%; the source/metadata SDK at least 70%. Test artifact production and failure cleanup, not just plan serialization. make docs-build validates docs strictly. make dev-test-install uses Docker to verify that direct files beat newer distro packages and resolve a carrier-only dependency through the index; CI runs it when the plan selects package work.

Build graph

Select changed recipes and descendants; base changes select that release. Documentation changes do not rebuild packages. Helper/workflow/module PRs validate one representative Nginx target with the changed helper. Main does not repeat that package validation; its utility workflow publishes the helper separately. A weekly Sunday 03:11 UTC full build/test covers every enabled family and architecture without release promotion. Its separate concurrency groups cannot lock main releases.

Each target publishes after its architecture builds finish. Children wait for publication, not parent tests. Separate upstream/install tests and every selected build must pass the aggregate merge check. Selected skips, cancellation or failed planning cannot pass. Forks build/test local carriers without publishing.

New pushes cancel older runs of the same PR. Different PRs run concurrently. Main pushes keep their runs; each target's publication uses a shared concurrency group with cancellation disabled and queue: max (up to 100 waiting jobs). After a force-push, planning first fetches the previous commit by object ID; if GitHub no longer retains it, planning selects all packages conservatively. Explicit comparison and PR errors still fail. Planning needs no registry credentials: recipe, base and prerequisite changes must increase the version or pkgrel against the Git base. Renovate base updates run make revise-base. Missing bumps fail before builds. Manual full-matrix dispatch validates candidates without publishing releases.

Main performs authenticated collision checks and fails on changed bytes under an existing revision. Bootstrap denial is deferred until after candidate publication; final promotion still fails closed. Identical manifest retries pass. A released child can use only published parent revision digests; PR candidates stay disposable. Version aliases advance to the highest published revision.

Trust and credentials

Every workflow starts with permissions: {} and grants permissions per job. Every checkout disables credential persistence. Pass secrets individually. Package workflows receive only registry secrets, never GH_APP_PRIV_KEY. Fork code never receives publishing secrets or uses the trusted self-hosted runner.

Use GH_APP_ID and GH_APP_PRIV_KEY for the existing lbops-renovate app. GHCR normally uses GITHUB_TOKEN; other registries use configured Docker helpers or REGISTRY_USER/REGISTRY_PASSWORD. Interactive local commands hide secret input. Missing noninteractive auth fails; credentials never enter arguments/logs.

Pin action commits, tool/base image digests and an exact Go toolchain. Use bundled BuildKit frontend features without a # syntax= image. Native subprocesses need bounded contexts and process-group cancellation; downloads need limits/timeouts. Source signatures or independently authenticated checksums are preferred. Trust-on-first-use updates need human review and cannot auto-merge.

Cleanup

Carriers publish as ghcr.io/<owner>/<package>. Cleanup requires the repository owner and verifies each package/repository association before touching its tags. Run only trusted main cleanup code for closed PRs. Match exact PR tag prefixes, wait for publishers, paginate and recheck candidates before deleting. Protect release/shared versions and reachable index children. Dry-run is local default. GHCR cleanup does not imply DockerHub/GAR GC. Never promote disposable digests outside their publishing workflow. Main candidates include run ID/attempt; successful-main cleanup checks workflow identity and protects released/reachable/shared digests. Failed-run candidates are retained; use a full rerun until partial manifest retry/recovery has been verified live. Active cache-main-* tags are reused per target and architecture. Daily cache collection quarantines only unreachable OCI manifests whose config media type is application/vnd.buildkit.cacheconfig.v0. A dated retirement tag must remain for seven days before deletion; fresh tags/identity are rechecked. Application images, unknown untagged artifacts, indexes and referenced/shared cache versions are protected. make registry-clean-cache previews this; DRY_RUN=0 applies it.

Workflow lint

make dev-lint runs pinned actionlint. Version 1.7.12 does not yet recognize GitHub's documented queue: max; use -ignore 'unexpected key "queue" for "concurrency" section' for that one compatibility error. Do not suppress unrelated findings. make dev-generate derives actionlint's additional versioned Ubuntu, Windows and macOS hosted labels from the workflows, so newer runner releases do not require a second manual version list. Custom runner labels still require explicit review.

Renovate updates ci/push.yml.template as valid GitHub Actions YAML and runs make dev-generate afterward. The generated push.yml is excluded from direct dependency extraction. Commit the updated template, generated workflow and generated .github/actionlint.yaml together.

The installation regression is a separate required job (make dev-test-install, 25-minute timeout). Package builds depend only on planning and their parent builds; Alpine mirror availability during this regression cannot delay them. The fixture uses any catalogued Alpine 3.24 base, independent of Nginx support.

Weekly validation schedules 32 architecture builds plus their tests, including multi-hour Java compilations. Budget runner hours, disk and concurrency for this full matrix; separate concurrency groups avoid serialization, but runner capacity is still shared with releases. Use dedicated validation runners if shared runner capacity becomes a bottleneck. Review actual Actions usage before expanding the matrix.

A failed scheduled run opens a repository issue linking the run, through validation-alert.yml. The maintaining team must watch repository Issues (custom notifications) or route those issues through its existing team integration; cron-owner email is not the alert channel. This workflow uses no PR checkout or registry credentials. GitHub delivery and issue notifications still need live verification before relying on this route.

GitHub can delay scheduled runs and disables public-repository schedules after 60 days without repository activity. Monitor the most recent scheduled run from outside GitHub. An external scheduler can dispatch push.yml with all=true using gh workflow run push.yml --ref main -f all=true; do not use keepalive commits. Manual full-matrix failures need the same operational review as weekly failures. A GitHub-only alert cannot detect a schedule that never starts.

See GitHub’s schedule limits and notification routing.

Go's full upstream suite runs with three explicit environment exclusions: TestTSAN requires ASLR controls unavailable in the build container (also excluded by Alpine's Go recipe). On AMD64 only, TestSigStackSwapping and TestSigaltstack abort because their C fixtures allocate musl's fixed SIGSTKSZ, below the kernel minimum on some GitHub runners. The same runner limitation is documented in the kernel signal-stack fix. These are exact test-name exclusions, not package/suite skips; ARM64 keeps the signal-stack tests. Review them when upstream fixtures or the runner environment change. Go's dist runner clears GOFLAGS. Only the sanitizer package runs separately with go test -skip; runtime and cgo linker variants remain in dist. The disposable AMD64 test stage inserts exact t.Skip calls in the two signal-stack fixtures so every linker variant honors them. Published sources and compiler output remain unchanged. Tests disable inherited Go flags/configuration, preventing trimmed source paths from breaking traceback assertions.

Ruby uses Alpine aports compiler settings, libucontext, and its checksum-pinned Linux/musl main-stack detection patch. Bundled gems remain upstream-provided. Ruby suites install Bash and timezone data. Both bootstrap and full unit suites use normal runtime stack defaults. Ruby 3.3/3.4 exclude only test_dlopen_linker_script_group_linux: its expected libc.so name differs from musl's ld-musl-<arch>.so. Other Fiddle tests and all timezone, recursion and crash-report tests remain enabled.

PHP's per-test timeout is 60 seconds. Its XML attribute memory-exhaustion test takes about 15 seconds on native ARM64; the former ten-second limit timed out before the expected memory-limit error. Keep the assertion enabled. All three PHP lines include upstream's INI-test fix: the no-config subprocess must not inherit make test's temporary INI-file argument.

Python exclusion lists apply to individual test names via upstream's --ignorefile; no module is disabled. Python 3.12/3.13 lists cover musl locale/coercion differences, loader error text and invalid-FD pathconf. Python 3.12 additionally excludes the signed NODEV value and a test-only use-after-close: _testcapi.run_fileexflags calls fileno() after fclose() (upstream fixture). The freeze test keeps running with a ten-minute timeout to rebuild CPython. Python 3.14 excludes only deleted-libpython remote execution: the interpreter cannot be found after unlinking its mapped library in this build container; other remote-execution tests remain enabled. This limitation remains visible in the roadmap pending upstream/container resolution. Review all exclusions when upstream or musl changes; they are compatibility limits, not full upstream passes.

Renovate runs its own pinned container on GitHub-hosted Ubuntu, including trusted Go updater commands. Optional repository variable RENOVATE_RUNNER selects a dedicated runner label; it must provide Docker and allow the shared Go updater.

The main package workflow builds docs only after every build/test job completes and the required checks pass. It refreshes registry-backed install tabs from the same commit and published manifests, then deploys Pages. There is no independent docs workflow that can race package publication. PRs never deploy docs. Run make docs-build locally for strict documentation validation. Both docs jobs explicitly tolerate skipped package ancestors and require their immediate gate to succeed; an unchanged target must not suppress Pages deployment. make dev-test-install executes the same named installer consumers use against an older-than-Alpine fixture and checks its carrier-only dependency.