Porter Valley Technology CompanyDiscuss your project

Forge automation

How Forge looks after itself

Forge is built so that nothing is set up by hand. Machines, networks and settings are written down as code, checked, and applied by pipelines - so a change is a file, a record and a report, not a memory in someone's head. The few exceptions are kept in a register, and we are working them down.

Three tiers

What is written down, and with what

  1. Infrastructure

    The machines, the network, the DNS and the storage. Each one is a file, so a lost machine is rebuilt from the files.

    OpenTofu · Proxmox · MikroTik

    170+ resources defined in code

  2. Configuration

    What runs on each machine and how it is set. Written as playbooks: step-by-step instructions the platform follows exactly, every time.

    Ansible

    82 playbooks

  3. Delivery and testing

    Every change goes through the same pipeline - linted, tested and planned before it is applied. Anything beyond routine waits for a person to approve it.

    GitLab CI · pytest · ansible-lint · promtool

    1,900+ automated tests

Making a change

What happens when we change something

A change is proposed in Git. The platform checks it and works out exactly what it will touch. Anything beyond routine waits for a person to approve it. Then the platform applies it and writes a report of what actually happened. The same steps, every time.

An illustration of the steps our pipelines run, with example figures. Real runs take a few minutes.

networks · example changepassed
  1. Change proposedcommit in Git
  2. Lint and testsall checks pass
  3. Planwhat will change
  4. Approvalapproved by a person
  5. ApplyOpenTofu · Ansible
  6. Report writtenposted to the ticket

Report

resources changed
3
destroyed
0
approval
1
start to finish
2m 14s

Detection and diagnosis

When something breaks that we have not seen before

The platform watches itself, and alerts a person as soon as a check fails. We are building the next step - when an alert fires, the platform gathers the evidence (logs, recent changes, certificate status) and compares it with the problems it already knows how to fix.

If it recognises the problem, it will fix it. If it does not, it calls a person. Not everything will be automatic, and we would rather say so. Once a person has fixed something new, the fix is written down as a playbook, so next time is quicker.

The example shows how this is designed to work. Alerting and playbooks are live today; automatic diagnosis is being built.

observability · exampleLearn
  1. Detect
  2. Diagnose
  3. Act
  4. Learn
  1. checks passing
  2. error: edge returning 502 for one site
  3. gathering: logs, last 3 changes, certificate status
  4. compared with known problems - no match
  5. alerted a person
  6. cause found: upstream health-check path changed
  7. fixed - checks passing
  8. written down as a playbook - next time is quicker
to fix
16 min
alerted
1 person
new playbook
+1