Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 4

[*.{yml,yaml,json,toml,html}]
indent_size = 2

[*.md]
trim_trailing_whitespace = false
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* text=auto eol=lf
14 changes: 6 additions & 8 deletions .github/workflows/final-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
continue-on-error: true
run: |
status=0
python -m pip install --disable-pip-version-check pytest > "$RUNNER_TEMP/install.log" 2>&1 || status=$?
python -m pip install --disable-pip-version-check 'pytest==8.4.2' > "$RUNNER_TEMP/install.log" 2>&1 || status=$?
tail -n 20 "$RUNNER_TEMP/install.log"
exit "$status"

Expand Down Expand Up @@ -91,20 +91,18 @@ jobs:
cat "$RUNNER_TEMP/config.log"
exit "$status"

# The repository only allows GitHub-owned actions pinned by full SHA.
# The organisation only runs GitHub-owned actions pinned by full SHA. The
# rule lives in tests/test_pinned_actions.py, which also covers the
# workflows the satellite template generates.
- name: Syntax and pinned actions
id: lint
if: steps.install.outcome == 'success'
continue-on-error: true
run: |
status=0
{
python -m compileall -q machine tests || status=1
unpinned=$(grep -nE '^\s*-?\s*uses:' .github/workflows/*.yml | grep -vE 'uses: actions/[a-z-]+@[0-9a-f]{40}( |$)' || true)
if [ -n "$unpinned" ]; then
echo "Not a GitHub-owned action pinned by SHA:"
echo "$unpinned"
status=1
fi
python -m pytest -q -p no:cacheprovider tests/test_pinned_actions.py || status=1
} > "$RUNNER_TEMP/lint.log" 2>&1
cat "$RUNNER_TEMP/lint.log"
exit "$status"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ jobs:
with:
python-version: "3.12"
- run: |
pip install pytest
pip install 'pytest==8.4.2'
python -m pytest -q
158 changes: 114 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,52 +2,75 @@

Repository management for the GetTechAPI organisation. Where
[TechEngine](https://github.com/GetTechAPI/TechEngine) processes the data,
TechMachine looks after the repositories that hold it.
TechMachine looks after the repositories that hold it. It does two things:
a health check over every repository, and a scaffolder for new satellite
data repositories.

## Health check

`machine/health.py` runs when a person starts it — the `health` workflow's
*Run workflow* button, or locally. For every repository in [`machine/repos.json`](machine/repos.json)
it reads the newest completed run of each active workflow on each tracked
branch, and it fetches the public endpoints the TechAPI homepage depends on.
## The organisation's repositories

Anything other than `success`, `skipped` or `neutral` is reported —
`cancelled` and `timed_out` included. That is deliberate: a job killed at
GitHub's six-hour limit reports itself as *cancelled*, and several automations
here failed that way for days without anyone noticing.
| Repository | What it is |
| --- | --- |
| [TechAPI](https://github.com/GetTechAPI/TechAPI) | The dataset and the site |
| [TechEngine](https://github.com/GetTechAPI/TechEngine) | The engine that processes the data; TechAPI pins it as a submodule |
| [game-catalog](https://github.com/GetTechAPI/game-catalog) | Satellite data repository: games |
| [cpu-engineering-samples](https://github.com/GetTechAPI/cpu-engineering-samples) | Satellite data repository: CPU engineering samples |
| [.github](https://github.com/GetTechAPI/.github) | The organisation profile and shared community files |
| TechMachine | This repository: health check and satellite scaffolder |

It queries each workflow separately rather than scanning "the last N runs",
because weekly jobs scroll out of any fixed window behind the daily ones — and
the weekly jobs are the ones that go quiet.
## Health check

It also reads each active workflow file and reports any that uses a secret
other than `GITHUB_TOKEN`. Automation here must act as `github-actions[bot]`,
never as a person: workflows running under a personal token are how a
maintainer's account got suspended. Disabling the workflow in the Actions
settings clears the finding; a non-token secret a repository genuinely needs
goes in that repository's `allowed_secrets` in `repos.json` — TechAPI's
label-triggered final check is the one place TechEngineBot's token is allowed.
`machine/health.py` runs when a person starts it: the `health` workflow's
*Run workflow* button, or locally. It reads [`machine/repos.json`](machine/repos.json)
and reports three kinds of problem.

1. **The latest completed run of each active workflow, per tracked branch.**
Anything other than `success`, `skipped` or `neutral` is reported,
`cancelled` and `timed_out` included. That is deliberate: a job killed at
GitHub's six-hour limit reports itself as *cancelled*, and several
automations here failed that way for days without anyone noticing. Each
workflow is queried separately rather than scanning "the last N runs",
because weekly jobs scroll out of any fixed window behind the daily ones,
and the weekly jobs are the ones that go quiet. Disabled workflows are
skipped, and so is GitHub's own *Dependabot Updates* workflow in a
repository that no longer has a `.github/dependabot.yml` (or `.yaml`).
2. **Secrets.** It reads each active workflow file on each tracked branch and
reports any that uses a secret other than `GITHUB_TOKEN` or the
repository's `allowed_secrets`. Automation here must act as
`github-actions[bot]`, never as a person: workflows running under a
personal token are how a maintainer's account got suspended. The one
allowed secret is `TECHENGINEBOT_TOKEN`, which the label-triggered final
check uses to post its comment. Disabling a workflow in the Actions
settings clears its finding.
3. **Endpoints.** It fetches each public URL listed under `endpoints` and
reports one that is unreachable, is not JSON, or lacks its `expect` field.
These are the files the TechAPI homepage depends on.

A repository that cannot be read is reported too, rather than passed over.

It only reads. The report goes to stdout and the run's step summary; it
writes nothing to GitHub, mentions nobody, and runs on no schedule — an
writes nothing to GitHub, mentions nobody, and runs on no schedule. An
account here was suspended over automation, and a report nobody asked for is
how that starts.
how that starts. The run itself always ends green: the report is the output.

```bash
python -m machine.health # print the report (GITHUB_TOKEN raises the rate limit)
python -m pytest -q
```

Adding a repository or endpoint is one entry in `machine/repos.json`.
Adding a repository or endpoint is one entry in `machine/repos.json`: a
`name`, the `branches` to watch and, where the final check runs,
`"allowed_secrets": ["TECHENGINEBOT_TOKEN"]`. A repository with no workflows
(the `.github` profile repository) is listed with its branch only.

## Final check

Adding the `final-check` label to a pull request asks
[TechEngineBot](https://github.com/TechEngineBot) for a last look, in place of
Copilot. The check job runs the tests (including a generated satellite
repository), validates `repos.json`, and checks syntax and that every action
is GitHub-owned and pinned by SHA. TechEngineBot then leaves **one** comment
with the verdict; adding the label again edits it.
[TechEngineBot](https://github.com/TechEngineBot) for a last look. The check
job runs the tests (including a generated satellite repository), validates
`repos.json`, and checks syntax and that every action is GitHub-owned and
pinned by full commit SHA, both in this repository's workflows and in the
ones the template generates (`tests/test_pinned_actions.py`). TechEngineBot
then leaves **one** comment with the verdict; adding the label again edits it.

The check job runs the PR's code with no secrets and a read-only token. Only
the comment job sees `TECHENGINEBOT_TOKEN`, and it posts nothing unless that is
Expand All @@ -57,27 +80,74 @@ as TechAPI's final check.

## Satellite repositories

Categories that do not belong in TechAPI live in their own repository (games:
[game-catalog](https://github.com/GetTechAPI/game-catalog)). The split
criterion is identity, not size — software and websites are tech data and
stay in TechAPI.
Categories that do not belong in TechAPI live in their own repository
([game-catalog](https://github.com/GetTechAPI/game-catalog),
[cpu-engineering-samples](https://github.com/GetTechAPI/cpu-engineering-samples)).
The split criterion is identity, not size: software and websites are tech
data and stay in TechAPI.

`machine/new_satellite.py` writes a new one from the layout game-catalog
proved out: a streaming validator, a site build that publishes
`summary.json` + `history.json` (never a listing of every record), CI, and
licences.
`machine/new_satellite.py` writes a new one from
[`machine/satellite_template/`](machine/satellite_template):

```bash
python -m machine.new_satellite --repo game-catalog --category game --title "Game catalog" --plural games --date-field release_date --range rating:0:5 --range metacritic:0:100 --out ../game-catalog
python -m machine.new_satellite --repo game-catalog --category game \
--title "Game catalog" --plural games --date-field release_date \
--range rating:0:5 --range metacritic:0:100 --out ../game-catalog
```

It only writes files. Creating the repository changes the organisation, so
that is left to a person; the remaining steps are printed at the end,
including adding `main` to the Pages environment's deployment branches —
without it every deploy fails and leaves no log.
### What it generates

```
.github/CODEOWNERS # * @Seungpyo1007
.github/pull_request_template.md
.github/workflows/validate-data.yml # validator + tests on PRs and pushes to develop/main
.github/workflows/deploy-pages.yml # builds and publishes the site from main
.github/workflows/final-check.yml # the label-triggered check and TechEngineBot's comment
.gitignore
DATA_LICENSE.md # CC BY-SA 4.0 for the records
LICENSE # MIT for the code
README.md
app/__init__.py
app/validate.py # required fields, slug, dates, ranges, duplicates
pyproject.toml
site/.nojekyll
site/build.py # summary.json + history.json, never a listing of every record
site/index.html
tests/test_validate.py
data/<category>/ # empty; records go in <bucket>/<slug>.json
```

The tests generate a repository and run *its* test suite and validator, so a
template change that breaks generated repos fails here.
The workflows follow the organisation's rules: GitHub-owned actions pinned
by full commit SHA, least-privilege `permissions:`, no schedules, nothing
that opens pull requests or issues, and no secret except
`TECHENGINEBOT_TOKEN` in the final check's comment job. There are no issue
templates; those are shared from the `.github` repository.

The tests generate a repository, compare its tree with the list above, and
run *its* test suite and validator, so a template change that breaks or
reshapes generated repositories fails here.

### Creating a satellite

The scaffolder only writes files. Everything else changes the organisation,
so it is left to a person; the same list is printed after generating.

1. Generate the files with the command above.
2. Create the repository: `gh repo create GetTechAPI/<repo> --public`.
3. Push the scaffold to `develop`, then `develop` to `main`.
4. Turn Pages on with the workflow as its source:
`gh api -X POST repos/GetTechAPI/<repo>/pages -f build_type=workflow`.
5. Add `main` to the Pages environment's deployment branches:
`gh api -X POST repos/GetTechAPI/<repo>/environments/github-pages/deployment-branch-policies -f name=main`.
Without it every deploy fails and leaves no log.
6. Add the repository to `machine/repos.json` here, with
`"allowed_secrets": ["TECHENGINEBOT_TOKEN"]`, and its `summary.json` URL
to `endpoints`.
7. Add it to `SATELLITES` in TechAPI's `site/src/scripts/techapi.js` so the
homepage counts it.
8. For the final check, create the `final-check` label and set the
`TECHENGINEBOT_TOKEN` secret to a fine-grained token of TechEngineBot with
*Pull requests: read and write* on that repository only.

## Branching

Expand Down
7 changes: 6 additions & 1 deletion machine/new_satellite.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,15 @@ def checklist(v: dict[str, str]) -> str:
3. gh api -X POST repos/{repo}/pages -f build_type=workflow
4. gh api -X POST repos/{repo}/environments/github-pages/deployment-branch-policies -f name=main
(skip this and every deploy fails with no log — game-catalog, 2026-09-17)
5. add {{"name": "{repo}", "branches": ["develop", "main"]}} to TechMachine machine/repos.json
5. add {{"name": "{repo}", "branches": ["develop", "main"],
"allowed_secrets": ["TECHENGINEBOT_TOKEN"]}} to TechMachine machine/repos.json
and the summary.json URL to its endpoints
6. add {{ key: "{v['plural']}", label: "{v['plural']}", base: "https://gettechapi.github.io/{v['repo']}/" }}
to SATELLITES in TechAPI site/src/scripts/techapi.js
7. for the final check: create the `final-check` label and set the
TECHENGINEBOT_TOKEN secret to a fine-grained token of TechEngineBot with
Pull requests: read and write on {repo} only. Without the secret the
verdict stays in the run summary and nothing is posted.
"""


Expand Down
16 changes: 16 additions & 0 deletions machine/repos.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,22 @@
"allowed_secrets": [
"TECHENGINEBOT_TOKEN"
]
},
{
"name": "GetTechAPI/.github",
"branches": [
"main"
]
},
{
"name": "GetTechAPI/TechMachine",
"branches": [
"develop",
"main"
],
"allowed_secrets": [
"TECHENGINEBOT_TOKEN"
]
}
],
"endpoints": [
Expand Down
2 changes: 2 additions & 0 deletions machine/satellite_template/.github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Default owners for everything in this repository.
* @Seungpyo1007
12 changes: 12 additions & 0 deletions machine/satellite_template/.github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!-- PR title follows Conventional Commits, e.g. data({{category}}): add <name>. Target `develop`; only a release PR targets `main`. -->

## What & why

<!-- What does this change and why? Use "Closes #123" for the issue it resolves. -->

## Checklist

- [ ] Targets `develop`
- [ ] `python -m app.validate` and `python -m pytest -q` pass locally
- [ ] Every new or changed record cites at least one public reference in `source_urls`
- [ ] Sidebar filled in by hand: assignees, labels, milestone, projects
9 changes: 5 additions & 4 deletions machine/satellite_template/.github/workflows/deploy-pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,25 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0 # history.json replays every data commit
- uses: actions/setup-python@v5
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Build summary + history
run: python site/build.py
- uses: actions/upload-pages-artifact@v3
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: site

deploy:
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1
Loading
Loading