Skip to content

About

A modern Python application for backing up GitHub repositories and metadata using GitHub App authentication

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

GitHub Backup App

A modern Python application for backing up GitHub repositories and metadata using GitHub App authentication. It creates local backups organized by account: git repositories as mirrors ready to push elsewhere or restore from, plus JSON metadata and release assets kept as a plain dump (there is no tool to restore those to GitHub).

The intended workflow: create one public GitHub App (a private App can only be installed on the account that owns it), install it on every org and user account you want to back up, then run nightly, naming those accounts on the command line (see GitHub App Setup). The tool discovers all installations automatically and backs up every accessible repository.

This project is a fork of python-backup with a different focus: multi-installation GitHub App backups, automatic incremental runs, and much faster repeat backups on large estates.

What This Tool Does

  • Git repositories — bare mirrors of all branches, tags and pull-request heads; deleted or force-pushed refs are archived, never lost; fetch skipped when no ref changed on GitHub
  • JSON metadata — issues, pull requests, comments, milestones, labels, hooks, releases
  • Organized storage — {output}/{owner}/repositories/{repo}/… mirrors GitHub's structure
  • GitHub App auth — one app, many installations; no personal access tokens or SSH keys to rotate

Key features

Feature Description
Multi-installation discovery One run backs up every org/user where the app is installed; filter with positional args (jschule myorg)
Automatic incremental No -i flag and no separate state files — skip decisions come from the backup data on disk (repo.json, per-item JSON)
Git skip One git ls-remote per repository; git fetch is skipped when every ref equals the backup
Per-item metadata skip Unchanged issues/PRs/milestones skip comment/event/commit fetches and file rewrites
GraphQL metadata bundle One paginated query per repo fetches issues, PRs, milestones, labels, and releases together
HTTP keep-alive Shared requests.Session reuses TLS connections across API calls
status.json monitoring Machine-readable run summary with last_success_at, skip counters, and exit codes for cron/NAS alerting
Resilient runs Per-repository error isolation; Ctrl-C saves partial progress (exit 130)

Comparison with python-backup

python-backup (PyPI: github-backup, v0.62+) is what this fork started from. It remains excellent for PAT-based backups with many optional resources (gists, starred repos, discussions, security advisories, attachments, …).

What python-backup does not do (and why repeat backups here are much faster):

python-backup github-backup-app
Git fetch on repeat runs Always runs git fetch when a clone exists Skips fetch when git ls-remote shows no changed ref
Metadata list API calls Separate REST list call per resource (issues ×2 states, pulls ×2, milestones, labels, releases, …) One GraphQL query bundles issues, PRs, milestones, labels, releases
HTTP connections urllib — new connection per request requests.Session with keep-alive (~3× lower per-call latency)
Incremental model Opt-in -i flag; last_update checkpoint files; API since= time filter Always on; compares stored JSON timestamps; no checkpoint files
Per-item skip Re-fetches and rewrites all items since the resource checkpoint Skips unchanged items entirely (comments/events/commits not re-fetched)
GraphQL usage Discussions only (REST has no discussions API) Bulk metadata (issues, PRs, milestones, labels, releases)
Run monitoring None status.json with last_success_at and skip statistics
Multi-installation App Single installation context Auto-discovers all app installations

python-backup still has features this fork does not yet support like gists, starred repos, discussions, security advisories, user-attachments, more granular flags. Or will not support like --prefer-ssh.

Observed performance

Real runs on the same backup directory (incremental, --all):

Scenario Duration Notes
5 repos — REST baseline ~65s Git skip only
5 repos — GraphQL + Session ~18s ~3.7× faster
111 repos (5 installations) — GraphQL + Session ~381s 110 git skips, 2223 issues + 1661 PRs skipped unchanged

On a quiet estate where most repos haven't changed, the dominant cost shifts from git fetches and metadata listing to wiki probes and hooks (still REST-only). These numbers were measured with the earlier pushed_at-based git skip, which needed no network call for unchanged repositories; the current ref comparison adds one git ls-remote round trip per repository and wiki.

Quick Start

Docker Quick Start (Recommended)

The fastest way to get started is with Docker:

# 1. Create GitHub App
mkdir -p ./creds
docker run --rm -it \
  --entrypoint github-backup-create-app \
  -u $(id -u):$(id -g) \
  -p 3000:3000 \
  -v "$(pwd)/creds:/creds" \
  ghcr.io/schlomo/github-backup-app:latest \
  --host 0.0.0.0 \
  /creds

This will:

  1. Start a web server on port 3000
  2. You'll need to open your browser to http://localhost:3000 to access the app creation interface
  3. Guide you through creating a GitHub App
  4. Save credentials to the ./creds directory
# 2. Run backup
mkdir -p ./backup
docker run --rm -it \
  -u $(id -u):$(id -g) \
  -v "$(pwd)/backup:/data" \
  -v "$(pwd)/creds:/creds:ro" \
  ghcr.io/schlomo/github-backup-app:latest \
  --app-id $(cat ./creds/*-app-id.txt) \
  --private-key /creds/$(ls ./creds/*-private-key.pem | head -1 | xargs basename) \
  --all \
  --output-directory /data

Installation

The Docker image is the primary way to run it. The package is also on PyPI (requires Python 3.14 and git >= 2.31):

uv tool install github-backup-app
# or
pipx install github-backup-app

Manual Installation with uv from GitHub

You can install it directly from the GitHub repository using uv:

# Install from GitHub repository
uv tool install github-backup-app --source https://github.com/schlomo/github-backup-app

# Or install in a virtual environment
uv venv
source .venv/bin/activate
uv pip install https://github.com/schlomo/github-backup-app

1. Create a GitHub App

Option A: Automated Creation (Recommended)

Use the provided automation script to create a GitHub App with the correct permissions. Choose Public App to back up more than one account (see the note under GitHub App Setup):

source .venv/bin/activate
github-backup-create-app .

The script will:

  1. Start a local web server with an HTML interface
  2. Open your browser to the app creation interface
  3. Guide you through a 3-step process to configure your app
  4. Automatically handle the GitHub App creation and callback
  5. Exchange the temporary code for permanent credentials
  6. Save all credentials (App ID, private key, client secret) securely
  7. Provide installation instructions

Take note of the App ID and private key. You will need them to run a backup.

Option B: Manual Creation

  1. Go to your GitHub organization settings https://github.com/organizations/YOUR_ORG/settings/apps or your user settings https://github.com/settings/apps
  2. Click "New GitHub App"; under "Where can this GitHub App be installed?" choose "Any account" (public) to back up more than one account, or "Only on this account" (private) for just the account that owns the App (see the note under GitHub App Setup)
  3. Configure permissions (see GitHub App Setup below) and click "Save"
  4. Note the App ID
  5. Generate and download the private key (PEM file)
  6. Install the app on your organization or user account

Take note of the App ID and private key. You will need them to run a backup.

2. Dry Run (See What Would Be Backed Up)

github-backup \
  --app-id YOUR_APP_ID \
  --private-key /path/to/your-app.pem \
  --dry-run

3. Run a Backup

# Basic backup of all repositories from all installations
github-backup \
  --app-id YOUR_APP_ID \
  --private-key /path/to/your-app.pem \
  --all \
  --output-directory ./backup

# Backup specific users/organizations only (using positional arguments);
# always do this with a public App, which anyone can install
github-backup \
  --app-id YOUR_APP_ID \
  --private-key /path/to/your-app.pem \
  --all \
  --output-directory ./backup \
  myorg myuser

GitHub App Setup

NOTE: You can choose between Public and Private GitHub Apps. Public GitHub Apps are visible to the public and can be installed by anyone. Private GitHub Apps are only visible to the organization or user account that owns them and can only be installed by that organization or user account. If you choose Public GitHub Apps, you need to be careful with the organization filtering to avoid backing up unintended orgs as anybody can install your app. If you don't choose an organization filtering, the app will backup all orgs and users it has access to.

Required Permissions

Repository permissions (Read access):

  • Contents
  • Issues
  • Metadata
  • Pull requests
  • Repository hooks

Organization permissions (Read access):

  • Members

Installation

  1. Install the app on your organization and/or user account
  2. Choose "All repositories" for comprehensive access

Command Line Options

github-backup --help

Key options:

  • --app-id: Your GitHub App ID
  • --private-key: Path to your GitHub App private key file
  • --output-directory: Where to store the backup
  • --all: Include nearly everything in backup
  • --dry-run: Show what would be backed up without doing it

Output Structure

Backups are organized as follows:

backup/
├── status.json                      # Run status for monitoring (see below)
├── organization1/
│   └── repositories/
│       ├── repo1/
│       │   ├── repo.json            # Repository metadata (also drives incremental skip)
│       │   ├── repository/          # Bare git mirror (see "Git repositories")
│       │   ├── wiki/                # Bare git mirror of the wiki (if any)
│       │   ├── issues/              # JSON files
│       │   ├── pulls/               # JSON files
│       │   └── milestones/          # JSON files
│       └── repo2/
└── organization2/
    └── repositories/

Monitoring (status.json)

At the end of every run, a status.json file is written (atomically) to the root of the output directory so you can monitor backups from cron/CI/NAS. It always contains at least a status and a finished_at field:

{
  "status": "success",
  "started_at": "2026-06-30T08:45:10.000000+00:00",
  "finished_at": "2026-06-30T08:49:58.755971+00:00",
  "duration_seconds": 288.756,
  "last_success_at": "2026-06-30T08:49:58.755971+00:00",
  "tool": "github-backup-app",
  "url": "https://github.com/schlomo/github-backup-app",
  "version": "0.3.0",
  "python_version": "3.14.6",
  "summary": {
    "installations": 5,
    "accounts": ["org1", "org2"],
    "repositories_total": 42,
    "repositories_succeeded": 42,
    "repositories_failed": 0,
    "repositories_git_skipped_unchanged": 110,
    "issues_skipped_unchanged": 2223,
    "pulls_skipped_unchanged": 1661,
    "milestones_skipped_unchanged": 24,
    "git_refs_archived": 3,
    "recovered_repositories": [],
    "lfs_warnings": [],
    "failed_repositories": []
  }
}

The status field is one of:

  • success — all repositories backed up successfully (exit code 0)
  • partial — the run completed but some repositories failed; see summary.failed_repositories (exit code 2)
  • failed — the run aborted with a fatal error; see the error field (exit code 1)
  • interrupted — the run was stopped with Ctrl-C; partial progress was saved (exit code 130)

Recommended monitoring: alert when last_success_at is older than your backup interval allows (e.g. > 48h). This field carries forward across failed runs, so it stays accurate even if the tool has been failing for a long time — catching exactly the kind of silent, long-running breakage that a simple "did it run?" check would miss. The non-zero exit codes above also let cron/CI detect failures directly.

Git repositories

Each repository (and wiki) is stored as a bare mirror under repository/ (wiki/). A mirror contains:

Ref Content
refs/heads/* every branch, exactly as on GitHub
refs/tags/* every tag
refs/pull/<n>/head the head of every pull request, including PRs from forks whose commits exist nowhere else in the repository
any other ref on GitHub e.g. refs/notes/*, or refs/remotes/* and refs/stash left by a git push --mirror from a working copy, or namespaces of tools such as git-bug (refs/bugs/*)
refs/backup/<timestamp>/<ref> refs that GitHub deleted or force-moved since the previous run, preserved at their old commit

Only GitHub's synthetic refs/pull/<n>/merge commits are not backed up. Refs that GitHub itself has under refs/backup/ cannot be mirrored (that prefix holds the archive); the log warns if a repository has any.

Nothing is silently destroyed. Every branch, PR head or other ref that disappeared on GitHub or moved in a way that drops commits (force-push), and every tag that changed at all, is preserved as refs/backup/<UTC timestamp>/<original ref>. The mirror's refs/heads and refs/tags therefore always equal GitHub's current state, while the rewritten history stays reachable (and is never garbage collected). Archive refs are never overwritten. status.json counts them in git_refs_archived. --no-prune is deprecated and has no effect.

This also holds when a run fails or is killed: before each fetch the current refs are written to a journal (backup-refs-journal inside the mirror), and whatever changed is archived after the fetch - successful or failed - or at the start of the next run. A first clone is built in repository.new-mirror/ and only moved into place once it succeeded.

A fresh copy is the fallback for any broken backup. If the local backup of a repository is unusable - git fsck finds missing or corrupt objects, a migration cannot be finished, or the directory exists but is not a git repository - it is moved aside as repository.broken-<timestamp>/ (never deleted) and a fresh copy is cloned from GitHub in the same run. The repository counts as backed up; status.json lists it under recovered_repositories with the reason and where the old copy went, and the log shows a warning. Inspect the old copy (its refs/backup/* archive is not carried over) and delete it when no longer needed. Only positive evidence of damage triggers this. Problems on the GitHub side (network, authentication, outages), ownership and permission problems, a repository format written by a newer git, or a killed process never do: they fail the repository and leave the backup alone. If the fresh clone fails too, the error names where the old copy went.

Housekeeping. Mirrors are packed with git gc --auto after each fetch (everything worth keeping is reachable from a ref, including the archive). Only one run at a time can use a backup directory (.github-backup.lock); a second run exits with code 3 and leaves status.json alone. Stalled transfers (below 1 KB/s for 10 minutes) are aborted, and git never follows HTTP redirects, so the token cannot be sent to another host.

No credentials on disk. The GitHub App installation token is passed to git as an HTTP header through GIT_CONFIG_* environment variables of each git process (requires git >= 2.31, checked at startup). It never appears in .git/config, on the command line (ps) or in logs; the stored remote URL is a plain https://github.com/<owner>/<repo>.git. Git LFS uses the same header for its API; LFS storage on other hosts does not receive it.

File ownership does not matter. git is always pointed at each mirror explicitly, so git's "dubious ownership" check does not apply and the backup works regardless of which user owns the files (e.g. NAS shares with a fixed owner). The user running the backup only needs read and write access; permission errors fail the repository, they never move a backup aside. git hooks are disabled for every git call, so files planted in a mirror's hooks/ directory never run. safe.bareRepository=explicit is supported.

Restore a repository into a new, empty GitHub repository:

git -C backup/<owner>/repositories/<repo>/repository push <new-remote-url> \
  'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'

Other refs (e.g. refs/notes/*) can be pushed the same way. Do not use git push --mirror: GitHub rejects pushes to refs/pull/*, and it would also publish the refs/backup/* archive. To look at the code, git clone backup/<owner>/repositories/<repo>/repository gives a normal working copy. To recover archived history, list it with git for-each-ref refs/backup and push the ones you need as branches.

Upgrading from 0.3.0 and earlier. Older versions stored a non-bare clone (branches only as refs/remotes/origin/*, with the access token in the remote URL) or, with --bare, a git clone --mirror. Both are converted in place on the next run, also for repositories whose fetch is skipped. --bare is now a no-op. The migration is designed to lose nothing:

  • No git object is copied, rewritten or deleted; the old .git directory becomes the mirror. Only the checked-out working tree (whose content is in git) is removed, and only after verifying that every branch was carried over.
  • Every remote-tracking branch becomes a real branch at the same commit.
  • Every commit or annotated tag that no ref points to any more - tips force-pushed away that only the old reflogs remember, branches and re-cut tags an earlier run deleted but git had not yet garbage collected - is rescued as refs/backup/<ts>-migration-unreferenced/<sha>.
  • A clone that fails git fsck is not converted; it is moved aside and replaced by a fresh copy (see "A fresh copy is the fallback" above). The commit-graph cache file is dropped before the check: older runs could leave it listing commits that git had already garbage collected, which makes fsck fail although no data is missing.
  • An old --bare mirror gets the same rescue of unreferenced commits and tags once, when it is converted.
  • Each step is idempotent: an interrupted run (Ctrl-C, container killed) resumes the migration on the next run (repository.mirror-migration/ is the in-progress state).
  • The token-bearing remote URL is replaced by a clean one.

Recommended procedure: take a filesystem snapshot of the backup directory (e.g. Btrfs/ZFS snapshot on a NAS), run once, check status.json for repositories_failed and recovered_repositories, then verify with grep -rl ghs_ <backup-dir> --include=config (expect no output). Do not roll back to 0.3.0 afterwards without its --bare flag: without it, 0.3.0 does not recognise the mirrors and silently stops backing up git content.

LFS (--lfs): git lfs fetch --all runs after every fetch that changed something, and on later runs until it has succeeded once; it covers all refs including the archive, and LFS objects are never pruned. Wikis are fetched without LFS. An LFS download that fails (e.g. objects GitHub no longer has) does not fail the repository: it is listed under lfs_warnings in status.json. git-lfs does not honour the reset of an extraHeader inherited from your global git config; do not configure one for the GitHub host.

Incremental backups

Incremental backups are automatic; there is no flag to enable them and no separate state file. Every skip decision is derived from the backup data already on disk, so the backup directory alone explains what was skipped and why.

Repository git and wiki — one git ls-remote lists every ref on GitHub (branches, tags, notes, pull request heads). If all of them equal the refs in the mirror, the (often dominant) git fetch is skipped. Unlike the repository's pushed_at, this also notices pull request heads updated from a fork, wiki edits, and branch deletions. Each repository's metadata is written to repo.json after a fully successful pass, so metadata stays current even when the git fetch is skipped.

Issues, pull requests and milestones — for each item, its stored <number>.json already records updated_at. If that is at least as new as the updated_at from the listing, the item is unchanged and its expensive sub-resources (issue/PR comments, events, commits) are not re-fetched and the file is not rewritten. An issue/PR updated_at reliably advances on comment, label and state activity, so this does not miss updates. The item listings are fetched in a single GraphQL query per repository (issues, pull requests, milestones, labels and releases together), which replaces several sequential REST list calls. Hooks still use REST (not available via GraphQL).

HTTP performance — all GitHub API traffic shares a requests.Session with keep-alive, so repeated calls to the same host reuse the TLS connection instead of paying a full handshake on every request.

Releases — release asset binaries are not re-downloaded when a file of the same name already exists on disk. Release metadata and asset listings come from the same GraphQL query as other repository metadata (no separate per-release assets REST call when GraphQL data is available).

Notes:

  • We deliberately never use the repository's own updated_at to skip issues/PRs, because it only reflects repository metadata changes, not issue/PR/comment activity — relying on it would risk silently missing data.
  • Use --force-full to ignore all stored timestamps and re-fetch everything (git content and every issue/pull/milestone). Deleting an item's JSON has the same effect for that item.
  • Interrupting a run with Ctrl-C stops cleanly; repositories finished so far are unchanged on the next run and are skipped quickly.
  • status.json reports how much was skipped via repositories_git_skipped_unchanged, issues_skipped_unchanged, pulls_skipped_unchanged and milestones_skipped_unchanged.

Development

Setup

# Clone the repository
git clone https://github.com/schlomo/github-backup-app.git
cd github-backup-app

# Set up development environment
./dev-setup.sh

# Or manually
uv sync --dev

Available Commands

First, activate the virtual environment (recommended for less typing):

source .venv/bin/activate

Then you can use the tools directly:

pytest -q tests/                      # Run tests (local git repos, no network)
flake8 github_backup/ tests/          # Run linting
black github_backup/ tests/           # Format code
python -c "import github_backup; print('Import successful')"  # Test import
github-backup --help                  # Show CLI help
uv build                              # Build package

Releases

CI/CD is defined in .github/workflows/ci-cd.yml. There are three jobs:

Job When it runs What it does
test-build Every pull request, every branch push that changes a build input (the paths list in the workflow), every tag push Lint (flake8), format check (black), tests (pytest), import/CLI smoke test, uv build + twine check → uploads dist/ artifact
docker Push to main that changes a build input, push of a v* tag, or a pull request Builds multi-arch (amd64 + arm64) image, pushes to ghcr.io/schlomo/github-backup-app and runs --help in it (pull requests from forks: build only)
release Push of a v* tag only Downloads the built wheel/sdist and publishes to PyPI via trusted publishing (GitHub environment pypi)

Docker image tags (docker/metadata-action):

  • latest — every push to main that changes the code or build (what the Quick Start uses)
  • main — branch ref tag on main pushes
  • X.Y.Z, X.Y, X — when you push an annotated semver tag like v0.2.0
  • pr-<number> — every push to a pull request from this repository, to try a change before merging (e.g. ghcr.io/schlomo/github-backup-app:pr-12)

Merging a code change to main updates ghcr.io/schlomo/github-backup-app:latest; pushes that only touch docs (for example backlog.md or docs/) skip CI; README.md is a build input and does not. No git tag is required for Docker.

Releases and PyPI — the PyPI job only runs for a version tag (v0.4.1, not 0.4.1):

  1. Bump the version in github_backup/__init__.py
  2. Merge to main (updates the :latest Docker image)
  3. Publish a GitHub release with a new tag vX.Y.Z on main (or push the tag)

That triggers the semver Docker tags and the PyPI upload. PyPI uses trusted publishing: GitHub's short-lived OIDC token is accepted for this repository, the workflow ci-cd.yml and the environment pypi; there is no API token secret. The publisher is configured on PyPI under the project's Publishing settings.

Requirements

  • Python 3.14+
  • Git 2.41+ (not sure exactly)
  • GitHub App with appropriate permissions

License

MIT License - see LICENSE.txt for details.

Sponsor

Sponsored by Tektit Consulting.

Acknowledgments

This project is based on the excellent work by Jose Diaz-Gonzalez in python-backup. Thank you for creating the foundation that made this derived work possible.

About

A modern Python application for backing up GitHub repositories and metadata using GitHub App authentication

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages