Purpose-built for Cloud Bigtable schema lifecycle management. Versioned YAML scripts manage tables, column families, GC policies, and app profiles — with distributed locking, immutable audit log, GC policy advisor, hotspot detection, column registry, and multi-instance pipelines. Enterprise tier adds contract checks, live schema analysis, instance diff, capacity projections, and CI/CD dashboards. 23 CLI commands. No JVM, no XML.
Every deploy follows a deterministic 8-step sequence. Scripts run in version order. Nothing runs twice unless you change it.
Connects to the target Bigtable instance and verifies it is reachable. Exits immediately with a clear error if the instance or project is missing.
Creates bigtablechange_history, bigtablechange_lock, and bigtablechange_audit if they don't exist. Zero manual setup required.
Writes an advisory lock row using the instance's own Bigtable API. Identifies the holder by hostname and run UUID. Blocks concurrent deploys.
Reads all rows in the history table and builds the applied-version set using latest-row-per-script logic — correctly handles rollback-then-redeploy cycles.
Applies pending versioned scripts in ascending semver order. Already-applied versions are automatically skipped — safe to re-run at any time.
Repeatable scripts rerun when their MD5 checksum changes. Always scripts run unconditionally — for GC policy syncs and permission refreshes.
A SUCCESS or FAILED row is written to history for every script. Every deploy event is appended to the immutable audit log with run_id, timestamp, and operator.
The deploy lock row is deleted. Slack and webhook notifications fire on configurable events: deploy_success, deploy_failed, script_failed, rollback_success.
Scripts declare the desired state of tables, column families, GC policies, and app profiles. The filename carries the version — no XML changelogs, no separate config.
Four script types give you fine-grained control over when and how often scripts run. The type is encoded in the filename prefix — no extra config needed.
Runs once in ascending semver order. Never re-runs. Creates tables, adds column families, declares app profiles.
Paired rollback for a versioned script. Only runs on rollback. Deletes profiles first (reverse order), then tables — idempotent.
Reruns on deploy when its MD5 checksum changes. Ideal for keeping GC policies in sync with policy-as-code definitions.
Executes on every deploy unconditionally. For seed data, app profile refreshes, and operations that must always be current.
Deploy, rollback, validate, status, repair, audit, and baseline — the foundational layer for schema lifecycle management.
--tag fences for incremental releases. Records every script in history and audit log.Beyond the core lifecycle, bigtablechange ships nine purpose-built commands for drift detection, GC policy analysis, script scaffolding, pipeline orchestration, reporting, verification, hotspot analysis, and column registry governance.
table, profile, gc, mixed. Writes correct YAML structure and # tag: header.--exit-code as a CI gate to fail if migrations are pending.Policy-as-code governance, live schema health scoring, instance comparison, capacity projections, monitoring snapshots, synthetic data seeding, and CI/CD analytics dashboards.
bigtablechange_monitor_snapshots for capacity projections.--seed.A single profile can target multiple Bigtable instances with independent migration paths. Every command — deploy, rollback, status, drift, analyse, monitor — runs against each instance in sequence automatically.
Form A (original): instance_id: dsi001 — single instance, unchanged behaviour.
Form B (new): instance_id as a mapping — each key is an instance ID with its own cluster_id and root_folder. Each instance has an independent history table and tracks applied versions separately.
You can be at V1.3.0 on the serving instance and V1.0.0 on the analytics instance simultaneously. The pipeline respects multi-instance profiles — each stage deploys all instances in that profile.
--ignore-profiles default (Bigtable always creates a default profile before any migration runs) or set drift.ignore_profiles in bigtablechange.yml.
bigtablechange compares migration history against the live Bigtable instance. Nine drift categories cover tables, families, GC policies, app profiles, and column registry. Pair with the GC policy advisor for proactive analysis.
Drift detection builds the expected state from all currently-applied migrations, then compares against live instance data. Internal bigtablechange tables are always excluded from detection. Exits 1 on ERROR-severity drift.
The advisor analyses every declared column family for common GC policy mistakes. Static analysis runs entirely offline. Live analysis cross-references write RPS and storage bytes from Cloud Monitoring to upgrade findings to actionable levels.
| Rule | Level | What it catches |
|---|---|---|
| NO_GC_POLICY | WARN | No GC policy — cells retained indefinitely |
| HIGH_MAX_VERSIONS | SUGGEST | max_versions > 10 is unusually high |
| LONG_MAX_AGE | SUGGEST | max_age_days > 365 — verify intentional |
| TRIVIAL_UNION | SUGGEST | union with fewer than 2 rules is pointless |
| NO_GC_HIGH_WRITE | ACTION | No GC + write rate > 100 req/s (live) |
| NO_GC_MODERATE_WRITE | WARN | No GC + write rate > 10 req/s or storage > 1GB |
| HIGH_VERSIONS_LARGE | WARN | max_versions > 3 on table > 10GB (live) |
bigtablechange hotspot is a fully offline row key pattern linter. It analyses table names in your migration scripts for naming patterns associated with write hotspots — before a single row is ever written.
| Finding | Level | Triggered by | Risk |
|---|---|---|---|
| SEQUENTIAL_TIMESTAMP | WARN | events, logs, metrics, audit, traces… | All writes land on last tablet |
| SEQUENTIAL_ID | WARN | orders, invoices, transactions, payments… | Sequential keys create write hotspot at tail |
| USER_PREFIX | SUGGEST | user_profiles, accounts, members… | High-traffic users concentrate load on one tablet |
| LOW_CARDINALITY_PREFIX | SUGGEST | status, type, category, region… | Low-cardinality prefix partitions data unevenly |
| NO_ROW_KEY_COMMENT | SUGGEST | Any table with no # row_key: comment | Undocumented row key makes review impossible |
Add a # row_key: comment to any YAML script to document the intended key design and suppress the NO_ROW_KEY_COMMENT finding. The comment serves as living documentation for your team.
Every finding includes a concrete recommendation with example row key patterns that distribute load evenly. The --level flag filters output to a minimum severity — use --level WARN in CI to catch high-risk patterns while suppressing informational suggestions.
No external dependencies. The history, lock, and audit tables live in the same Bigtable instance as the data being migrated — no Postgres, no Redis, no Cloud Spanner required.
bigtablechange_history
Row key: script_path#installed_on_iso
h:script — relative pathh:version — semver stringh:checksum — MD5 of YAMLh:status — SUCCESS / FAILED / ROLLED_BACKh:execution_ms — wall-clock timeh:installed_by — operator hostnameh:tag — deploy fence tagh:run_id — UUID linking all rows per runbigtablechange_lock
Row key: global
The lock row contains the holder's identity (hostname:run_id_prefix), acquisition timestamp, and full run UUID.
On acquire, bigtablechange reads the row and checks if the run_id column belongs to the current process. If not, LockError is raised with the holder's identity.
repair --release-lock performs an unconditional delete of the lock row to recover from stuck locks left by dead processes.
Handles rollback-then-redeploy
The history table is append-only. bigtablechange reads all rows per script and examines only the most recent row to determine current state:
Deploy to multiple Bigtable instances in a defined sequence — dev → staging → prod — with per-stage confirmation gates, tag fences, and automatic rollback of completed stages on failure.
Deploys automatically. No confirmation gate. First stage validates the scripts against a real instance before any human review is needed.
pause_before: true — operator must press Enter to continue. Per-stage tag fence can restrict to a subset of pending scripts.
pause_before: true — second confirmation gate before production. on_failure: rollback reverses all completed stages in reverse order if this stage fails.
Configure on_failure per pipeline: rollback reverses completed stages in reverse order, stop halts and leaves completed stages in place, continue logs the failure and proceeds.
A complete GitHub Actions pipeline with Workload Identity Federation, pre-deploy analysis, post-deploy verification, and diff-based PR gates.
Offline checks on every PR. YAML parse, naming, GC policy structure, row key risk analysis, and column registry drift detection. Zero network calls.
auto on PRPolicy-as-code gate against the live instance. GC mandates, naming conventions, PII retention. GitHub PR annotations. Exits 2 on contract failures — blocks merge.
auto on PRCompare staging schema against prod before promoting. Exits 2 on BREAKING GC policy mismatch. Show pending scripts without executing via diff.
Deploy with Workload Identity. Follow with verify to confirm every resource from the just-applied scripts actually exists in the live instance.
Post-deploy drift check confirms the live instance matches history. Monitor snapshot captures health metrics for capacity projections. Dashboard updated.
environment gateThe only tool with a Bigtable-native history table, YAML-based declarative scripts, GC policy advisor, row key hotspot detection, column registry, policy-as-code contracts, live schema health scoring, instance diff, capacity projections, and multi-instance pipeline — all 23 commands without a JVM or external database.
| Capability | bigtablechange | Terraform | Custom Scripts | Manual gcloud |
|---|---|---|---|---|
| Versioned migration history | ✓ In Bigtable | — | ⚠ DIY | — |
| Rollback with undo scripts | ✓ | ⚠ destroy/apply | ⚠ DIY | — |
| Distributed deploy lock | ✓ Advisory | — | — | — |
| Immutable audit log | ✓ | — | — | — |
| GC policy management | ✓ YAML + advisor | ✓ HCL | ⚠ DIY | ✓ |
| GC policy advisor | ✓ + live metrics | — | — | — |
| App profile lifecycle | ✓ | ✓ | ⚠ DIY | ✓ |
| Schema drift detection | ✓ 9 categories | ✓ plan | — | — |
| Row key hotspot linting | ✓ Offline | — | — | — |
| Column registry & PII audit | ✓ Opt-in | — | — | — |
| Multi-instance pipeline | ✓ | ⚠ modules | — | — |
| Multi-instance profiles | ✓ Form B config | ⚠ workspaces | — | — |
| Post-deploy verification | ✓ verify | — | — | — |
| Policy-as-code contracts | ✓ Enterprise | — | — | — |
| Live schema health scoring | ✓ 0–100 / table | — | — | — |
| Instance diff (schema compare) | ✓ BREAKING/WARN | ✓ plan | — | — |
| Capacity projections (OLS) | ✓ Enterprise | — | — | — |
| Test data seeder | ✓ Schema-aware | — | — | — |
| CI/CD analytics dashboard | ✓ HTML, no server | — | — | — |
| Incremental tag fencing | ✓ | — | — | — |
| No JVM / no external DB | ✓ Python only | ✓ Go binary | ⚠ varies | ✓ |
Python 3.8+. Authentication via Application Default Credentials, service account key, or the Bigtable emulator for local development.
pip or from package
bigtablechange.yml
V1.0.0__initial_tables.yaml
validate → advise → deploy → verify