# Datalogz Self-Hosted Capture

This package contains the Datalogz CLI for running a self-hosted metadata
capture.

## Files

- `datalogz`: CLI binary.
- `extensions/`: bundled DuckDB extensions, loaded at runtime (no download).
- `README.md`: this guide.

Run the commands below from this extracted directory.

## 1. Confirm the Binary

```bash
./datalogz version
```

## 2. Network Requirements

All connections are outbound HTTPS from the capture host; nothing listens for
inbound connections during normal use. Allow egress to:

- Your Tableau Server / Tableau Cloud host (the `--host` you configure).
- `login.microsoftonline.com`, `api.powerbi.com`, and
  `api.fabric.microsoft.com` for Power BI sources.
- Nothing else. Release packages bundle the DuckDB extensions (the `ducklake`
  extension and the `httpfs` crypto provider used for at-rest encryption) in
  the `extensions/` directory next to the binary, so no extension download
  happens at runtime. (Source builds without that directory fetch extensions
  from `extensions.duckdb.org` on first run, or use a pre-installed copy via
  `DATALOGZ_DUCKDB_EXTENSION_DIR`.)
- The presigned upload URL supplied by Datalogz, only when running
  `datalogz upload`.

## 3. Set Up Sources

```bash
./datalogz setup
```

Setup prompts for Tableau and Power BI connection details, stores them locally
under `./.data`, and can validate source API access immediately.

Run setup again to add more Tableau hosts/sites or more Power BI tenants.

Prompt-free setup is also available:

```bash
./datalogz setup powerbi --name powerbi-main --tenant-id TENANT --client-id CLIENT --client-secret SECRET
./datalogz setup tableau --name tableau-prod --host https://tableau.example.com --site default --token-name TOKEN_NAME --token-secret TOKEN_SECRET
```

You can review configured sources without showing secrets:

```bash
./datalogz sources list
./datalogz sources show --name SOURCE_NAME
./datalogz sources validate --name SOURCE_NAME
```

Remove a local source with:

```bash
./datalogz sources remove --name SOURCE_NAME
```

Large Power BI tenants can take longer to capture. The CLI writes completed
workspace scan batches as they finish and keeps local resume state, so an
interrupted run can continue without starting from the beginning.

## 4. Validate Setup

```bash
./datalogz preflight
./datalogz sources validate
```

`preflight` checks local runtime readiness. `sources validate` checks source
auth/API reachability and updates each source's validation status.

## 5. Run Capture

```bash
./datalogz run
```

The run writes local state, stored source setup, and warehouse files under
`./.data`.

To run only one configured source:

```bash
./datalogz run --source SOURCE_NAME
```

## 6. Inspect or Export Data

```bash
./datalogz query --sql "SELECT run_id, status, started_at, completed_at FROM datalogz_runs ORDER BY started_at DESC LIMIT 5"
./datalogz query --sql "SELECT platform, source_name, status, row_count, warning_count, error_count FROM datalogz_run_sources"
./datalogz query --sql "SELECT event_index, level, event_type, message FROM datalogz_run_events WHERE run_id = 'run-...' ORDER BY event_index"
./datalogz query --sql "SELECT COUNT(*) FROM warehouse.powerbi.groups"
./datalogz export --sql "SELECT * FROM warehouse.tableau.workbooks LIMIT 1000" --output workbooks.csv
```

Query/export are intended for post-run read-only inspection.
`query` prints a table by default. Add `--format json` when you need
machine-readable output.

Power BI captures include both admin endpoint tables such as `powerbi.reports`,
`powerbi.datasets`, and `powerbi.dashboards`, plus workspace scan-derived tables
such as `powerbi.workspace_reports`, `powerbi.workspace_datasets`,
`powerbi.workspace_dashboards`, `powerbi.workspace_dataflows`,
`powerbi.workspace_datasources`, and `powerbi.workspace_users`.

The simplest way to inspect the workspace after `./datalogz run` exits is the
built-in `query` command — it opens the workspace and its encryption key
automatically:

```bash
./datalogz query --sql "SELECT run_id, status, started_at, completed_at FROM datalogz_runs ORDER BY started_at DESC LIMIT 5"
./datalogz query --sql "SELECT COUNT(*) FROM warehouse.powerbi.groups"
```

By default the workspace is a single **encrypted** DuckDB file,
`.data/datalogz.db`, with its key in `.data/datalogz.key`. Because it is
encrypted, opening it directly with the DuckDB CLI requires supplying that key,
so `./datalogz query` is the recommended way to inspect it.

## 7. Package Data for Datalogz

After the capture finishes, create a bundle when Datalogz asks you to send the
captured metadata for analysis:

```bash
./datalogz bundle inspect
./datalogz bundle create --mask-emails --output datalogz-bundle.tgz
./datalogz bundle inspect datalogz-bundle.tgz
```

`bundle create` writes a fresh package workspace into the archive. It excludes
stored Tableau and Power BI source credentials. Email values are replaced
locally with deterministic tokens before the archive is written; this masking
is on by default (`--mask-emails=false` disables it).

## 8. Upload an Approved Bundle

Datalogz will provide an approved short-lived upload URL when a bundle should be
sent back for analysis.

```bash
./datalogz upload datalogz-bundle.tgz --url "https://..."
```

The upload command sends only the bundle file you name to the URL you provide.
It prints the uploaded file size and SHA-256 checksum after the upload
completes.

## Local State

This package is self-contained. Local Datalogz state lives under `./.data`. By
default it is a single encrypted DuckDB file:

- `.data/datalogz.db` — configured sources, stored source credentials, run
  history, checkpoints, and captured warehouse data, all in one file.
- `.data/datalogz.key` — the owner-only encryption key for that file. **Back it
  up together with the database**; without the key the workspace cannot be
  opened.

Deleting `./.data` removes all of the above. Tableau and Power BI remain the
source systems; setup can be rerun after downloading a fresh CLI package.

## Security Stance

The self-hosted package keeps source credentials and captured metadata local to
the host where you run it. It does not automatically upload the local
workspace, exports, bundles, or support evidence to Datalogz.

Treat the entire `./.data` directory as sensitive:

- `.data/datalogz.db` holds configured sources, stored source credentials, run
  history, checkpoints, and captured warehouse data. It is encrypted at rest by
  default. CLI commands such as `sources show` redact secrets, but direct
  database access (with the key) and backups include the stored auth material.
- `.data/datalogz.key` is the encryption key for that file — keep it owner-only;
  anyone holding both files can read everything.
- CSV, JSON, or Parquet files written by `./datalogz export` are outside the
  managed workspace and inherit the controls of their output location.

Run the CLI on an approved host with disk encryption and restricted filesystem
access. Protect `./.data` backups with the same controls as the live workspace,
and do not send `.data` or exported records to Datalogz unless your team has
approved the contents. For sensitive environments, avoid passing source secrets
through shared terminals or shell history; use your approved secret handling
process.

## Upgrading

To move to a new release, stop any running capture, then replace the package
contents (`datalogz` binary, `extensions/`, this README) with the new
package's files. Keep your existing `.data` directory in place — setup,
source credentials, run history, and captured warehouse data carry forward.
Run `./datalogz version` and `./datalogz preflight` afterward to confirm the
upgrade.

Your existing workspace keeps whatever storage layout it was created with, so
upgrading does not change or move your data. Newer releases create new
workspaces as a single encrypted `datalogz.db` file; a workspace from an older
release created a `datalogz.duckdb` state file plus a `ducklake/` directory and
continues to work as-is. To convert an older workspace to the single-file
layout, run `./datalogz warehouse migrate --root ./.data` (it verifies the copy
and keeps the original under `backup-ducklake/`).

## Backup and Restore

Back up `./.data` only after `./datalogz run` exits. The archive includes the
encryption key (`.data/datalogz.key`), so store the backup as securely as the
live workspace:

```bash
tar -czf datalogz-data-backup.tgz ./.data
```

Restore the whole data directory together, then validate with a read-only query:

```bash
tar -xzf datalogz-data-backup.tgz
./datalogz query --sql "SELECT table_schema, table_name FROM information_schema.tables WHERE table_catalog = 'warehouse'"
```

## Support Bundle

For first-run support, Datalogz may ask for:

- `./datalogz version` output.
- `./datalogz sources list` output.
- `./datalogz sources show --name SOURCE_NAME` output; secrets are redacted.
- Terminal output from `preflight`, `sources validate`, and `run`.
- Run-history table output from `datalogz_runs`, `datalogz_run_sources`,
  `datalogz_run_entities`, and `datalogz_run_events`.
- A `datalogz-bundle.tgz` created with `./datalogz bundle create --mask-emails`
  and uploaded with an approved Datalogz-provided URL.
