[codex] Add incremental scan queue worker MVP

.dockerignore

@@ -0,0 +1,51 @@
+.git
+.gitignore
+.dockerignore
+
+private
+targets.local.yaml
+targets.local.yml
+*.local.yaml
+*.local.yml
+
+.env
+.env.*
+*.pem
+*.key
+*.pfx
+credentials
+credentials.json
+*.tfvars
+
+.venv
+venv
+.uv
+__pycache__
+.pytest_cache
+.mypy_cache
+.ruff_cache
+coverage
+.coverage
+htmlcov
+dist
+build
+*.egg-info
+
+findings
+reports
+graphify-out
+graphify*
+*graphify*
+.scanner
+workspaces
+*.log
+*.db
+*.sqlite
+*.sarif
+gitleaks-report.json
+*.report.json
+
+.DS_Store
+.idea
+.vscode
+.worktrees

Dockerfile

@@ -0,0 +1,53 @@
+FROM python:3.13-slim
+
+ARG GH_VERSION=2.94.0
+ARG GLAB_VERSION=1.102.0
+ARG GITLEAKS_VERSION=8.24.0
+ARG TARGETARCH
+
+ENV PYTHONUNBUFFERED=1 \
+    UV_SYSTEM_PYTHON=1
+
+WORKDIR /app
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends ca-certificates curl git \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN set -eux; \
+    arch="${TARGETARCH:-$(dpkg --print-architecture)}"; \
+    case "$arch" in \
+        amd64) gh_arch="amd64"; glab_arch="amd64"; gitleaks_arch="x64" ;; \
+        arm64) gh_arch="arm64"; glab_arch="arm64"; gitleaks_arch="arm64" ;; \
+        *) echo "unsupported gitleaks architecture: $arch" >&2; exit 1 ;; \
+    esac; \
+    curl -fsSL \
+        "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${gh_arch}.tar.gz" \
+        -o /tmp/gh.tar.gz; \
+    tar -xzf /tmp/gh.tar.gz -C /tmp; \
+    mv "/tmp/gh_${GH_VERSION}_linux_${gh_arch}/bin/gh" /usr/local/bin/gh; \
+    rm -rf /tmp/gh.tar.gz "/tmp/gh_${GH_VERSION}_linux_${gh_arch}"; \
+    gh --version; \
+    curl -fsSL \
+        "https://gitlab.com/gitlab-org/cli/-/releases/v${GLAB_VERSION}/downloads/glab_${GLAB_VERSION}_linux_${glab_arch}.tar.gz" \
+        -o /tmp/glab.tar.gz; \
+    tar -xzf /tmp/glab.tar.gz -C /tmp bin/glab; \
+    mv /tmp/bin/glab /usr/local/bin/glab; \
+    rm -rf /tmp/glab.tar.gz /tmp/bin; \
+    glab --version; \
+    curl -fsSL \
+        "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_${gitleaks_arch}.tar.gz" \
+        -o /tmp/gitleaks.tar.gz; \
+    tar -xzf /tmp/gitleaks.tar.gz -C /usr/local/bin gitleaks; \
+    chmod +x /usr/local/bin/gitleaks; \
+    rm /tmp/gitleaks.tar.gz; \
+    gitleaks version
+
+COPY pyproject.toml uv.lock README.md ./
+COPY src ./src
+
+RUN pip install --no-cache-dir uv \
+    && uv pip install --system .
+
+ENTRYPOINT ["security-scanner"]
+CMD ["--help"]

README.md

@@ -73,10 +73,36 @@ uv run security-scanner gate   --findings private/findings.jsonl --max 0
 
 DynamoDB-compatible backend는 로컬에서 조회 패턴을 검증하기 위한 저장소입니다. 관리형 저장소 연동은 현재 지원 범위가 아닙니다.
 
-로컬 DB는 저장소에 포함된 `docker-compose.yml`의 Dynalite 컨테이너로 띄웁니다. `http://localhost:4567`에서 응답합니다(로컬 검증 전용, 컨테이너를 내리면 데이터 소멸).
+로컬 DB는 저장소에 포함된 `docker-compose.yml`의 DynamoDB Local 컨테이너로
+띄웁니다. Host에서는 `http://localhost:4567`에서 응답하고, 데이터는 named
+Compose volume에 유지됩니다.
 
 ```bash
-docker compose up -d dynalite
+docker compose up -d dynamodb-local
+```
+
+Host의 `4567` 포트가 이미 사용 중이면 `SECURITY_SCANNER_DYNAMO_HOST_PORT`로
+바꿔 띄울 수 있습니다. Worker 컨테이너는 compose 내부 endpoint를 사용하므로
+그대로 동작합니다.
+
+```bash
+SECURITY_SCANNER_DYNAMO_HOST_PORT=14567 docker compose up -d dynamodb-local
+```
+
+새 PC에서 public HTTPS repo 하나를 바로 검증하려면 Docker 경로를 사용할 수 있습니다.
+
+```bash
+SECURITY_SCANNER_QUICKSTART_TARGET=https://github.com/<owner>/<repo> \
+  docker compose up --build --abort-on-container-exit --exit-code-from worker worker
+```
+
+커스텀 GitLab 도메인은 URL만으로 provider를 판별할 수 없으므로 provider hint를
+함께 지정합니다.
+
+```bash
+SECURITY_SCANNER_QUICKSTART_TARGET=https://source.example.test/<group>/<repo> \
+SECURITY_SCANNER_SCM_PROVIDER=gitlab \
+  docker compose up --build --abort-on-container-exit --exit-code-from worker worker
 ```
 
 ```bash
@@ -101,7 +127,7 @@ uv run security-scanner gate \
 
 스캔을 실행하면 `Scan run ID`가 출력됩니다. 특정 실행 결과만 보고 싶으면 그 값을 `--scan-run-id`로 넘깁니다. 저장소 전체를 대상으로 판단할 때만 생략합니다.
 
-카탈로그(`add-target`)에 등록한 여러 저장소를 한 번에 스캔하는 `scan-all` 흐름은 [시작하기 가이드의 "주기 스캔 로컬 테스트"](docs/views/getting-started.md#주기-스캔-로컬-테스트-dynalite--scan-all) 절을 참고합니다.
+카탈로그(`add-target`)에 등록한 여러 저장소를 한 번에 스캔하는 `scan-all` 흐름은 [시작하기 가이드의 "주기 스캔 로컬 테스트"](docs/views/getting-started.md#주기-스캔-로컬-테스트-dynamodb-local--scan-all) 절을 참고합니다.
 
 Schema와 조회 기준은 [소스 스캔 결과 NoSQL Schema](docs/views/source-scan-results-nosql-schema.md)에 정리되어 있습니다.
 

deploy/systemd/README.md

@@ -38,7 +38,7 @@ On the target Ubuntu host:
 - `gh` and `glab` CLIs installed and reachable on `PATH`. Required for
   GitHub/GitLab clone/fetch (spec §6).
 - A reachable DynamoDB-compatible backend on `http://localhost:4567`. For a
-  single host, run the Dynalite container shipped in the repo's
+  single host, run the DynamoDB Local container shipped in the repo's
   `docker-compose.yml` (see "Start the local DB" below). This requires Docker
   with the Compose v2 plugin.
 - A non-root service user (`scanner` by default) owning:
@@ -54,23 +54,49 @@ sudo install -d -o scanner -g scanner /var/log/security-scanner
 sudo install -d -o scanner -g scanner /var/cache/security-scanner
 ```
 
-**Start the local DB.** From the project tree, bring up Dynalite and create the
-table (and its query index) once:
+**Start the local DB.** From the project tree, bring up DynamoDB Local and
+create the table (and its query index) once:
 
 ```bash
-docker compose up -d dynalite
+docker compose up -d dynamodb-local
 
 uv run security-scanner init-storage \
   --storage-backend dynamodb \
   --dynamodb-endpoint-url http://localhost:4567 \
   --dynamodb-table security_scanner_local_dev
 ```
 
-Dynalite here is for single-host, local-only use; its data is in-memory and is
-lost if the container is removed. Register scan targets with `add-target`
-before the first scheduled run — see the
+If host port `4567` is already in use on a test box, set
+`SECURITY_SCANNER_DYNAMO_HOST_PORT=<free-port>` for the compose command. The
+worker service still talks to DynamoDB Local through the compose network.
+
+DynamoDB Local here is for single-host, local-only use; its data is persisted
+in the named Compose volume. Register scan targets with `add-target` before
+the first scheduled run — see the
 [getting-started guide](../../docs/views/getting-started.md). Managed DynamoDB
-and DynamoDB Local are out of scope for now.
+is out of scope for now.
+
+**Incremental worker local proof.** The repository's Docker Compose file also
+contains a `worker` service. It is for local verification only, not a production deployment target.
+
+```bash
+SECURITY_SCANNER_QUICKSTART_TARGET=https://github.com/<owner>/<repo> \
+  docker compose up --abort-on-container-exit --exit-code-from worker worker
+```
+
+For custom GitLab domains, add the provider hint:
+
+```bash
+SECURITY_SCANNER_QUICKSTART_TARGET=https://source.example.test/<group>/<repo> \
+SECURITY_SCANNER_SCM_PROVIDER=gitlab \
+  docker compose up --abort-on-container-exit --exit-code-from worker worker
+```
+
+That command exercises `security-scanner quickstart` against the local DynamoDB
+Local service, creates a current-tip queue job, and runs the worker. Keep
+credentials out of compose files and inject them through the host environment
+or the normal service manager only when you intentionally test private
+repository access.
 
 ---
 
@@ -83,8 +109,8 @@ For private repos, pass SCM tokens to the service. Two options.
 ```bash
 sudo install -d -m 700 /etc/security-scanner
 sudo tee /etc/security-scanner/scm.env >/dev/null <<'EOF'
-GH_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-GITLAB_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
+GH_TOKEN=<GH_TOKEN>
+GITLAB_TOKEN=<GITLAB_TOKEN>
 EOF
 sudo chmod 600 /etc/security-scanner/scm.env
 sudo chown scanner:scanner /etc/security-scanner/scm.env
@@ -132,14 +158,14 @@ the user manager reads without a desktop keyring).
 
 Prereqs (as the scanning user): the project checked out at `~/security-scanner`
 with `uv sync` run; `uv`/`git`/`gitleaks`/`gh`/`docker` on `PATH`; the local
-Dynalite DB reachable (the unit's `ExecStartPre` brings it up).
+DynamoDB Local DB reachable (the unit's `ExecStartPre` brings it up).
 
 ```bash
 # 1. Authenticate gh (token never leaves the host).
 gh auth login          # GitHub.com → HTTPS → paste a read-scoped token
 
-# 2. Bootstrap the catalog table once (Dynalite must be up).
-docker compose up -d dynalite
+# 2. Bootstrap the catalog table once (DynamoDB Local must be up).
+docker compose up -d dynamodb-local
 export SECURITY_SCANNER_STORAGE_BACKEND=dynamodb
 uv run security-scanner init-storage
 

deploy/systemd/user/security-scanner-scan-all.service

@@ -9,7 +9,7 @@ Type=oneshot
 # home directory, so this unit is portable across operators without edits.
 WorkingDirectory=%h/security-scanner
 
-# DynamoDB-compatible backend (Dynalite via the repo's docker-compose.yml).
+# DynamoDB-compatible backend (DynamoDB Local via the repo's docker-compose.yml).
 Environment=SECURITY_SCANNER_STORAGE_BACKEND=dynamodb
 Environment=SECURITY_SCANNER_DYNAMO_ENDPOINT=http://localhost:4567
 Environment=SECURITY_SCANNER_DYNAMO_TABLE=security_scanner_local_dev
@@ -20,8 +20,8 @@ Environment=SECURITY_SCANNER_DYNAMO_TABLE=security_scanner_local_dev
 # EnvironmentFile=-%h/.config/security-scanner/scm.env
 
 # Ensure the local DB container is up before scanning (no-op if already running).
-# Drop this line if you manage Dynalite separately.
-ExecStartPre=-/usr/bin/docker compose up -d dynalite
+# Drop this line if you manage DynamoDB Local separately.
+ExecStartPre=-/usr/bin/docker compose up -d dynamodb-local
 
 # Adjust the uv path if it lives elsewhere (`command -v uv`).
 ExecStart=%h/.local/bin/uv run security-scanner scan-all \

docker-compose.yml

@@ -1,33 +1,58 @@
-# Local-only DynamoDB-compatible backend for security-scanner.
+# Local-only DynamoDB Local backend and turnkey worker for security-scanner.
 #
-# Starts Dynalite on http://localhost:4567 — the default endpoint used by the
-# `dynamodb` storage backend (see README "로컬 NoSQL 저장소" and
-# docs/views/getting-started.md "주기 스캔 로컬 테스트").
+# Host CLI endpoint: http://localhost:4567
+# Container endpoint: http://dynamodb-local:8000
 #
-# Scope: local verification of the `scan-all` catalog/query path ONLY.
-# Not production. Data is in-memory and is lost when the container stops.
-# Managed DynamoDB / DynamoDB Local are intentionally out of scope for now.
+# Scope: local verification of scan-all and incremental queue paths.
+# Not production. Data is persisted in the named local Compose volume.
 services:
-  dynalite:
-    image: node:20-alpine
-    container_name: security-scanner-dynalite
-    # In-memory store (no --path) → pure-JS memdown, no native build needed.
-    command: ["npx", "--yes", "dynalite@3", "--port", "4567"]
+  dynamodb-local:
+    image: amazon/dynamodb-local:2.6.1
+    user: root
+    working_dir: /home/dynamodblocal
+    command:
+      - "-jar"
+      - "DynamoDBLocal.jar"
+      - "-sharedDb"
+      - "-dbPath"
+      - "./data"
     ports:
-      - "4567:4567"
+      - "${SECURITY_SCANNER_DYNAMO_HOST_PORT:-4567}:8000"
     volumes:
-      # Cache the npx download so restarts don't re-fetch dynalite.
-      - dynalite-npm:/root/.npm
-    healthcheck:
-      test:
-        - "CMD"
-        - "node"
-        - "-e"
-        - "require('net').connect(4567,'127.0.0.1').on('connect',()=>process.exit(0)).on('error',()=>process.exit(1))"
-      interval: 5s
-      timeout: 3s
-      retries: 5
+      - dynamodb-local-data:/home/dynamodblocal/data
     restart: unless-stopped
 
+  worker:
+    build: .
+    depends_on:
+      - dynamodb-local
+    environment:
+      SECURITY_SCANNER_STORAGE_BACKEND: dynamodb
+      SECURITY_SCANNER_DYNAMO_ENDPOINT: http://dynamodb-local:8000
+      SECURITY_SCANNER_DYNAMO_TABLE: SecurityScannerLocal
+      SECURITY_SCANNER_QUICKSTART_TARGET: ${SECURITY_SCANNER_QUICKSTART_TARGET:-}
+      SECURITY_SCANNER_QUICKSTART_NAME: ${SECURITY_SCANNER_QUICKSTART_NAME:-quickstart-target}
+      SECURITY_SCANNER_SCM_PROVIDER: ${SECURITY_SCANNER_SCM_PROVIDER:-auto}
+    volumes:
+      - repo-cache:/root/.cache/security-scanner/repos
+    entrypoint:
+      - /bin/sh
+      - -lc
+    command:
+      - |
+        test -n "$$SECURITY_SCANNER_QUICKSTART_TARGET" || {
+          echo "Set SECURITY_SCANNER_QUICKSTART_TARGET to a public HTTPS repo URL." >&2
+          exit 2
+        }
+        security-scanner quickstart "$$SECURITY_SCANNER_QUICKSTART_TARGET" \
+          --name "$$SECURITY_SCANNER_QUICKSTART_NAME" \
+          --storage-backend dynamodb \
+          --dynamodb-endpoint-url "$$SECURITY_SCANNER_DYNAMO_ENDPOINT" \
+          --dynamodb-table "$$SECURITY_SCANNER_DYNAMO_TABLE" \
+          --scm-provider "$$SECURITY_SCANNER_SCM_PROVIDER" \
+          --storage-wait-seconds 60 \
+          --max-jobs 10
+
 volumes:
-  dynalite-npm:
+  dynamodb-local-data:
+  repo-cache: