Implement ANS structured ingestion

[gabrool]

Codex Design Contract: ANS Structured Ingestion

You are adding the next data source to Brazil-RV: ANS open data, Agência Nacional de Saúde Suplementar.

Follow the existing architecture:

raw -> bronze -> source-specific silver -> later gold/research panels

Keep this PR narrow. Do not add raw-to-research panels, modeling, backtesting, portfolio logic, event-risk overlays, NLP, PDF/XLSX parsing, browser automation, broad scraping, new dependencies, or non-ANS sources. Do not modify B3/BCB/IBGE/Tesouro/FRED/CVM/ONS/ANBIMA code except for tiny shared config compatibility if required by tests.

Read first:

• AGENTS.md
• docs/PROJECT_PLAN.md
• docs/TIMING_AND_AVAILABILITY_POLICY.md
• existing ingestion modules for ONS/CVM/FRED/Tesouro/IBGE/BCB
• src/bralpha/infra/
• src/bralpha/metadata/
• src/bralpha/parsing/common.py
• src/bralpha/timing/availability.py

---

Source decision

ANS data is useful, but not all ANS data is worth first-pass live ingestion. For our daily Brazil RV model, ANS is mainly useful as a slow-moving regulated services / health inflation / household budget / health-sector stress source.

The first ANS PR should prioritize low-to-medium dimensional, deterministic datasets:

beneficiary coverage / coverage rate
operator registry
operator financial statements
premium/repricing information
complaint/quality pressure index

Do not ingest huge beneficiary-by-operator state files or detailed consumer-demand files in this first PR.

---

Official ANS facts to respect

ANS publishes public open data under:

https://dadosabertos.ans.gov.br/FTP/PDA/

Observed official directory facts:

• The top-level FTP/PDA index lists ANS open-data directories including beneficiary datasets, DIOPS/accounting statements, active operators, IGR, coverage-rate, pricing/repricing, consumer complaints, SIP, TISS, and other datasets.
• dados_de_beneficiarios_por_operadora/ contains state-level active/inactive beneficiary ZIPs; some files are very large, e.g. SP inactive around GB scale. Do not live-ingest these in this PR.
• dados_de_beneficiarios_por_regiao_geografica/ exposes benef_regiao_geog.zip and a data dictionary.
• informacoes_consolidadas_de_beneficiarios-024/ exposes month directories from 2019 onward and per-UF monthly files; useful but high-dimensional/large. Source-map only in this PR.
• taxa_de_cobertura_de_planos_de_saude-047/ exposes one coverage-rate CSV and a data dictionary.
• operadoras_de_plano_de_saude_ativas/ exposes Relatorio_cadop.csv and a data dictionary.
• demonstracoes_contabeis/ exposes annual directories from 2001 onward. Recent directories contain quarter ZIPs such as 1T2025.zip, 2T2025.zip, 3T2025.zip, 4T2025.zip, plus a dictionary at the root.
• percentuais_de_reajuste_de_agrupamento-055/ exposes a repricing/reajuste CSV and a dictionary.
• IGR/IGR_versao_2023/ exposes IGR.csv, a dictionary, and a technical note.
• demandas_dos_consumidores_nip/ exposes large yearly demand files; useful but source-map-only for now.
• painel_precificacao-053/ exposes a large pricing ZIP around hundreds of MB; source-map-only for now.

Use direct deterministic URLs. Do not crawl directory listings at runtime except in tests/mocks if needed. Do not guess hidden endpoints.

---

Dataset decision

P0 live datasets

Implement these live end-to-end if fixture-tested:

ans_coverage_rate_monthly
ans_beneficiaries_region_monthly
ans_operator_registry_current
ans_operator_financial_statements_quarterly

Rationale:

• ans_coverage_rate_monthly: low-dimensional health-plan coverage and population-coverage pressure; useful for household/services state.
• ans_beneficiaries_region_monthly: beneficiary-count state by geographic/product segment; useful for health-plan penetration and demand.
• ans_operator_registry_current: reference table for operators; not historical model feature by itself.
• ans_operator_financial_statements_quarterly: healthcare operator revenue/cost/capital stress, service-cost pressure, health-sector financial regime.

P1 live if fixture-tested and small enough

Implement these live if the fixture/schema is straightforward:

ans_repricing_group_percentages
ans_igr_current

Rationale:

• ans_repricing_group_percentages: regulated/contract premium repricing pressure, healthcare-services inflation channel.
• ans_igr_current: complaint/quality pressure and service-stress index; useful but noisier than coverage/financials.

P1/P2 source-map only for this PR

Add to config/source map, but do not implement live in this PR:

ans_beneficiaries_operator_by_state
ans_beneficiaries_consolidated_monthly_uf
ans_consumer_demands_nip
ans_consumer_complaints_beneficiaries
ans_pricing_panel
ans_sip_assistance_map
ans_sus_reimbursement_effective_payment
ans_hospital_network_change_requests
ans_penalties_operators
ans_tiss
ans_idss_history
ans_plan_history

Reasons:

• Beneficiary-by-operator files are huge and high-cardinality.
• Consolidated monthly UF files are useful but large; source-map first and decide later whether to aggregate in ingestion or process only selected dimensions.
• Consumer-demand/NIP files are large and more event/complaint-specific; source-map first.
• Pricing panel is large; useful later if a dedicated pricing/inflation PR is justified.
• SIP assistance map is potentially useful for utilization, but slower and requires a dedicated schema decision.
• SUS reimbursement, penalties, hospital-network changes, TISS, IDSS, and plan history are specialized or slower-moving.

No fake endpoints for source-map-only datasets.

---

Files to add

Create only files with real code:

docs/ANS_SOURCE_MAP.md

configs/datasets/ans.yaml
configs/sources/ans.yaml

src/bralpha/ingestion/ans/common.py
src/bralpha/ingestion/ans/downloads.py
src/bralpha/ingestion/ans/resources.py

src/bralpha/parsing/ans_tabular.py
src/bralpha/normalization/ans_health.py
src/bralpha/pipelines/ans_ingest.py

tests/test_ans_source_map.py
tests/test_ans_resources.py
tests/test_ans_downloads.py
tests/test_ans_tabular_parsing.py
tests/test_ans_health_normalization.py
tests/test_ans_ingest_pipeline.py

Do not create empty modules.

Update src/bralpha/infra/config.py only as needed to add:

load_ans_dataset_registry(repo_root)

---

Storage layout

Use existing conventions:

data/raw/ans/{dataset_id}/{download_date}/
data/bronze/ans/{dataset_id}/
data/silver/{dataset_id}/
data/manifests/ans/downloads.jsonl

Do not commit downloaded data.

---

Config: datasets

Create:

configs/datasets/ans.yaml

Minimum top-level shape:

source: ans
free_access_default: true
requires_auth_default: false
point_in_time_required: true

raw_storage:
  path_template: data/raw/ans/{dataset_id}/{download_date}/
  manifest_path: data/manifests/ans/downloads.jsonl

bronze_storage:
  path_template: data/bronze/ans/{dataset_id}/
  format: parquet

silver_storage:
  path_template: data/silver/{dataset_id}/
  format: parquet

Datasets:

datasets:
  - dataset_id: ans_coverage_rate_monthly
    source: ans
    priority: P0
    frequency: monthly
    raw_format: csv
    canonical_table: ans_coverage_rate_monthly
    partition_keys: [year]
    primary_keys: [ref_date, geography_level, geography_code, coverage_type]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
      - ref_date_not_null
      - available_date_not_null
      - available_date_on_or_after_ref_date
    source_map_status: live_download
    source_urls:
      - name: taxa_cobertura
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/taxa_de_cobertura_de_planos_de_saude-047/pda-047-taxa_cobertura.csv"
        filename_template: "pda-047-taxa_cobertura.csv"
      - name: data_dictionary
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/taxa_de_cobertura_de_planos_de_saude-047/dicionario-pda-047-taxa_cobertura.csv"
        filename_template: "dicionario-pda-047-taxa_cobertura.csv"
        role: dictionary
    availability_policy: ans_monthly_conservative_45d
    notes: Coverage-rate table. Preserve official rates/counts only; no derived penetration changes.

  - dataset_id: ans_beneficiaries_region_monthly
    source: ans
    priority: P0
    frequency: monthly
    raw_format: zip_csv
    canonical_table: ans_beneficiaries_region_monthly
    partition_keys: [year]
    primary_keys: [ref_date, geography_level, geography_code, coverage_type, plan_type]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
      - ref_date_not_null
      - available_date_not_null
      - available_date_on_or_after_ref_date
    source_map_status: live_download
    source_urls:
      - name: beneficiarios_regiao_geografica
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/dados_de_beneficiarios_por_regiao_geografica/benef_regiao_geog.zip"
        filename_template: "benef_regiao_geog.zip"
    availability_policy: ans_monthly_conservative_45d
    notes: Beneficiary counts by geographic region/product dimensions. Keep official rows; no high-cardinality operator dimension.

  - dataset_id: ans_operator_registry_current
    source: ans
    priority: P0
    frequency: daily_snapshot
    raw_format: csv
    canonical_table: ans_operator_registry_current
    partition_keys: [snapshot_year]
    primary_keys: [operator_id]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
    source_map_status: live_download
    source_urls:
      - name: relatorio_cadop
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/operadoras_de_plano_de_saude_ativas/Relatorio_cadop.csv"
        filename_template: "Relatorio_cadop.csv"
    notes: Current active-operator registry/reference table. Not historical model feature by itself.

  - dataset_id: ans_operator_financial_statements_quarterly
    source: ans
    priority: P0
    frequency: quarterly
    raw_format: zip_csv
    canonical_table: ans_operator_financial_statement
    partition_keys: [year]
    primary_keys: [ref_date, operator_id, account_code]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
      - ref_date_not_null
      - available_date_not_null
      - available_date_on_or_after_ref_date
    source_map_status: live_download
    direct_url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/demonstracoes_contabeis/{year}/{quarter}T{year}.zip"
    filename_template: "{quarter}T{year}.zip"
    availability_policy: ans_quarterly_conservative_90d
    notes: DIOPS/accounting statements by operator/account. Preserve official account values only; no loss ratios.

  - dataset_id: ans_repricing_group_percentages
    source: ans
    priority: P1
    frequency: event_or_monthly
    raw_format: csv
    canonical_table: ans_repricing_group_percentage
    partition_keys: [year]
    primary_keys: [ref_date, operator_id, contract_group_id]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
    source_map_status: live_download
    source_urls:
      - name: percentuais_reajuste_agrupamento
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/percentuais_de_reajuste_de_agrupamento-055/pda-055-Percentuais_de_Reajuste_de_Agrupamento.csv"
        filename_template: "pda-055-Percentuais_de_Reajuste_de_Agrupamento.csv"
    availability_policy: ans_date_or_month_conservative
    notes: Premium/repricing percentages. Useful inflation-pressure source. Preserve official percentages; do not average or aggregate in ingestion.

  - dataset_id: ans_igr_current
    source: ans
    priority: P1
    frequency: monthly_or_snapshot
    raw_format: csv
    canonical_table: ans_igr_current
    partition_keys: [year]
    primary_keys: [ref_date, operator_id]
    quality_checks:
      - required_columns_present
      - no_duplicate_primary_keys
    source_map_status: live_download
    source_urls:
      - name: igr_2023_current
        url_template: "https://dadosabertos.ans.gov.br/FTP/PDA/IGR/IGR_versao_2023/IGR.csv"
        filename_template: "IGR.csv"
    availability_policy: ans_monthly_conservative_45d
    notes: Complaint/quality pressure index. Preserve official IGR fields only.

  - dataset_id: ans_beneficiaries_operator_by_state
    source: ans
    priority: P1
    frequency: snapshot_or_monthly
    raw_format: zip_csv_large
    canonical_table: ans_beneficiaries_operator_by_state
    partition_keys: [state_code]
    primary_keys: [ref_date, state_code, operator_id, product_id]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_large_high_cardinality
    source_urls: []
    notes: Very large active/inactive per-state files. Defer; do not live-ingest in this PR.

  - dataset_id: ans_beneficiaries_consolidated_monthly_uf
    source: ans
    priority: P1
    frequency: monthly
    raw_format: zip_csv_large
    canonical_table: ans_beneficiaries_consolidated_monthly_uf
    partition_keys: [year, month, state_code]
    primary_keys: [ref_date, state_code, municipality_code, coverage_type, plan_type]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_large_pending_aggregation_plan
    source_urls: []
    notes: Monthly per-UF consolidated beneficiary files are useful but large. Defer until aggregation plan is approved.

  - dataset_id: ans_consumer_demands_nip
    source: ans
    priority: P1
    frequency: annual_or_monthly
    raw_format: csv_large
    canonical_table: ans_consumer_demand_nip
    partition_keys: [year]
    primary_keys: [demand_id]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_large_event_data
    source_urls: []
    notes: Large consumer-demand files. Useful later as complaint/event stress, but not first-pass live ingestion.

  - dataset_id: ans_pricing_panel
    source: ans
    priority: P1
    frequency: snapshot
    raw_format: zip_csv_large
    canonical_table: ans_pricing_panel
    partition_keys: [year]
    primary_keys: [record_id]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_large_pricing
    source_urls: []
    notes: Large pricing panel. Potentially useful for health premium inflation; defer.

  - dataset_id: ans_sip_assistance_map
    source: ans
    priority: P2
    frequency: annual
    raw_format: csv_annual
    canonical_table: ans_sip_assistance_map
    partition_keys: [year]
    primary_keys: [ref_date, operator_id, assistance_metric]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_slow_utilization
    source_urls: []
    notes: Utilization/assistance map is slower. Dedicated PR later if needed.

  - dataset_id: ans_sus_reimbursement_effective_payment
    source: ans
    priority: P2
    frequency: monthly
    raw_format: csv_monthly
    canonical_table: ans_sus_reimbursement_effective_payment
    partition_keys: [year]
    primary_keys: [ref_date, operator_id]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_specialized
    source_urls: []
    notes: Specialized regulatory cash-flow/stress source; defer.

  - dataset_id: ans_penalties_operators
    source: ans
    priority: P3
    frequency: event
    raw_format: csv
    canonical_table: ans_penalty_operator
    partition_keys: [year]
    primary_keys: [penalty_id]
    quality_checks: [required_columns_present, no_duplicate_primary_keys]
    source_map_status: source_map_only_low_priority
    source_urls: []
    notes: Regulatory event source, lower priority.

If any configured direct URL has a filename mismatch after fixture verification, correct the URL/template and document the correction. Do not implement a runtime crawler to compensate.

---

Config: source metadata

Create:

configs/sources/ans.yaml

Minimum:

source: ans
portal:
  name: ANS Dados Abertos / PDA FTP
  url: https://dadosabertos.ans.gov.br/FTP/PDA/
  license_note: Dados públicos; cite ANS and the dataset/source.

official_pages:
  - name: pda_root
    url: https://dadosabertos.ans.gov.br/FTP/PDA/
  - name: coverage_rate
    dataset_id: ans_coverage_rate_monthly
    url: https://dadosabertos.ans.gov.br/FTP/PDA/taxa_de_cobertura_de_planos_de_saude-047/
  - name: beneficiaries_region
    dataset_id: ans_beneficiaries_region_monthly
    url: https://dadosabertos.ans.gov.br/FTP/PDA/dados_de_beneficiarios_por_regiao_geografica/
  - name: operator_registry_current
    dataset_id: ans_operator_registry_current
    url: https://dadosabertos.ans.gov.br/FTP/PDA/operadoras_de_plano_de_saude_ativas/
  - name: financial_statements
    dataset_id: ans_operator_financial_statements_quarterly
    url: https://dadosabertos.ans.gov.br/FTP/PDA/demonstracoes_contabeis/
  - name: repricing_group_percentages
    dataset_id: ans_repricing_group_percentages
    url: https://dadosabertos.ans.gov.br/FTP/PDA/percentuais_de_reajuste_de_agrupamento-055/
  - name: igr_current
    dataset_id: ans_igr_current
    url: https://dadosabertos.ans.gov.br/FTP/PDA/IGR/IGR_versao_2023/

endpoint_policy:
  live_download_requires:
    - stable_direct_url_or_template
    - no_login
    - no_browser_automation
    - fixture_test
    - no_new_dependency

---

Source map doc

Create:

docs/ANS_SOURCE_MAP.md

Must include table columns:

dataset_id
priority
status
source_page_or_endpoint
raw_format
expected_frequency
silver_output
known_limitations

Document:

• Why ANS is useful: health-plan coverage, private healthcare demand, premium/repricing pressure, operator financial stress, complaint/service pressure.
• Which datasets are live in this PR.
• Which datasets are source-map-only and why.
• The huge-file caveat for beneficiary-by-operator and consolidated monthly UF data.
• Availability assumptions and conservative lag policies.
• No derived ratios, health inflation indices, claims ratios, loss ratios, growth rates, rolling features, stress labels, or model features in ingestion.

---

Resource generation

Create:

src/bralpha/ingestion/ans/resources.py

Implement:

@dataclass(frozen=True)
class ANSResourceRequest:
    dataset_id: str
    resource_name: str
    url: str
    filename: str
    year: int | None = None
    quarter: int | None = None


def ans_static_resources(dataset_config) -> list[ANSResourceRequest]: ...
def ans_diops_quarterly_resources(dataset_config, start: date, end: date) -> list[ANSResourceRequest]: ...

Rules:

• Static datasets return their configured data files and, optionally, dictionaries if marked role dictionary. Dictionaries should be downloaded raw/manifest but not parsed into bronze unless the pipeline explicitly skips them by role.
• DIOPS quarterly resources generate one ZIP per quarter intersecting [start, end]:

https://dadosabertos.ans.gov.br/FTP/PDA/demonstracoes_contabeis/{year}/{quarter}T{year}.zip

• Do not request daily files.
• Do not crawl directory listings at runtime.
• Sort resources chronologically.
• Tests cover static resources, DIOPS multi-quarter, and inverted windows.

---

Downloader

Create:

src/bralpha/ingestion/ans/common.py
src/bralpha/ingestion/ans/downloads.py

Reuse existing infrastructure:

HttpClient
RawStore
ManifestWriter
ManifestRecord
sha256 helpers
client_context pattern

Implement:

def download_ans_dataset(repo_root, dataset_id, start=None, end=None, client=None, downloaded_at=None):
    ...

Rules:

• Source-map-only datasets raise a clear error and write no data.
• ans_operator_financial_statements_quarterly requires start and end and uses quarterly resource generation.
• Other live static datasets ignore start/end unless explicitly needed; reject start/end to avoid false partial downloads.
• Write one raw file per resource.
• Write manifest records for success/failure.
• Do not parse inside downloader.
• Reuse one HTTP client per run.
• No retries beyond existing HttpClient.

---

Parser

Create:

src/bralpha/parsing/ans_tabular.py

Support:

csv
zip_csv

Use standard library only for ZIP:

zipfile
io

Bronze columns:

row_index
resource_name
year
quarter
inner_filename
all raw_<normalized_source_column> fields
source
source_dataset
download_timestamp_utc
raw_path
sha256

Rules:

• Read CSV as strings first.
• Support UTF-8, Latin-1, and CP1252 decoding.
• Detect semicolon/comma/tab delimiters.
• Normalize source columns with existing normalize_column_name.
• Preserve all raw columns as raw_<normalized_source_column>.
• Do not create per-row raw_fields_json.
• Keep resource_name, year, quarter, and ZIP inner_filename lineage.
• If a ZIP contains multiple CSV/TXT members, parse and concatenate with inner_filename.
• Skip dictionary resources in the pipeline unless explicitly requested; dictionaries remain raw/manifest documentation inputs.
• Use Polars.
• Do not parse XLSX or PDF.

---

Normalization

Create:

src/bralpha/normalization/ans_health.py

Implement silver normalizers for live data resources only. Use data dictionaries and fixture-realistic aliases. If a schema is ambiguous, keep that dataset bronze-only pending normalizer rather than guessing.

Shared timing policies

Use conservative historical availability:

ans_monthly_conservative_45d:
  available_date = next_business_day(month_end(ref_date) + 45 calendar days)

ans_quarterly_conservative_90d:
  available_date = next_business_day(quarter_end(ref_date) + 90 calendar days)

ans_date_or_month_conservative:
  if exact event/publication date exists -> next_business_day(date)
  else month-end + 45 calendar days -> next_business_day

Never use download timestamp as historical availability, except for snapshot/reference-only snapshot_date.

ans_coverage_rate_monthly

Output:

data/silver/ans_coverage_rate_monthly/

Required columns:

ref_date
available_date
availability_policy
geography_level
geography_code
geography_name
state_code
coverage_type
plan_type
beneficiary_count
population_count
coverage_rate
unit
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• Preserve official beneficiary/population/rate fields where present.
• If geography-level fields are ambiguous, derive only from explicit official fields; do not infer from names.
• Do not compute coverage changes, growth rates, or ratios beyond official coverage rate.
• Primary key: ref_date, geography_level, geography_code, coverage_type.

ans_beneficiaries_region_monthly

Output:

data/silver/ans_beneficiaries_region_monthly/

Required columns:

ref_date
available_date
availability_policy
geography_level
geography_code
geography_name
state_code
region_name
coverage_type
plan_type
beneficiary_count
unit
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• Preserve official beneficiary-count rows.
• Do not compute changes, penetration rates, or shares.
• Primary key: ref_date, geography_level, geography_code, coverage_type, plan_type.

ans_operator_registry_current

Output:

data/silver/ans_operator_registry_current/

Required columns:

operator_id
operator_cnpj
operator_name
trade_name
modality
city
state_code
region
registration_date
status
snapshot_date
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• Map common Relatorio_cadop.csv fields such as Registro_ANS, CNPJ, Razao_Social, Nome_Fantasia, Modalidade, Cidade, UF, Regiao_de_Comercializacao, Data_Registro_ANS when present.
• snapshot_date = download_timestamp_utc.date() unless an official snapshot/reference date exists.
• Reference table only; not a historical model feature by itself.

ans_operator_financial_statements_quarterly

Output:

data/silver/ans_operator_financial_statements_quarterly/

Required columns:

ref_date
available_date
availability_policy
year
quarter
operator_id
account_code
account_name
account_value
unit
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• ref_date = quarter_end(year, quarter) unless an official statement date is present and fixture-verified.
• Preserve official account code/name/value rows.
• Do not compute revenue, expenses, loss ratio, claims ratio, capital ratios, or profitability metrics.
• Primary key: ref_date, operator_id, account_code.

ans_repricing_group_percentages

Output:

data/silver/ans_repricing_group_percentages/

Required columns:

ref_date
available_date
availability_policy
operator_id
contract_group_id
product_id
contracting_type
reajuste_percent
application_start_date
application_end_date
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• Preserve official repricing percentage and effective/application dates where present.
• If no exact effective date is present, use conservative month/reference-date policy.
• Do not aggregate, average, or winsorize percentages.
• Primary key: ref_date, operator_id, contract_group_id.

ans_igr_current

Output:

data/silver/ans_igr_current/

Required columns:

ref_date
available_date
availability_policy
operator_id
operator_name
igr_value
complaint_count
beneficiary_count
segment
source
source_dataset
download_timestamp_utc
raw_path
sha256
source_version

Rules:

• Preserve official IGR and complaint/beneficiary fields where present.
• Do not compute complaint ratios or stress labels.
• Primary key: ref_date, operator_id.

---

Pipeline

Create:

src/bralpha/pipelines/ans_ingest.py

CLI:

python -m bralpha.pipelines.ans_ingest \
  --repo-root . \
  --dataset ans_coverage_rate_monthly

For DIOPS:

python -m bralpha.pipelines.ans_ingest \
  --repo-root . \
  --dataset ans_operator_financial_statements_quarterly \
  --start 2010-01-01 \
  --end 2026-01-01

Behavior:

download raw -> parse bronze -> normalize silver -> quality checks

Rules:

• Do not run source-map-only datasets.
• Skip dictionary resources from bronze/silver processing.
• For DIOPS, process quarter resources incrementally: parse -> write bronze -> normalize -> filter -> quality-check -> write silver.
• For static datasets, one-shot processing is acceptable if file sizes are modest; avoid concatenating huge source-map-only files because they are not live.
• Idempotent reruns must not duplicate primary keys.

---

Efficiency requirements

• Do not live-ingest ANS files that are GB-scale or high-cardinality in this PR.
• Parse large ZIP/CSV resources one resource at a time.
• Do not create per-row JSON blobs.
• Use Polars.
• Partition silver by year where date-indexed.
• Keep operator registry as snapshot/reference.
• No distributed compute libraries.

---

Tests

Use fixtures/mocks only. No live ANS calls.

Source map/config

• configs/datasets/ans.yaml loads.
• configs/sources/ans.yaml loads.
• docs/ANS_SOURCE_MAP.md lists all configured datasets.
• P0/P1 live datasets have direct URLs or deterministic templates.
• Source-map-only datasets have empty source_urls.
• No fake endpoints for huge/deferred datasets.

Resource generation

• Static dataset resources render configured direct URLs.
• DIOPS range renders correct quarter ZIP URLs.
• Dictionary resources are marked by role and skipped by parser pipeline.
• Inverted DIOPS window raises.

Downloader

• Mocked static download writes raw file and manifest.
• Mocked DIOPS multi-quarter download writes one raw file per quarter and manifests.
• Source-map-only dataset raises clear error and writes no data.
• Downloader does not parse.

Parser

• CSV fixture reads as strings.
• ZIP fixture with multiple CSV members preserves inner_filename.
• Raw columns become raw_<normalized_name>.
• resource_name, year, quarter, and lineage columns are preserved.
• No raw_fields_json.

Normalizers

Use realistic ANS-style fixture columns and alias matching.

Tests should cover:

• Coverage-rate normalizer maps reference date, geography, coverage type, beneficiary count, population, and official coverage rate.
• Beneficiary-region normalizer maps reference date/geography/coverage/product type and beneficiary count.
• Operator registry maps Registro_ANS, CNPJ, Razao_Social, Nome_Fantasia, Modalidade, Cidade, UF, and registration date.
• DIOPS normalizer maps quarter, operator ID, account code/name/value, and 90-day conservative availability.
• Repricing normalizer maps official percentage and effective/reference dates where present.
• IGR normalizer maps official index and complaint/beneficiary fields where present.
• No download timestamp is used as historical availability.
• Primary keys are unique.

Do not add static banned-feature denylist tests.

Pipeline

• Mocked static raw -> bronze -> silver works.
• Mocked DIOPS two-quarter raw -> bronze -> silver works incrementally.
• Silver is partitioned by year where applicable.
• Idempotent rerun does not duplicate primary keys.
• Source-map-only dataset fails clearly.

---

Acceptance criteria

The PR is complete when:

1. docs/ANS_SOURCE_MAP.md exists.
2. configs/datasets/ans.yaml exists.
3. configs/sources/ans.yaml exists.
4. P0 live ANS datasets ingest through mocked direct CSV/ZIP responses.
5. P1 live datasets ingest only if fixture-tested; otherwise mark source-map/bronze-only clearly.
6. Huge/high-cardinality ANS datasets are source-mapped only with no fake endpoints.
7. Raw CSV/ZIP, bronze, silver, manifests, and quality checks work end to end.
8. Strict timing policy is respected with conservative monthly/quarterly availability lags.
9. No ratios, changes, loss ratios, claims ratios, inflation indices, rolling features, z-scores, stress labels, or model/backtest features are added.
10. No new dependencies are added.
11. No XLSX/PDF/browser/scraping workflow is added.
12. python -m pytest passes.
13. python -m ruff check . passes.
14. No generated data is committed.

Keep the implementation deterministic, source-map disciplined, point-in-time safe, and lean.