GhanaLSS: Multiple incomplete features across country and wave levels

[ligon]

Found in GhanaLSS/_/CONTENTS.org:

Country-level TODOs:

1. ghana_lss.py — common code module, not yet complete
2. food_items.org — food label harmonization needed (references Google Sheet for cross-survey labels)
3. conversion_to_kgs.json — not yet created
4. other_features.py — TODO
5. household_characteristics.py — TODO
6. food_acquired.py — TODO (has documentation notes about survey design differences)
7. food_prices_quantities_and_expenditures.py — TODO
8. Food Conversion Table — TODO (references West Africa FCT Google Sheets)

Specific sub-issue — food_expenditures:

This parquet file built in a non-standard fashion, and is missing indices (t,m). Code here acquires those indices from household_characteristics.

Wave-level TODOs:

• household_characteristics.py — TODO for all waves
• food_acquired.py — TODO for all waves
• other_features.py — TODO for all waves

This is one of the least complete countries in the library.

---

Filed from CONTENTS.org audit.

[ligon]

Agent investigation confirms: GhanaLSS (GLSS 5, 2005-06) does not contain a shocks module. The survey instrument does not include household shock questions. This is a data gap — shocks cannot be added for this country.

[ligon]

GhanaLSS shocks: not feasible. Exhaustive check of all 7 GLSS rounds (1987-88 through 2016-17) confirms the GLSS questionnaire does not include a household shocks/coping module in any wave.

Section 13 in GLSS 6-7 is "Governance, Peace and Security" (crime, sexual assault, judicial trust, democracy) — structurally different from the LSMS shocks schema.

If shock data for Ghana is needed, alternatives:

• GhanaSPS may have a shocks module
• Community-level questionnaires (g7comSEC*) have community-level disaster data
• External sources (EM-DAT) could supplement

Removing shocks from the GhanaLSS coverage gap list.

[ligon]

Cross-country sanity impact (for prioritization)

While doing a sanity check on ll.Feature('food_expenditures')() today (#242), discovered that GhanaLSS is the last remaining blocker for cross-country index alignment on the derived food tables. After the Ethiopia + Nigeria fixes in #242, all other 12 microdata-bearing countries return the canonical 5-level [i, t, v, j, s] index. GhanaLSS returns 4-level (no s), and pd.concat falls back to a flat single-level index when shapes mismatch — destroying country provenance on the all-15 Feature() call.

Diagnosis of GhanaLSS state:

• 1987-88 / 1988-89: wave-level parquets are wide-form pre-Phase-3. Columns are purchased_value, produced_value_daily, produced_value_yearly (no quantity columns at all — value-only). Won't fit food_acquired_to_canonical cleanly because that helper expects Quantity, Expenditure, Produced.
• 1991-92: completely different shape, index [h, w, j, u, visit], columns [Purchased, Price, Produced]. Some data already in canonical-adjacent form but with non-canonical level names.
• Country-level _/food_acquired.py: also crashes during a fresh build with AssertionError: Non-unique index in id_walk for 1988-89. So Ghana's food_acquired pipeline is broken upstream of any reshape concern; the 1.5M-row cache currently in user-data dirs is stale from before the id_walk regression.

Given this, the fix path for #109 likely involves:

1. Repair the id_walk non-unique-index regression so the country-level script runs at all.
2. Decide on a Quantity-imputation policy for 1987-88 / 1988-89 (NaN-propagation? skip the canonical reshape and emit s='purchased' constant level? something else?).
3. Refactor 1991-92's wave-level data to match canonical level names.

I tried a band-aid (adding s='purchased' constant level at country-level) earlier in this session but reverted it — it's dead code while the id_walk assertion breaks the country-level rebuild upstream of where the band-aid would run. The proper fix wants to address all three subproblems together; #109 is the right place for it.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[ligon]

Progress: food_acquired pipeline + cross-country sanity

PR #243 lands the food_acquired part of #109. Cross-country Feature('food_expenditures')() now returns the documented 6-level [country, i, t, v, j, s] shape over all 14 microdata-bearing countries (Nepal silently dropped per the no-microdata list), with GhanaLSS contributing 1.3M rows (was 0 before — silently dropped via groupby(dropna=True) after the v-join failed).

Three pathologies the country-level concatenator wasn't handling:

1. i ↔ j inversion in 1987-88, 1988-89, 1998-99, 2005-06 — wave-level scripts rename HID -> j, FOODCD -> i, opposite of canonical.
2. Non-canonical level names in 1991-92 (h, w) and 2012-13 / 2016-17 (w instead of t).
3. i format mismatch for 1998-99 and 2005-06 — wave scripts emit f"{clust}/{nh:02d}" ('4002/01') but those waves' sample uses concatenated digits ('400201'). Zero overlap on the v-join.

Plus the id_walk / change_id assertion regression that was crashing rebuilds at 1988-89 — root cause was the i↔j swap making change_id apply a HH-id linkage map to the food-item level, garbling the result. Per CONTENTS.org "each round is designed as an independent nationally representative cross-section", cross-wave id_walk is meaningless by design for GhanaLSS, so the rewrite drops it entirely.

The PR uses a per-wave SCHEMA config dict at the country level to handle all three pathologies uniformly without touching the wave-level scripts (which would also break the legacy food_prices_quantities_and_expenditures.py path).

Still open under #109

• Wave-level food_acquired.py rewrites — long-term we should eliminate the i↔j swap and h/w naming at the source rather than working around them.
• s='purchased' constant tag is a Phase-3 placeholder — proper purchased / produced reshape with imputed Quantity is still TODO.
• food_quantities for 1987-88 / 1988-89 / 1998-99 / 2005-06 / 1991-92 — these waves are value-only or only have produced quantity; their Quantity column is mostly NaN. food_quantities() currently returns just the 2016-17 wave (341,403 rows) for GhanaLSS.
• ghana_lss.py, conversion_to_kgs.json, other_features.py, household_characteristics.py, food_items.org, food_prices_quantities_and_expenditures.py, West Africa Food Conversion Table — all listed in the original CONTENTS audit, all still TODO.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[ligon]

Data-availability sleuth (2026-06-07): CODE-GAP. 395 .dta on disk; food_items.org already fully harmonized (all 7 waves). No var parquets built — pure build/script incompleteness; conversion + FCT exist as sheets, just wire them.