Technical Deep Dive: HPD Complaint-Risk Forecasting
Large-scale tabular learning, sparse-event prediction, temporal validation and operational evaluation.
← Back to the client-facing case study
This supplement covers the data and model design behind the HPD forecasting project. It is an independent applied ML project based on public NYC HPD data, not a claim that NYC HPD commissioned or used the system. Results from 2026-Q1 held-out testing and a 2026-Q2 retrospective of previously generated scores are reported separately because they measure different periods and evaluation views.
1. Prediction target and evaluation unit
The task was to estimate the probability of at least one complaint in the next quarter, for a granular combination of building identifier (BIN), apartment/unit and major complaint category. Thirteen complaint categories were modeled, with Heat/Hot Water much denser and more seasonal than many others.
A simple citywide volume forecast would miss the purpose of this problem: to rank and characterize risk across a very large set of distinct entities. Even a standard row-level classifier would be misleading without attention to when information became available and which negative examples belong in the scoring universe.
2. Building a usable training and scoring grid
Historical complaint data had to be reconstructed into a canonical, point-in-time structure: records grouped by building, apartment, category and quarter, with predictors available before the target quarter.
Feature families included:
- recent complaint counts, such as prior-quarter activity;
- longer lookbacks such as prior-year and older complaint history;
- same-category counts and counts from other categories;
- calendar/seasonal indicators and building attributes;
- contextual signals for combinations with limited unit-level history.
The modeling data distinguished historically observed combinations from category-cold-start combinations introduced to complete the prediction grid. This increased coverage but changed the class prevalence substantially: evaluation on only observed rows would overstate how dense the positive class was for the full universe.
An important distinction: “synthetic” here refers to candidate rows added for previously unseen category combinations, not fabricated positive complaints.
3. Modeling choices
The core model was a single LightGBM long-format binary classifier, with category included in the feature representation. This allowed shared patterns across complaint types while retaining category-specific characteristics.
Why this approach fit the problem:
- millions of sparse tabular examples with nonlinear interactions;
- heterogeneous buildings and complaint behaviors;
- useful lag and count features without requiring deep sequence models;
- a manageable, reproducible training and scoring pipeline.
I explored alternative grids and features, including category-specific history and cross-category context. Optuna was used to tune the boosted-tree model on a forward validation period. Prediction quality depended as much on target construction and evaluation discipline as on tree hyperparameters.
4. Time-aware training and leakage controls
A representative finalized experiment used:
| Phase | Time period | Purpose |
|---|---|---|
| Training | 2019-Q1 through 2025-Q3 | Fit the model on past quarters |
| Validation | 2025-Q4 | Tune and select the configuration |
| Held-out test | 2026-Q1 | Assess untouched future-quarter outcomes |
| Later retrospective | 2026-Q2 | Compare previously generated scores with observed results |
The model for the 2026-Q1 test was refit through 2025-Q4 after selection and did not use 2026-Q1 outcomes to tune hyperparameters. The later Q2 retrospective did not simply report a re-score using information from that quarter; it compared already-generated scores with subsequently observed complaints.
For this type of task, a random row split would be misleading: the same entities and adjacent observations appear across time, creating opportunities for temporal leakage or unrealistic estimates of out-of-sample performance.
5. Why multiple evaluation populations matter
For the 2026-Q1 test, the same trained model was evaluated on three related populations:
| Evaluation population | Rows | Actual complaint rate |
|---|---|---|
| Full augmented category grid | 5,309,629 | ~1.52% |
| Historically observed rows | 1,110,742 | ~5.14% |
| Category-cold-start additions | 4,198,887 | ~0.56% |
These populations differ in prevalence, which heavily affects the interpretation of PR-AUC, log loss, threshold precision and raw probability scores. The historically observed slice should not be presented as though it represents the full future scoring universe.
Held-out test metrics — 2026-Q1, full augmented grid
| Metric | Result | Interpretation |
|---|---|---|
| ROC-AUC | ~0.883 | Discrimination across positive/negative pairs |
| PR-AUC | ~0.272 | Precision–recall tradeoff at ~1.52% prevalence |
| Log loss | ~0.0568 | Quality of probability predictions |
| Relative improvement against prevalence-only log-loss baseline | ~27.8% | Better score than predicting a constant event rate |
The baseline-relative improvement is not business ROI, an increase in model accuracy, or a reduction in complaints. Likewise, F1 at a default 0.5 probability threshold is not especially informative for an extremely rare-event risk-ranking application.
6. Later real-world evaluation — 2026-Q2
The follow-up retrospective checked whether previously generated risk scores remained useful when compared with subsequent complaints.
| 2026-Q2 evaluated population | Scored records | Complaint rate | Top-decile lift |
|---|---|---|---|
| Full scored grid | ~7.0M | 0.85% | 6.3× |
| Historically observed | ~1.5M | 2.7% | 5.2× |
| Cold-start | ~5.5M | 0.34% | 4.2× |
The published project evaluation also recorded 23% usefulness relative to a constant-rate baseline for the full grid; the underlying briefing defines this as a score-based usefulness measure, not a 23% lift in business output or classification accuracy.
The operationally intuitive result is the 6.3× top-decile lift: complaint outcomes were meaningfully concentrated in high-scoring groups. That can support prioritization if reliability and workflow constraints are handled appropriately; it does not show that a field team actually adopted the system or that complaints were prevented.
7. Calibration, failure modes and guardrails
Two models can rank records similarly while giving very different numerical probabilities. Here, both properties mattered: who is riskier, and whether an assigned probability is reasonable.
Calibration checks included mean predicted rate versus actual prevalence, probability bins, decile gaps and category-specific reliability. The Q1 evaluation was broadly reasonable globally, with some underprediction in high-score cold-start/category slices. The Q2 retrospective exposed further operational cautions:
- Historically observed cases offered stronger, denser signal.
- Elevator and Outside Building cold-start groups were unreliable and should not be used for decisions without additional work.
- Heat/Hot Water remained useful as a ranker in observed rows but showed seasonality-related probability mismatch in warmer periods.
- The recommended option was broad scoring with category-specific exclusions and calibration guardrails, rather than blanket trust in all predictions.
These limitations are part of the project story, not disclaimers to hide. Identifying where models fail is essential to safe operational use.
8. Engineering and reproducibility
The pipeline brought together longitudinal open data, feature construction, grid augmentation, LightGBM training, hyperparameter tuning, batch scoring and reproducible evaluations. Related data engineering included public-data extraction and snapshot handling; the exact infrastructure varied by development stage and need not be presented as one monolithic product.
Technical themes: Python · SQL/DuckDB · parquet · time-aware feature engineering · LightGBM · Optuna · rare-event learning · PR-AUC / ROC-AUC / log loss · calibration · score deciles · operational guardrails.
9. What this work demonstrates
This project was not just a model trained on a large table. It required defining a prediction problem that matches a future decision, building a defensible scoring universe, preventing evaluation leakage, measuring performance at meaningful granularities, and distinguishing reliable output from weak segments.