Your doctor can’t tell you what a visit will cost. They might say something about “it depends on your plan” and you’ll get a bill in the mail weeks later. In 2026, why is this still a thing?
The sheer number of insurance companies and plans in the marketplace, all with their own rules for billing patients, makes this a really hard question to answer. Waiting weeks for a surprise bill in the mail isn’t a great experience for anyone, so Headway believes it’s really important to try and to answer this question correctly up front. As hard as it may be, wrong estimates, or worse yet, undercharging or overcharging a patient, can break trust and can ultimately prevent, delay, or worsen the care experience. Our goal is to push the envelope on what’s possible for upfront billing accuracy, minimize after-the-fact bills, and be a trusted benefits partner for our patients and providers.
The challenge of determining patient costs forced us to fundamentally reconsider our approach to benefits interpretation, and ultimately led us to machine learning. In the process, we learned a lot about the limitations of rule-based approaches, the inherent messiness of claims data, and how to efficiently scale from Jupyter notebook to production.
271 Fun in the Sun
To set context, behind the scenes healthcare providers and companies like Headway ask payers (insurance companies) for patient eligibility and benefits by sending “270” requests and receiving “271” responses (part of HIPAA’s standard electronic data interchange). There’s a spec, which means that it’s machine parseable, but the meaning of the data can still be ambiguous. Some information is structured; much is free text. A toy example:
Those with a discerning eye might already have questions like “What constitutes a specialist visit?” or, “Why would you pick one service type code over another?”. The answer, unfortunately, is that these are ambiguous, and different payers and plans can and do use them differently. While we do have other data sources (patients upload photos of their insurance cards, our ops team makes phone calls to insurers), the 271 is our richest source of benefit data at scale. However, any amount of ambiguous data ultimately means ambiguous billing outcomes; a non starter for our goal of building “wildly predictable” insurance.
Rules Don’t Scale
When Headway launched, we built a rules engine to interpret benefits from 271 responses like the one above. For a long time, they worked really well:
- Headway was serving mostly telehealth talk-therapy patients, a narrow slice of mental healthcare billing.
- The largest payers we supported followed similar recognizable patterns for encoding benefits.
But then the platform grew:
- More diverse patient care needs are being met (specialty care, …)
- More diverse providers are joining the platform (prescribers, …)
- More diverse payers and plans are contracted with us …
The rules engine was no longer meeting our needs. Adding rules (we had over 500 rule sets!!) became a losing game, and every failure was making it harder for patients to receive care. The rule-based system hit a plateau for several reasons:
- Humans can’t scale this. Staring at piles of failed claims to extract new heuristics is slow, noisy, and error-prone. And new rules might interact with old ones.
- No obvious grouping signal. Similar billing behavior hides behind oddities like member-ID prefixes/suffixes, plan numbers, group numbers, group descriptions, and so on. What defines a plan is different for each plan, and grouping by all possibly relevant attributes leads to impossibly sparse data.
- Varied scenarios also lead to sparse data when data is carved up by rules. Telehealth vs. in-person, treatment codes, provider license types, geography, …
- 271 responses contain structured and unstructured information. A lot of critical info about a copay could be tucked away in dense free text that is totally unique to the plan and can change at any time, and thus hard for rules to reliably use.
- And worst of all, sometimes the 271 is just completely ambiguous:
Which one applies? No service type, no notes, no context. With such ambiguous data, and a rule system at its limits, we reassessed the problem from the ground (truth) up.
(Cat)Boosting Accuracy
We placed a core bet: insurance companies are inscrutable, but they’re not random. If we’ve seen a plan behave a certain way before, we can predict how it’ll behave again. The historical record of claims (what we billed, what they paid) is a training set hiding in plain sight.
After exhausting hand-coded and even machine-generated rules, we turned our eyes to supervised learning. Our data is largely tabular (historical claims and appointments). Many of our features are high-cardinality categorical (hundreds of thousands of plan/group identifiers and descriptions).
That profile pointed us to gradient-boosted trees; ensemble models that stack many small decision trees to make predictions. If you’re not familiar, you can think about this like a flowchart or series of if/else statements. “Is telehealth? Yes → go left. Is payer Aetna? No → go right.” Each new tree focuses on the examples the previous trees got wrong, and you sum their outputs. It’s iterative refinement, and it often achieves state-of-the-art results on structured data like ours!
The open source library CatBoost was a natural fit: it handles high-cardinality categorical features natively, without much fuss about feature encoding. In head-to-head tests, CatBoost outperformed other boosting models (e.g. the well-known XGBoost) on our data.
The model’s job: given everything we extracted from the 271, plus context about the appointment, pick the right cost-share category and amount. Is this patient paying a $40 copay? 20% coinsurance? Full deductible? The answer hides somewhere in the intersection of plan structure, provider type, service codes, and the payer’s own idiosyncratic interpretation of their own rules.
In early evaluations we hit a surprising finding: for our feature mix and dataset sizes, CatBoost on GPU trained much faster but produced worse models than CPU, even after matching flags and hyperparameters, and giving GPU an advantage (more data, broader hyperparameter search)!! The GPU version uses different algorithms under the hood in order to parallelize; for our feature mix, as the above link explains, that trade-off hurt quality. CPU won. This was a great reminder not to make any assumptions. In another universe where we just shipped with GPU support because that’s “the fast” thing to do, we’d have missed this impact.
Ground Truth Is a Lie (Sort Of)
Supervised learning needs labels. Ours came from ERAs (Electronic Remittance Advice), the responses payers send back after we submit claims to them, telling us what they actually paid and what the patient owes. In theory, this is the cleanest signal you can get: not what the payer said they’d do, but what they did.
In practice? The ground truth was lying to us.
Some payers submit claims with the wrong patient responsibility codes. A visit that should’ve been labeled as “patient owes deductible” would come back coded as “patient owes copay”, and our model would learn the wrong lesson.
Triaging inaccurate predictions by hand, we started discovering noise from the edges of the system:
- Stale point in time data. Historical data revealed quirks and edge cases like not having fresh enough benefits on file for a given visit. To the model, cases like these looked impossible to solve.
- Eligibility denials. A claim gets denied because the patient’s coverage lapsed, not because we misread their benefits. That’s not a billing interpretation error, it’s a coverage error. But it looks the same in the data.
- Misrouted claims. Sometimes claims get sent to the wrong payer entirely. Again, this rejection has nothing to do with our benefit prediction.
Cleaning this up was the simple repeated process of backtesting, triaging, cleaning, repeating. We built filters to exclude claims with known data quality issues, flagged suspicious patient responsibility code patterns for manual review, fixed upstream bugs, and tightened our benefit-fetch timing. Each fix shrank the “impossible to predict” bucket and made the remaining signal cleaner, allowing us to help more patients. The model improved not because we made it smarter, but because we cleaned its inputs.
From Notebook to Nightly Deploys
Like many projects, the model started in a Jupyter notebook. Once we saw promise from our prototype, we moved into a Metaflow run via Outerbounds and unlocked a better way to work:
- Onboarding. folks joined the effort and didn’t need any local configuration to start training and evaluating the model immediately.
- Remote execution + more compute. Our MacBooks got us reasonably far, but to fully utilize our historical data we needed more memory + CPU. Parallel experiments also came for free.
- Versioned artifacts. Early code to solicit experiment descriptions, hyperparameters and scores meant a clean history of experiment performance.
- Not thinking about infrastructure let us focus on the work. We did iterative, methodical feature work. One change at a time, measured impact, kept the wins, reverted the regressions.
As features and data grew, training time crept from minutes to hours. We prototyped splitting our large multi-class classification model (predicting copays and coinsurances with many different amounts, and other labels) into smaller, single-purpose models trained in parallel. Quality stayed the same; training time dropped to roughly a third!
We tuned decision thresholds automatically on holdout sets; no more hand-picking cutoffs like in the rule sets. We used SHAP values to understand which features actually mattered, and pruned the rest. Most importantly, we used out-of-time backtesting: instead of random train/test splits, we asked “would this model have gotten last month’s claims right?” That’s the metric that matters, because it matches how the model would be used in production.
With performance looking compelling, it was time for rubber to meet the road. We decided to run a shadow mode experiment in production and set up a nightly flow to train and deploy the model to AWS SageMaker, added guardrails for data and model quality, and instrumented basic observability at the model and API layers. SageMaker was clunkier than we’d hoped; wiring up a custom Docker image for inference meant fighting the framework’s assumptions about model packaging. But the deployment primitives (blue/green rollouts, versioned endpoints, built-in metrics) ultimately outweighed the friction.
We wired up our billing product to ask the model for its opinion at critical billing junctures, and logged inputs/outputs. We stored all inference requests and responses, and ingested them into Snowflake, which made detecting train-serve skew trivial. Claims take a long time to come back adjudicated, so while our shadow mode ran for a week, it would be 40 days before we could view final results.
Coming out of the Shadows
When the results came in, they showed that ML could serve as the authoritative billing source for 70% of our total appointment volume, and in total we measured a 40% reduction in billing interpretation errors platform-wide, more than any heuristic (or stack of heuristics) had previously achieved.
The gains weren’t evenly distributed. For major payers, our rules were already high-quality; we’d invested years of human attention into understanding their patterns. The model’s biggest wins came from the long tail: small regional TPAs, obscure employer plans, the thousands of payers that each represent a tiny slice of volume but collectively add up. It also learned subtleties in a plan’s different benefits, where e.g. a copay might change based on provider type. These are exactly the plans where ML shines — whereas human attention was only focused on the head, the model optimized for every plan where we had data.
We rolled this out across the platform in late 2025, and finally closed the loop: fewer incorrect bills, fewer refunds, fewer trust-breaking moments. Better yet, unlike our rules, the model will adapt as plans, benefits and claims change over time. This change frees our operations staff to focus on the truly hard corrective cases, and is a big step forward in our patient billing experience.
Takeaways
- Backtests that mirror production build real confidence. Optimizing for “would this have helped last month?” beats optimizing for accuracy on a random holdout.
- Make iteration cheap. Metaflow/Outerbounds, remote execution, and versioned artifacts paid for themselves in development velocity.
- GPU is not a free lunch. For our data and feature mix, CPU consistently produced better CatBoost models than GPU, despite slower training.
- CatBoost shines on tabular data with high-cardinality categoricals. Minimal preprocessing, solid out-of-the-box performance, and interpretable feature importances.
What’s Next
We see many adjacent problems in the insurance space. Claims routing, benefits routing, eligibility interpretations, and beyond. If high-leverage ML on messy, real-world data sounds fun and you’re interested in making mental healthcare better for everyone, come build with us.
