New

Medicare Shoppers Face A Hidden Al Problem This Open Enrollment

The most expensive Medicare decisions do not feel like decisions. They feel like reading one page, getting one clean answer and moving on.

That is how somebody ends up in a plan that drops her cardiologist in March, or else she never learns she qualified for coverage built for her exact situation.

Open enrollment opens Oct. 15 and closes Dec. 7, and what you choose carries through all of 2027. A beneficiary can pick from 32 Medicare Advantage plans with drug coverage this year, a count that excludes employer plans and Special Needs Plans, according to KFF.

Sorting that out used to mean a broker at your kitchen table, a stack of mailers or an afternoon on the government’s Plan Finder.

This fall, far more retirees will type the question into Google and read whatever the AI summary tells them.

A Medicare data researcher spent two months testing what those systems actually return, and found that the answer can turn on a single word the reader happens to choose.

Why Medicare questions are hard for AI to answer

Medicare is not one product. A beneficiary can choose Medicare Advantage, a Special Needs Plan, a standalone Part D drug plan, a Medicare Supplement policy or Original Medicare, and the right answer depends on county, plan year and personal circumstances.

Retirees are already bringing those questions to AI. More than 20% of U.S. adults use AI chatbots for health questions at least sometimes, while only 18% rate the answers as either extremely accurate or very accurate, Healthcare Dive reported, citing a Pew Research Center survey.

That wary response to AI answers is justified. AI chatbots answered everyday health questions accurately 76.2% of the time, representing roughly double the error rate of human physicians, according to a Penn State study covered by Medical Xpress.

A Medicare data researcher spent two months testing how well AI systems work to inform beneficiaries, and found that the answer can turn on a single word.

What the Yuma County search test found

David Bynon has published Medicare data since 2012 and now runs a measurement project called the Medicare Visibility Monitor. In Yuma County, Ariz., he typed the kind of query an ordinary beneficiary would: medicare plans yuma county az.

More Medicare/Medicaid:

Conventional Google results read that phrase as Medicare Advantage and filled the page with Advantage advertising and insurance carrier web pages, Bynon told TheStreet. Somebody who did not already know the coverage types could reasonably conclude that Advantage was the entire menu.

After Google’s late-August update he ran the test again. The broad query produced an AI Overview centered on Medicare Advantage, and changing one word to medicare options yuma county az expanded the answer to include Advantage, Special Needs Plans and Medicare Supplement coverage.

“A consumer should not need to know the taxonomy of Medicare before asking what their Medicare choices are,” Bynon said.

That result is what pushed him to build a resource laying out every Medicare coverage type available in a given county. He put it on MedicarePlans.com, his experimental publishing site, then moved the implementation about two months later to Medicare.org, the production site he manages.

He then gave the system a realistic person: turning 65 next month, covered by Medicaid, living with type 2 diabetes and taking Metformin, and unsure what to do.

The AI Overview did well at first. It recognized that holding both Medicare and Medicaid mattered and identified Dual-Eligible Special Needs Plans, known as D-SNPs, as the plan class that fit.

Then it slipped. Its recommendation led to a page listing standard Medicare Advantage plans rather than the D-SNP choices the system had just identified as relevant, according to Bynon’s account and screenshots he supplied.

Nothing in that answer was fabricated. The plans were real and the benefits were real, but a beneficiary following the recommendation would still have been shopping the wrong aisle.

Bynon calls it a context-continuity failure, where a system resolves who you are and what you need, then drops that thread when it picks the sources it hands you.

One stage of the test went better. When the beneficiary said he still did not know which plan to choose, the system stopped recommending and pointed him to the State Health Insurance Assistance Program and Medicare.gov.

5 ways AI answers go wrong on Medicare

Mixing plan years, so 2026 costs get quoted for a 2027 plan
Reading “Medicare plans” as Medicare Advantage only, dropping Part D, Supplement and Original Medicare
Misassigning county service areas, so a plan looks available where it is not
Inventing benefits such as buyback allowances or over-the-counter credits that a plan does not offer
Losing track of your circumstances between diagnosing your situation and recommending where to go next

Source: Trust Publishing Institute baseline report, February 2026, and Bynon’s Yuma County testing supplied to TheStreet

His institute put the average hallucination rate in Medicare plan explanations above 27% in a 50-query sample. That figure comes from Bynon’s own testing, but the report does not publish the queries or the scoring method behind it.

Medicare enrollment runs Oct. 15-Dec. 7, while AI answers may shape choices.

MoMo Productions / Getty Images

Who is feeding AI its Medicare facts

This is the part of my reporting that should matter most to anyone shopping this fall.

Bynon argues the root problem sits upstream. The Center for Medicare and Medicaid Services (CMS) publishes authoritative plan data, he said, but spreads it across datasets and interfaces that are not organized around the way consumers actually ask questions.

That gap, in his account, leaves AI systems leaning on private publishers to resolve those questions and present the answers in a form built for retrieval.

To measure who that is, Bynon built a monitor that tracks 2,146 CMS plan IDs and records which publishers get retrieved and cited across Google, Bing, ChatGPT and Copilot. Google’s AI cited Medicare.org for 2,011 of those plan entities and generated 6,169 citations, his September export shows.

Medicare.org is a site Bynon manages. It is owned and operated by Health Network Group LLC, an Allstate (ALL) company. HealthCompare, a licensed insurance agency, may compensate the website when a reader enrolls through Medicare.org’s phone number or MedicareEnrollment.com, according to the site’s disclaimer.

The disclaimer adds that the arrangement does not influence what Medicare.org publishes. The roles are worth separating too, because Bynon runs the publishing technology and editorial systems while the enrollment arrangement belongs to the website’s owner and its licensed agency partner.

“I am compensated by HealthNetwork Group/Allstate to manage its web property,” Bynon told TheStreet. “I have no stake in its outcomes.”

I read the September export against that disclaimer twice. The publisher supplying a large share of the factual assertions inside Google’s Medicare answers is owned by an insurer and can be paid when a reader enrolls.

Bynon is careful about what his numbers prove. “I cannot establish that a particular markup change caused a particular ranking or citation outcome,” he said, describing the work as live-site observation rather than controlled experiment.

One finding in the export deserves a retiree’s attention anyway. MedicarePlanLookup.com was almost absent from conventional Google and Bing results, yet ChatGPT cited it for 1,394 plan entities. MedicarePlans.com ran the same way in reverse, drawing 1,051 ChatGPT citations against 114 from Google’s AI.

“I therefore no longer treat search visibility and answer visibility as interchangeable measurements,” Bynon said. A site you would never find on page one can still be shaping the answer on your screen.

How to pressure-test an AI answer before you enroll

None of this means dropping AI during open enrollment is a good idea. It means treating the answer as a place to start rather than a verdict.

Checks worth 10 minutes before you pick a plan

Ask about your Medicare options rather than Medicare plans, so the answer covers Advantage, Part D, Supplement and Original Medicare
Ask which plan year the answer describes and discard anything still quoting 2026 figures
Say plainly that you have Medicaid, a chronic condition or nursing home care, then check that every plan you are shown actually serves that group
Look up any plan by name on Medicare’s Plan Finder with your own drugs and pharmacy entered
Confirm directly with the plan that your doctors are in network for 2027, rather than relying on a chatbot’s provider-network information
Get free help from your State Health Insurance Assistance Program or 1-800-MEDICARE

Bynon expects Plan-ID queries and AI assistants to take over more of Medicare shopping through 2028, and he has argued to federal regulators that the ecosystem needs a neutral structured publisher. He runs one of the most-cited ones.

Until CMS publishes plan data in a form the machines can read cleanly, the retiree at the kitchen table is the last line of verification. Ten minutes on Plan Finder and one call to the plan cost nothing, and they are the only part of this process that nobody is paid to influence.

Back to Listing

credit: