Billionaires Are Fleeing California. The Tax Agency Is Preparing To Chase Them.

SACRAMENTO, California — California tax officials have spent decades pursuing wealthy residents who claim they’ve moved out of state. But if voters approve a controversial tax on billionaires next month, the Franchise Tax Board, one of the country’s largest tax collection agencies outside of Washington, will have to do that on an unprecedented scale: determining which billionaires still live in California, valuing much of their worldwide assets and collecting a tax expected to raise about $100 billion.

The billionaires have every incentive — and ample resources — to fight back, and they have made it clear they will not willingly part with their money.

Sergey Brin, the centibillionaire co-founder of Google, began cutting ties with the state late last year, as did his old business partner Larry Page. Peter Thiel, the PayPal and Palantir founder, said he was moving to Miami and pointed to California’s hostile tax environment as the prime reason. Car loan magnate Don Hankey picked up and headed to friendlier tax climes in Nevada, telling news outlets on his way out that he felt like he “wasn’t wanted” anymore.

Some of the state’s leading tax lawyers claim to have helped at least a half dozen other unpublicized billionaire clients leave the state.

At stake is a tax bill equal to roughly a third of the state’s annual budget, setting up an inevitable clash between a small army of lawyers promising the billionaires they can help save every dollar and a tax agency regarded as an extremely dogged — and competent — pursuer of tax debts.

“The FTB is probably the best equipped to handle something like this,” said Shail Shah, a former auditor at the FTB who now works as a principal at a tax advisory firm. “But I don’t envy them the job.”

There is no one-line definition of what it means to be a Californian.

In 1991, a California inventor named Gilbert P. Hyatt — set to make $150 million dollars from a microprocessing patent — sold his house, rented an apartment in Nevada, opened a bank account in his new state, and re-registered his vehicles. After all was said and done, he declared that he was no longer a resident of the state, and made his motive plain: He was about to receive an enormous windfall, and he didn’t think California was entitled to it.

The state disagreed. After initiating an audit, FTB inspectors deployed aggressive investigation tactics in their efforts to prove that Hyatt’s move was a fraud, rooting around in his trash for evidence, “rifling through his private mail” and sending letters to his rabbi. The investigation ultimately turned into an epic, quarter-century long tax battle, as Hyatt alleged that auditors had made antisemitic comments about him and challenged the case in court. A lawsuit surrounding the case came before the Supreme Court three separate times. But after countless appeals and court hearings, Hyatt still ended up paying $11 million in taxes and penalties (although he continued to fight that assessment).

“If the question is, ‘What is the extent that FTB will go to tax someone they think has money?’” said Michael Cataldo, a shareholder with Cataldo Tax Law. “The answer is pretty far.”

There is no doubt that billionaires are taking it seriously. Tim Noonan, a state tax lawyer, said he is seeing “many taxpayers who fit the bill” taking steps to move. The level of urgency has ramped up since Prop 40 qualified for the ballot, even if some billionaires might not have thought much about it eight months ago.

But as Hyatt learned, shedding a California tax residency is a challenging task. And as billionaires prepare to leave the state, they will likely take many of the immediate steps that Hyatt did. They will get a new driver’s license, open a bank account in their state of choice, register to vote again, perhaps even re-register their Land Rovers in Nevada or Florida or Texas. But it will not be enough.

In a statement, an FTB spokesperson declined to comment beyond stating that officials “continue to monitor” the billionaire tax measure’s progress and that, if approved, the agency “will need to be ready to administer the new tax program within six months of its passage.”

But California has been building up case law surrounding the issue of residency and taxes for decades, according to residency tax lawyer Christopher Manes, since steel magnates from Chicago in the 50’s, 60’s, and 70’s started wintering in Palm Springs to flee the Illinois snow and wind. At the time, the highest net worth residency audits typically surrounded wealthy snowbirds who accidentally spent too much time in California. It is only in recent years, as Silicon Valley engineers and business people found themselves on the verge of massive wealth from an IPO, that there has been a growing culture of wealthy people fleeing the state to avoid taxes.

In New York, another tax-hungry state, residents face a bright red line regarding days spent in the state. But California’s rules are more subjective and arguably more rigid.

Over the years, the rules surrounding residency in California have been distilled by a handful of significant cases. In a 1975 case, Klemp v. Franchise Tax Board, a couple from Illinois avoided becoming California residents even though they spent more time in their vacation home in Palm Springs than their home in Chicago, because their primary business connections and life remained in the Midwest. In a separate 1992 case, courts ruled that people who were in the state for a “temporary or transitory” purpose would not be considered residents. In the 2003 case of a rancher named Stephen Bragg who claimed to have moved to Arizona, the courts established a series of residency determinations that tax lawyers now refer to as “the Bragg factors.”

The goal of the Bragg factors is to establish someone’s “center of life,” an informal concept that, in California, is far more significant than a driver’s license or voter registration or home address. And meeting those factors (including where your credit card transactions originated from and where your gym or church or doctor are located, among many other measurements) is only “necessary, but not sufficient,” as Manes put it. FTB investigators, he pointed out, will also look at where your business interests are located, where your spouse and children live, where you keep your collectible objects or items of value, or even where you have most clothes.

Many of the Bragg factors fall apart somewhat when applied to the lifestyle of billionaires, who typically have homes all over the world and may not visit a neighborhood gym. But their business interests and entanglements in California could prove the most compelling evidence for the FTB, likely one of the reasons that the Yes on Prop 40 campaign has been so insistent on pointing out any time that Brin spends at Google, intent on illustrating that his move out of state is a farce.

“You could buy the Palace of Versailles,” Manes said. “But you have to actually leave, you can’t pretend to leave.”

According to Shah, there are three buckets of billionaires in their approach to Prop 40: those who believe it will not pass and see no reason to worry; those who believe that they had to flee the state before Jan. 1 of this year, the measure’s cutoff date; and those who believe that a lawsuit challenging Prop 40’s retroactivity clause will make Nov. 4 — the day voters could approve it — the real drop-dead date.

That divergence of opinion is evidence of the broader reality surrounding Prop 40, and another challenge for the FTB — no one is exactly sure how it will work.

Betty Yee, who chaired the Franchise Tax Board as state controller, supports a wealth tax in principle but said it would be enormously challenging to implement. In an interview, Yee said officials should already be gaming out what that might look like, even if polls suggest the measure could fail.

“If I were still controller, I’d have the Franchise Tax Board in a war room right now trying to figure this out,” Yee said.

Why is that? “Oh my God,” Yee said, before launching into a list of the thorny considerations. Many of the assets billionaires are still holding onto haven’t been valued yet. Bean counters would need to decide if they’re looking at shares of ownership or estimates of economic interest. They’d need to set up a process for appeals. If an affected taxpayer gets divorced, she said, “Heaven help us.”

And the tax would almost certainly face legal challenges, Yee said, saying a state wealth tax “automatically triggers, in my mind, constitutional issues” — another hurdle among many.

Voters appear to have similar fears. A POLITICO poll from late last winter indicated that a majority of voters believe Prop 40 would drive billionaires from the state. Gov. Gavin Newsom warned that many more billionaires were fleeing in anticipation of the tax, and longtime Newsom pollster David Binder, in a poll for one of the committees formed to combat the wealth tax, found many voters believe “it is likely unconstitutional and could mean years of litigation funded by taxpayers.”

Beyond those challenges, the FTB, for all its reputation as a tax hound, has only ever had to collect personal income and corporate tax. Prop 40 is not, in fact, a tax on personal income, but rather a levy on all personal property and accumulated assets. A billionaire's art collection would qualify, which means that the FTB would suddenly find itself as an asset evaluator as well as a tax agency.

It makes for an extremely precarious position for a government bureaucracy – trying to learn a new method of taxation on the fly and conducting residency audits of a number of billionaires who are unlikely to be cooperative, all while Prop 40 is tied up in a series of legal challenges that could undermine the entire process.

“This will be asking an agency to learn the technicals in a very short amount of time to do something they've never done before,” Shah said. “It's a big ask.”

Or as Yee put it, “It could collapse under its own weight.”