Boss Threatening Me For Trying To Use Fmla Leave

Location: California

Long story short (I hope):

I work for a "small" company (100+ employees but the main office is just me and the owner). I am the everything guy (HR, Accounting, Payroll, etc.). 6 months before my first child was born, I told the boss my wife and I were expecting and thought he would hire help, he didn't. Cut to 1 week before my baby was due, the owner comes in and says "Oh, my wife mentioned that your generation likes to be involved in the kid's life, but we have important stuff here to do, can't your wife just take care of it?"

I was pretty shocked and didn't really know how to respond so I said I was hoping to take some time off and receive my California Paid Family Leave too. He said "Well we're a small office so if you leave, I don't know if your job will be here when you get back..." which was language our attorney gave him years ago for a work-stress disability leave case that extended past FMLA leave time. So I said I could work reduced hours to help keep things flowing while still receiving reduced benefits. I am hourly so I said 10 hours per week, maybe 15 tops would let me get my benefits in full so I could come back to work within 2-3 months. He agreed to this.

Cut to 5 months later:

The first 2 weeks postpartum were 20-25 hours, and since then have averaged around 35+ hours. He has consistently assigned me extra work and written emails basically blaming me for every delay even when it's not my fault. The last two weeks I have worked over 40 hours each week and he had the gall to ask me when I'm coming back full-time. I am basically doing 3 days in office, 2 from home. He has constantly threatened me with his "small company" line. Last week, the day after he asked about me being "back" and I told him I still had benefits to collect because he has given me so much work, he posted a meme on his "humor board" in our office building that had a boss telling an employee "We've decided we'd like you to work from home...preferably for someone else."

My question: do I have a case or do I have to actually formally request FMLA and then have him fire me when I get back to have a case? I have been applying for other jobs for almost 2 years because the owner has been so awful and finally got my first job offer at basically a 40% pay cut which wouldn't pay my bills. So I don't want to lose my job unless I have a way to recoup my losses (via lawsuit/labor board fines/etc.).

Added context: He has treated me poorly since he took over for his dad 2.5 years ago. He does most things verbally because he comes from corporate-life, but he has:

* made racial comments about my wife who is mixed race

* allowed my predecessor to viciously bully me non-stop for 9 months while "training" me because she was there 40 years and we needed her knowledge of the old system/accounting methods. He would laugh it off and say "You'll have such thick skin once she's gone"