Ca Neighbor Is Threatening Legal Action Over A Fence Dispute But The Survey Shows He Is Actually Encroaching More Onto Our Property Than We Are On His

Location: Oakland, California
My partner and I (both 25M) just purchased our first house/ multi unit investment property. We put our life savings into it with plans to live there for years to come. Going into the sale, we knew there was a potential dispute going on with a shared fence, as the seller mentioned that the neighbor said that there was a "slight encroachment" from our backyard into his.

As soon as we started moving items in, the neighbor spotted my partner helping me park in our tight driveway (which I gave up on as our car was too big and had to park across the street), invited himself into our gated driveway, and started trying to show him the fence issue. After parking across the street, I politely told him it wasn't a good time as we had our cars open with valuables inside. I got his business card and told him I’d reach out.

During the next few days, he called and berated our Realtor, saying it was "terrible" to allow us to buy a house with pending legal action. I didn't want him to have our phone numbers because of this, so I reached out via email:
"Hi [Name], Great to meet you the other day and excited to be neighbors! We didn’t get too much info from the sellers but want to work with you on finding a fair solution. I’ve attached the 2022 survey that the seller sent us. Do you have any other documents that we should review related to the potential encroachments? How are you hoping that we’ll be able to resolve this?"

He responded:
"I spoke to your buyer's agent, [Realtor NAME] XX, yesterday, and he was somewhat "Aloof" about the 5154 20-inch fence encroachment into my property.

In my 40+ yr professional opinion as a Realtor, [Realtor NAME] did a good job of securing [ADDRESS] on your behalf but dropped the ball completely by exposing his client to pending legal action, which was totally avoidable and unacceptable, and which exposes EVERYONE to litigation over a simple resolution that has been pending for ten years!

I'm confident we can reach a mutual agreement to resolve this encroachment, but first it will be necessary to review your correspondence with my attorney, [Attorney NAME], to ensure this matter is settled in a proper legal manner. I'm sure you understand."

After a 3-week delay, he just sent this:
"I've consulted with [Attorney Name], and he has requested complete contact information for you & your partner for the ownership of [Address]. Once received, he will draft a simple agreement between us addressing the cost and responsibility of removing the rear 100 ft of the fence encroaching on my property by 20+ inches."

For additional context here’s what the boundary line survey said:
His encroachment on us: He is encroaching on our property over a 100 ft span by about 3.5 to 6 inches.

Our encroachment on him: We are encroaching on his property over a 30 ft span by 0.56 to 12.05 inches.

He is accusing us of a 100 ft / 20+ inch encroachment. Every time he reaches out, the amount of land he claims we are taking increases. My partner and I don't care about the 20-25 sq ft of land we are actually potentially encroaching on and are happy to fix it, but he keeps on demanding more and more land. I feel like he is trying to take advantage of us because we are young first-time buyers.

I don't want to go to court and pay huge attorney bills over 20 sq ft of land.

My questions:
Should I write a response explaining the actual math from the survey (pointing out he encroaches on us more than we do on him) and propose we just split the cost to rebuild the fence exactly on the boundary line?

Do I have to give his attorney our phone numbers? I am scared he will just use it to harass us, and I strongly prefer everything in writing via email.

Because he keeps tossing around the threat of his attorney and litigation, do I need to hire my own real estate attorney now, or can I still try to navigate this boundary line agreement myself?