Landlord (illegally?) Witholding Pet Deposit

Hi everyone- wanted to get some thoughts on an issue we're having with a landlord in New Jersey. We rented a townhouse for six months while we waited for our new home to be built. We have kids and pets (dog and cat). The landlord agreed that our pets could come if we added a pet deposit which was part of the security deposit plus $100 extra per month to our rent. Their realtor said the pet deposit was refundable and in the lease it says it is part of the security deposit. Fast forward, we moved out. The landlord was awful the entire time we lived there not responding to messages and taking weeks to over a month to respond to issues about broken appliances, etc. She came over an hour late for the final walkthrough, agreed we'd get our full deposit back, sent me a text (b/c I told her I wanted it in writing) indicating that. She had 30 days to return the deposit and didn't. Finally on day 32 after multiple outreaches from our relator and me, she sent the security deposit back minus the pet deposit. She tried to say the pet deposit was nonrefundable. Then we pushed back and said, no it wasn't, she said that there was damage to the floors. Then our relator sent her a demand letter saying we'd file a claim with small claims court if she didn't send it back. She now is trying to say that there was damage to the floors (she's saying almost $2K worth) that was never noted or said to us until now. Should we pursue the claim to get our $500 back? Per NJ law I believe we can get double so would be $1000. Or just cut our losses?

Thanks for reading!

location: New Jersey, USA