‘we Don’t Touch Criticism’: How The Areto Ai Hides Abuse But Allows Free Speech With Mps

CANBERRA — Jacqueline Comer is the founder and chief product officer of Areto, an AI-powered app which POLITICO revealed Australian MPs were using to protect themselves against gendered abuse on social media.

In this interview, conducted by phone, Comer reveals that almost half of all comments to politicians’ social media posts meet the thresholds MPs set that merit hiding them because they are abusive. Areto’s data confirms Anthony Albanese’s warnings about the personalization of abuse at women MPs or, as Comer says, abuse “about who they are and their right to exist.”

This transcript has been edited for length and clarity.

Q: When was Areto created and what was it designed to do?

April 2020, and we are incorporated in Canada. Our mission was to make the online world more positive and inclusive. And the way that we were looking at doing that was by doing something about online harms. We started a nonprofit that was working towards gender parity in government in Canada. We were hearing barriers in the early mid 2010s, around [candidates] not knowing how to raise money for campaigns and we were able to put together workshops to remove some of those barriers, and then it was almost overnight in around 2016-2017 when we were meeting people to ask them to run, they were saying, “Actually, I don't want to put myself or my family in the line of fire online.”

Looking back, it seems like quite a benign time online, really when Twitter was fun. But we also saw high profile election candidates and elected leaders start to pretty directly and nastily attack their opponents on social media. It's nothing compared to now, and there was very little research on the problem at that time.

We were able to put together some hackathons and created a Twitter bot called ParityBOT that responded with positive comments to any abusive Tweets directed at women running for election. We did that for four elections: two in Canada, one U.S. presidential election in 2020, and then the New Zealand general election in 2020. And then from there, it was really you “know there might be a there might be a business in this.”

Q: How did the capability of the tool evolve?

The bot is dead. Long live the bot. That’s long gone. What we have now is a content moderation platform that uses some pretty sophisticated AI to automatically remove abuse and spam from your social media.

How it works is that spam is usually produced by bots, not real people, so we delete it. No feelings hurt, no risk of retaliation, and it all happens within three seconds of the comment coming in.

Abuse is different. Our users set the threshold for their own community: they tell us what’s OK and what isn’t. Anything they deem unacceptable is hidden from public view and stored in their Areto account, where they can still see it, undo the action and learn more.

We don’t block, and I’m quite strict on that. Blocking cuts your constituents off from important government information. We also don’t delete abusive comments. Our AI is very accurate, but it isn’t perfect, and you can’t undo a delete. You can undo a hide. We don’t want our system making a decision that could cause harm.

Hiding also works with the platforms’ algorithms. A hidden comment doesn’t get likes, so it isn’t promoted, and the person posting isn’t reinforced by their behavior. We’re using the platforms’ own systems to take as much abuse as possible out of circulation.

And we don’t touch criticism. You need criticism for a healthy democracy and for constructive conversations with your elected members around policy, so that stays up. What we hide is the racist abuse, the sexist abuse, the threats. Right now in Australia, we’re seeing that sexist abuse is higher than the racist abuse, things like misogyny, sexism, body shaming, transphobia.

Q: Do you have a perspective about whether or not the platforms are shirking their responsibility as publishers by not moderating to begin with?

Our focus is the people we protect and the evidence we can produce. Should we exist? The goal has always been for us to not be needed, but we saw this problem and this major gap, and that was even before moderation responsibilities changed with Meta and across TikTok and YouTube.

The interesting challenge we have is that social media was meant to connect us. Instead, it’s driving people out of public life. We saw that creep into candidate conversations 10 years ago. It’s much, much worse today.

Q: What are your metrics for success?

Personally, the metric I care most about is the number of emails and texts we get from elected members and their staff telling us things are better. They feel in control again, control of their time, of their audience, of their own social media.

And the numbers show why that matters. I can tell you that in Australia, from the beginning of May till the end of September this year, we analyzed, and this is for federal, state and local elected members and election candidates, we analyzed just under 1 million comments, and again, that's across public social media accounts, so Instagram, Facebook, TikTok, and YouTube, and moderated just under 500,000, so around 50% of all comments that the politicians we were working with received were considered abusive, too abusive for the thresholds that the politicians set on their social media channels.

The average sports team would have about 5% of all comments hit the threshold to hide. In a playoff series if we see 20%, that's really bad. And in public office, we're seeing about 50%. The higher the profile, the more you're going to get. If you're a woman, you'll get more. If you are not white, you'll get more. If you are gay or from the LGBTQ+ community, you'll get more.

So add on any intersections of that, and things get nastier and a lot more personal. And really, what we see is a lot of times for men in public office, the abuse that comes in tends to be directed more towards the post and the policy, as opposed to the abuse that comes through for a woman politician tends to be more about who they are and their right to exist.