AI Detection Rules Keep Changing. Trust Doesn’t.

Last week, Substack launched its AI detection feature in partnership with Pangram — the shift the company previewed in co-founder Chris Best’s post, “Against Claudefishing.” The tool lets anyone scan a post, Note, reply, or comment of 100+ words for an estimate of how much was written by hand versus with AI help. Best was upfront that the scan measures whether AI touched the text, not whether real care went into it — and Substack added a “How I make this” statement option so creators can explain their own process, and a way to contest a scan they believe is wrong.

I’ve researched, experimented, read all the POVs, and rewrote this article at least four times. It raised questions and ideas and presented the opportunity to rethink how I am using AI. This happens regularly due to the constant changes and what people perceive as important. In this article, I will share my thoughts and an overview of the changes.

I ran experiments with my own content through Substack’s new AI detector. One instance was pulled from an interview transcript — 100% my content, my questions, my listening. I had AI help me pull the highlights, then I wrote the copy myself, the way I always do.

It came back 100% AI-written.

I ran another piece, one of the Sustainable Rhythm articles I’ve been publishing all summer — work I sat with, argued with, rewrote three times. AI 39%, Human 61%.

I wrote an article using AI with a basic prompt, an article I would not share, and the result… 100% human. The one piece of actual AI slop in the batch is the one that passed.

None of those numbers are true.

It’s not really about Substack

Here’s my prediction: this isn’t solely a Substack story. This is going to be everywhere. Every platform is going to have some version of a scanner, it’s already happening. Creators are going to spend a lot of energy either performing for it or panicking about it. That is a choice.

These detectors look at how the words were assembled. They have no way to see the thinking underneath. The years of experience and creativity a person spent living the material before typing a word. Pangram can tell you about the structure of the content. It cannot tell you whether the person writing it had to dig to get there.

And on Substack, a platform built on writers and journalists caring deeply about their process, I envision we’re going to see this tool get weaponized against people, or used to make split-second judgments about what’s “good” based on an arbitrary number from a machine most of us don’t fully trust in the first place. That’s the part that concerns me more than the scan itself.

How I actually work with AI

I want to share a bit about my process, because it’s the argument.

I don’t open a blank AI chat and ask it to write something. I start in Gemini Notebook, where I’ve created what I call living libraries of my own content. Years of interviews, articles, training materials, and my stories. When I draft, I’m pulling from what I’ve already created, not from the internet. The first pass usually comes from that library, or from me talking, in my voice, into a voice recorder, usually in Claude or Perplexity. That is how this article started.

You cannot prompt “make this sound like me” and expect anything of value to come back. Without providing context and teaching AI tools who you are, about your audience, and what your voice sounds and feels like, it’s word jumble. AI is built to look for the next logical word.

My entire methodology is built on this premise. Tried & New isn’t about generating new content. It’s about excavating what you’ve already built, organizing it, and being able to see it clearly enough to name it. It’s an iterative practice that evolves. Eight months ago, I wrote what I thought was a complete eight-article series covering my methodology. This summer it became ten, not because the work changed, but because Claude pointed out that I had not named all of my work. I excluded three things I’d been doing all along. I wasn’t looking for new material. I was looking at what was missing from my services or my messaging. The whole time, the missing pieces were already sitting inside the work I’ve always done. And it was AI that helped me see the patterns in my own work that I was too close to see.

No detector is built to see that kind of depth. It’s not designed to look for it.

---

Know someone who would enjoy this article? Click below to share.

---

Why I became disillusioned with LinkedIn

I want to give Substack’s founders some credit here, because I understand exactly what they’re trying to protect. I’ve been active on social media since the beginning. I have stepped away from Facebook, Instagram, and Twitter. LinkedIn is where I have spent most of my time over the past few years. It’s become increasingly harder to be visible. They built AI into the posting process and the algorithms. This amped up spam and streams full of what Chris Best coined as Claudefishing. It became a waste of my time. It was a large part of why I decided to spend more time here on Substack.

I’m still on LinkedIn, but I’m not investing as much time or content there.

And… LinkedIn just followed suit

As I predicted a few paragraphs above. Last week, LinkedIn announced its own version of the battle against AI slop. Chief Product Officer Hari Srinivasan said the company is rolling out a “Seems like AI slop” button. Users can flag any post or comment that reads as fully AI-written, right from the three-dot menu. A flagged post takes a hit in the algorithm, similar to clicking “not interested,” and the flags feed LinkedIn’s backend classifiers so it can catch more of this before it ever hits a feed. Creators will also start seeing a private dashboard alert if enough people flag their content as sounding artificial.

The more telling move, to me: LinkedIn is canceling its own “enhance with AI” feature. That thing that rewrote people’s posts into generic slop. It’s being replaced with a simple proofreading tool that fixes spelling and grammar without touching anyone’s voice.

This validated for me that, although I have mixed feelings, I like that Substack allows creators to choose how they interact with scanning tools. Substack clearly rattled LinkedIn into acting, but LinkedIn’s version doesn’t put control into creators’ hands. LinkedIn is deciding for you.

NOTE: LinkedIn must be rolling this out. I do not see it yet on my account. Be it’s coming.

Trust is what really matters

This is the direction every platform is headed. We have no control over that. So far, Substack is doing it better and named TRUST in their announcement.

Our unfair advantage is discernment and listening to our instincts. I don’t need a tool to tell me something was written by a robot, and neither do you. I also don’t care if people are using AI. I do care if they are handing over the practice of writing and creation to a tool with no soul. I know from personal experience that there are many ways to work with AI to become a better writer and to partner and create things you never considered.

Most people focus on productivity and automating tasks. I have never taken that direction. I curate my own content. I want to spar and work with AI. I enjoy pushing past the layer of AI pond scum, that top layer that most people never go beyond. That is where the fun and actual transformation happens.

I also have a very simple business model. I don’t need to automate tasks. But if I did, I would do the groundwork first with solid processes in place and a check and balance system so I could trust that the tools were doing what I wanted them to.

I hosted a roundtable last week with 15 women, and trust was the center of the conversation. The question we started with: “What makes us trust something, and how do we build trust into what we create?” The responses were consistent: being present, smaller groups, being in a room together. All of it is the kind of trust that disappears the moment we hand the whole process over to tools.

And let’s not lose the irony about AI detection: these detection tools are AI calling out AI. They cannot assign a score to real life. They cannot initiate trust.

And now it’s law, not just a platform feature

I took most of Friday off and ran out of time before I could send this officially on a Friday. When I sat down early on Saturday morning to send it, I learned that the ground had moved again. So I shut my laptop, and I’m sharing this now.

Yesterday, August 2nd, the EU’s AI Act transparency rules went into effect. AI systems are now required to tell people when they’re talking to AI instead of a human. Deepfakes must be labeled. AI-generated content has to carry a machine-readable mark so it can be detected. This is an EU law, but it impacts everyone. It stands regardless of where the content was created and is in effect NOW. If you want the primer, the European Commission’s own explainer is here.

I’m not going to unpack the whole thing here. There is a lot to cover, that’s its own article, perhaps for someone else to write. But the message here is that this is happening at the level of law, not product features. Substack scans for trust. LinkedIn scans for slop. The EU is now requiring disclosure by regulation. Three different places, three different tools, all reaching for the same thing at the same time. All within a week.

That’s more than a coincidence, it’s the direction everything is headed, and matches what I have been saying all along… We need to be the humans in the equation. We cannot take our hands off the wheel and let AI take over.

So, no. This doesn’t change my plans

I’m continuing to build on Substack. Nothing I’m creating, teaching, or offering shifts because of this.

I do think creators should understand the mechanics of what changed, so here’s the practical part:

On Substack. On July 20, Substack updated its Publisher Agreement to make explicit that creator content can be run through automated analysis tools for safety and integrity purposes. On July 21, they launched the detection feature itself, built with a tool called Pangram. It scans articles, Notes, replies, and comments of at least 100 words. Video, audio, and custom-domain sites cannot currently be scanned. You also cannot scan content published prior to July 21st. You can scan your own work, disable scanning on a post, or add a “How I make this” statement describing your process. If a scan comes back wrong, you can report it. You own your content, AI isn’t banned, and nothing is publicly labeled unless a creator or reader chooses to scan it.

Something to consider: because the scanner can’t read custom-domain sites, domain choice is no longer purely an SEO decision. It now also determines whether your work can be scanned at all. I’m staying on my Substack domain for now. I’ll wait-and-see if that changes.

Pangram itself claims 99.98% accuracy. My own experiments say otherwise, and plenty of other creators are reporting the same thing. I’d treat it as one signal among many, not a verdict. Again, discernment and a gut-check are the tools I will focus on.

On LinkedIn. The mechanism is a “Seems like AI slop” flag in the three-dot menu on any post or comment, plus a private dashboard warning if enough people flag your content. There’s no “How I make this” option and no way to contest a flag — LinkedIn isn’t asking you to explain your process, it’s just telling you the room doesn’t buy it.

Under EU law. As of yesterday, AI systems must disclose when you’re interacting with AI, deepfakes have to be labeled, and AI-generated content needs a machine-readable mark. This one isn’t a platform choice you can opt out of — it’s regulation, and it applies regardless of where the content was created.

I’ll keep testing, keep reporting the misfires, and keep telling you what I find.

There are all kinds of conversations happening on Substack, LinkedIn, and now under EU law. Do a quick search to read beyond my take. This is all new territory and is by no means final. Rules and processes will change on various platforms and in various countries based on feedback and tech changes. It’s all part of this new world we are living in together. Stay calm and stay tuned.

- Donna