blog header image keyboard with button labeled scam

Can AI Tell You Whether an Email Is a Scam?

By[email protected]

An invoice arrives from a vendor you’ve used for years, and something about it just sounds wrong. Not wrong enough to call anyone. Just wrong enough that you don’t want to be the person who paid it.

So you copy the email, paste it into ChatGPT, and ask: is this a scam?

Four seconds later you get a thoughtful-sounding answer. And you feel better. However, feeling better and being safer aren’t the same thing.

Here’s the short version. AI is genuinely good at spotting the language of a scam, and genuinely blind to the evidence that indicates one. When you paste the text of an email, you leave behind the technical bits that prove where it came from. So use AI’s answer only to review the text itself – if it says scam, it might be a scam. If it doesn’t say scam, that doesn’t mean it’s still not a scam, just that the message itself looks ok. Then, turn on your real protection in your email system, which probably already includes tools you’re paying for and may not have turned on.

Three things to note:

  • Never trust an AI’s “this looks fine.” In one peer-reviewed test, a leading model cleared 98% of the scam emails it was shown as legitimate.
  • Every email has technical bits, the header, that don’t come through when you copy the text — a technical record of where it really came from. These email headers require dedicated email security systems or trained IT professionals to inspect properly.
  • Pasting client or patient information into a free chatbot is its own problem, separate from whether the answer is right.

Use AI for a Warning, Never for an All-Clear

If you remember one rule from this post, make it this one.

Researchers ran 381 emails past the leading AI models.One caught 97% of the scams. Another caught 2% (Heiding et al., Expert Systems With Applications). Same emails, same week, and one model was nearly perfect while the other waved through 98 out of 100. The model behind a chat window gets swapped out or updated regularly, so review results may not be consistent.

Treat the two results completely differently:

  • “This looks suspicious” — useful. Go verify.
  • “This looks legitimate” — ignore it. It tells you nothing about whether the email is real.

Every Email Has a Hidden Half, and You Can’t Just Copy It

Every email that arrives has two parts. There’s the part you see: the subject, the sender’s name, the message, the attachment. And there’s a second part you don’t see, attached to every single message, recording how it actually got to you — which computer really sent it, whether the sender’s domain passed its identity checks on the way, every server it traveled through, and where its links truly lead. It’s called the header, and it’s usually a page or two of dense technical text.

You’ve almost certainly never looked at one. Your email program deliberately hides it, because it’s unreadable and because you don’t need it 99% of the time. In Gmail you’d have to open the three-dot menu on a message and choose Show Original. In Outlook it’s buried under message properties. 

So when you copy the text of a suspicious email, you get the visible part only. You cannot accidentally include the header. There’s nothing to drag, nothing highlighted, no sign anything was missed. You paste what looks to you like the entire email, and you’ve handed over the part that proves nothing while missing the part that can prove everything.

Think of a paper letter. The message is the letter. The header is the envelope — the postmark showing where it was really mailed from, the return address, the routing stamps. An AI reading your pasted text is studying the handwriting on the letter while the envelope with key info sits in the trash. 

The research shows how important those headers are. With headers, AI models scored perfectly. Your email platform reads the header on every message automatically, which is a point I’ll come back to. 

Fakes can pass every authenticity check. A scammer registers a domain one character off from your vendor’s — swapping “rn” for “m” so michelle@ becomes rnichelle@, which your eye reads straight past. Because they actually own that domain, their mail is properly signed and fully authenticated. It is not technically forged. The only tells are that the domain was registered nine days ago and that your company has never once corresponded with it. Neither of those facts is in the email at all — one lives in public domain registration records, the other in your own mail history. No amount of careful reading of the message surfaces either.

The worst case involves no fake domain at all. If the scammer got in your email, the most expensive email of your career will come from your vendor’s real address, at their real company, properly authenticated, written by someone who has read your entire thread and knows exactly what you’re expecting. Everything about it will seem genuine.

Which brings up the advice everyone was given for twenty years: watch for typos and awkward grammar. That advice is finished. Bad writing used to be a clue. Now that scammers use AI to write and check emails, it’s not as easy to identify that way. (The exception would be if someone you’ve known for a while emails and it just doesn’t sound like them – trust your gut and give them a call.)

Don’t Paste Client or Patient Information Into a Free Chatbot

Separate issue, and it applies even when the AI’s answer happens to be right.

On free and personal plans, what you paste may be kept and used to improve the model. Google’s own consumer Gemini notice puts it about as plainly as anyone could: “Please don’t enter confidential information that you wouldn’t want a reviewer to see.” Chats selected for human review “are retained for up to three years” and are not deleted when you delete your activity (Gemini Apps Privacy Hub). Anthropic keeps consumer chats for five years if you’ve allowed them to be used for improving Claude (Anthropic).

So the employee pasting a client invoice into a free chatbot to check whether it’s a scam has just handed a client’s payment details to an outside company, in the name of security. Employee AI use jumped from 15% to 45% in a single year, and this kind of unmanaged AI use is now the third most common way company data accidentally leaks out (Verizon 2026 DBIR).

Business and enterprise plans are meaningfully different. OpenAI, Anthropic, Microsoft, and Google all state they don’t train on business-tier content by default (OpenAI; Anthropic; Microsoft; Google). If your team uses AI at work, that’s the tier to be on.

If you handle patient information, this becomes a compliance risk rather than a preference. A quick disclaimer: I’m a cybersecurity practitioner, not an attorney. What follows relies on published HHS guidance and vendor terms, and it isn’t legal advice — check your own obligations with your legal or compliance counsel. HHS specifically lists an AI chatbot handling patient information as a business associate, meaning you’d need a signed agreement with that vendor (HHS). ChatGPT Free, Plus, and Team are not on OpenAI’s list of HIPAA-eligible products (OpenAI). Pasting a patient-identifying email into those plans is a disclosure to a company you have no agreement with.

The same logic applies if you’re an attorney, accountant, or insurance agency working under a client NDA. The question isn’t whether the AI is trustworthy. It’s whether your engagement letter lets you hand client material to an outside company that may keep it for years.

Pro Tip: if you’re going to paste work email into AI at all, use a business plan, turn training off, and strip out names and account numbers first. 

The Real Protection

Remember that hidden header in the email? Your email platform reads it on every message, automatically, before the message ever reaches you. That’s the whole difference between a chatbot’s opinion and an actual security check. And you may already be paying for the tools that act on it.

Some Microsoft Defender licenses/plans check the sending server and signatures, follow links to see where they really go, open attachments safely in a sandbox to watch what they do, compare senders against who you actually correspond with, and can reach back into everyone’s inbox to pull a message out after delivery once it turns out to be malicious (Microsoft).

Here’s the catch, and it’s the reason I keep writing about this. Impersonation protection — the exact feature that catches those one-character lookalike domains — is off by default (Microsoft). On Google Workspace, enhanced pre-delivery scanning and Security Sandbox are also off by default (Google).

If you pay for a business plan and nobody has been into that admin console, those are almost certainly off right now. 

Six Things to Do This Week, and Two That Matter Most

In order of how much protection you get per minute spent.

  1. Call the number you looked up yourself. For any payment request or bank-detail change, phone the vendor at a number from your own records or their website — never the one in the email. Every time, no exceptions when it’s urgent. This single habit defeats nearly all of this, and it’s free.
  2. Turn on impersonation protection in Microsoft Defender, or enhanced pre-delivery scanning and Security Sandbox in Google Workspace. You’re likely already paying for these.
  3. Turn on external sender warning banners. One setting, and the visual cue holds up on a busy Tuesday better than any training session.
  4. Write down a payment-change procedure and require two people to approve wires above a threshold you pick. Then nobody has to make that judgment call alone at 4:50 on a Friday.
  5. Set up SPF, DKIM, and DMARC — the digital identity checks that stop scammers from sending mail as your domain. They’re free, and they’re settings your IT person or web host can add (Google; Microsoft).
  6. Use app-based or hardware MFA on finance and admin accounts, not text messages (aka SMS).

Notice that not one of these asks anyone to be a good judge of writing.

Items 1 and 4 are not technology, they’re process. A phone call to a number you looked up yourself, plus a second person’s sign-off on the money, cannot be beaten by a better-written email. It doesn’t matter whether the message was written by a teenager or by a model that hasn’t shipped yet, whether it comes from a lookalike domain or your vendor’s real compromised mailbox, or whether it passes every authentication check perfectly. A fraudulent email only works if the decision to pay gets made inside the email thread. Move that decision out of the thread and onto a phone call and a second signature, and in the vast majority of cases, the scam will have been avoided.

Every other item on this list reduces how many of these reach your team. Those two decide whether the ones that do get through can actually cost you money.

Where AI Genuinely Helps

I’m not telling you to stop using it. I’m telling you to give it the jobs it’s good at (provided you’ve made sure you’re not sharing anything you shouldn’t).

  • Explaining something unfamiliar. “What does a DMARC record do, and do I need one?” AI is excellent at questions like that.
  • Naming the pressure. Ask “what manipulation techniques is this message using?” instead of “is this a scam?” You’ll get something more useful.
  • Drafting the verification email. “Write a short, friendly note asking our vendor to confirm this bank change by phone.” Genuinely handy, zero risk.
  • Summarizing a long forwarded thread so you can see who introduced the payment change, and when.

An AI can tell you what an email is trying to do. Only your systems and your phone can tell you whether it’s true.

Questions People Ask About AI and Scam Emails

What is an email header, and why does it matter? The header is a hidden technical record attached to every email, showing which computer actually sent it, which servers it passed through, whether the sender’s domain passed its identity checks, and where its links really point. Your email program hides it by default because it’s dense and unreadable. It matters because that’s where evidence of forgery lives, and it’s the part that doesn’t come along when you copy and paste an email.

Can ChatGPT tell if an email is a scam? Not reliably. Pasted text also leaves out the delivery details, link destinations, and sender history that actually determine whether an email is fraudulent. Treat a warning as worth checking and an all-clear as meaningless.

Is it safe to paste work emails into ChatGPT? Not on free or personal plans, if the email contains client, patient, or financial information. Those plans may use your content to improve the model and keep it for years — Anthropic retains consumer chats for five years when training is allowed, and Google’s consumer Gemini keeps human-reviewed chats up to three years even after you delete your activity. Business and enterprise plans don’t train on your content by default.

Can I paste an email with patient information into ChatGPT? No. ChatGPT Free, Plus, and Team are not on OpenAI’s list of HIPAA-eligible products, so no business associate agreement covers them, and HHS treats an AI chatbot handling patient information as a business associate. Remove all patient identifiers first or use a HIPAA-eligible plan. This reflects standard HHS guidance rather than legal advice — have your legal or compliance counsel review your software agreements.

How can I tell if an invoice email is fake? Don’t try to tell just by reading it. Call the vendor at a number you look up independently, from your own records or their website rather than the email, and confirm the payment details out loud. Treat that call as mandatory for any bank-detail change, however routine the request looks. 

Is “watch for typos and bad grammar” still good advice? No, and it can now work against you. Poor writing was only ever a clue that an unskilled attacker wrote the message, and AI removed that clue. Polished, well-targeted, error-free email is now more dangerous than clumsy email. What still works is verifying who actually sent it.

What should a small business use instead of a chatbot to catch phishing? Use business email systems. They don’t have to be expensive. They examine the delivery details, link destinations, and your real sender history, none of which a chatbot can see. Check that impersonation protection is switched on, because it is off by default.

Not sure what’s switched on in your email system?

Most of what I’ve described here is either free or already included in what you’re paying for. The catch is that several of the highest-value settings ship off by default, so it’s worth having someone actually look. If you’d like a second pair of eyes, book a free consult and we’ll go through your configuration together.

[Book a free consult]

In the next blog post, I’m staying on the AI theme with one that’s caught a few people off guard: what happens to the recording when an AI notetaker joins a client call, and what are your confidentiality obligations, from a security point of view.

If you missed it: who eats the loss when a wire goes to a scammer.

Sources

Similar Posts