Everyone worries the model will memorize their users' data. The leak that actually gets most apps is one they built themselves, and it has nothing to do with training.
PII (personally identifiable information: anything that points at one specific human, like a name, email, phone number or social security number) gets out of an AI app two very different ways.
The first is the one people picture. Training-time memorization: the model swallowed text while it was being built and can repeat pieces of it back word for word. Researchers have pulled real names, phone numbers and email addresses out of models this way. It is real, and mostly not yours to fix: you cannot reach into a trained model and delete one person's details.
The second is runtime leakage, and that one is entirely yours. Someone types their phone number into your chat box to ask where their order is. That number is now in your database, in whatever tool you use to inspect requests, in the crash report from the call that failed, and in the file someone exported to check answer quality. The model memorized nothing, and the number is sitting in four places that can each hand it to the wrong person.
The line worth remembering: if it's in the prompt, it's in the logs.
So the scrubbing goes on the way in, before anything is stored. Open-source detectors like Presidio find names, emails and card numbers in text and mask them first. They miss things, so treat one as a filter, not a guarantee.
Quick check before you scroll: Your RAG agent stores and embeds every user's chat for long-term memory. Months later, a different user's query surfaces someone else's email address in a retrieved snippet. Is this training-time memorization or runtime/context leakage - and why does that distinction matter?
Full breakdown + the answer: frankduah.me/learnings/2026-09-08-pii-and-privacy-in-llm-applications
New here? I post a bite-size AI / ML concept like this every day - follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#PII #AI #LLM #AIAgents #MachineLearning
The answer
Runtime/context leakage - the email was never in the model's training data, it entered through your own memory pipeline (logged, embedded, then retrieved for the wrong user). It matters because this leak path is entirely within your control to fix: redact or scope data before it's stored, not after.