3 ways attackers use AI to recover protected data & how to stop it

AI has changed a lot of things. One of those is how we think about protecting sensitive data. Now that models sit on both sides of protected data, as consumers that grade it and attackers that probe it, it only makes sense to reassess how we protect sensitive data. 

In this post, we review the three potential ways AI can be used to recover sensitive data and how tokenizing your data helps significantly reduce that risk. If all the model reads is tokens, it only learns tokens and maybe policy. 

Why the decision is being re-graded

Today, the systems reading protected data include LLMs and model grade protection on properties with a unique checklist:

• Is the protected value shaped like plausible data? 

• Does the structure of a record survive?

• Are the same entities recognizable across sources?

For the choice between masking vs encryption vs tokenization, that has a specific consequence because each option behaves differently on that criteria:

• A protection that preserves the structure and distribution of the data tends to preserve its usefulness to a model. 

• A protection that flattens the structure, whether by destroying values outright or collapsing them into something the model has never seen the likes of, tends to take the model's performance down with it.

The gap between the best and worst choices here is larger than intuition suggests. It also doesn't correlate neatly with security strength. 

3 ways an attacker can try to use AI to recover sensitive data

Let’s look at an example of what an attacker can do with the information they can steal, based on the way the data was protected. Consider a scenario where credit card information has leaked: what’s the hacker’s next move?

• Redaction: The attacker has the last 4 digits, so there is nothing to pursue. 

• Probabilistic encryption: The attacker holds unreadable blobs and is racing to get key custody to make sense of them.

• Format-preserving encryption: The blobs are now well-formed values, which reveals structure and a very small field can be searched offline. 

• Deterministic encryption: Even without the key, frequency work on a low-cardinality field starts immediately because equal values sit in the stolen table as equal ciphertexts.

• Tokenization platform: Stolen vaulted tokens are useless without the vault. Stolen vaultless tokens require both the algorithm and a policy grant to detokenize. In both cases, the theft needs to be followed by a second compromise before it produces anything and the complexity of that second theft is significantly higher with vaultless tokenization systems than vaulted systems. 

Give the attacker a model and two new moves get proposed, one aimed at models trained on protected data, one aimed at the protection itself. Both need sober answers because both tend to be invoked with more fear than precision.

Attack #1: The model as a copy

Reconstruction attacks are real for plaintext. Models memorize rare, high-entropy strings, and the right prompting can bring them back verbatim. A card number sitting in a training corpus is exactly the kind of string that gets memorized. But, no extraction attack can surface a value that never entered the corpus. Train on tokenized fields and the model's entire knowledge of the card number is the token. The most successful reconstruction would return the stand-in token and turning that into the real number would still require the algorithm and a policy grant. The model became a copy of the data without becoming a path back to the raw values.

How much a model memorizes is not uniform. Three conditions raise the likelihood of memorization:

1. Repetition: A value appearing across thousands of records is more likely to come back than one seen once. 

2. Small corpora: Fine-tuning on a few thousand internal documents memorizes far more per document than pre-training on the open web ever did.

3. Uniqueness: A string unlike anything else in the corpus gives the model no way to generalize, only to store. 

An account number repeated through a fine-tuning set sits at the intersection of all three.

Tokenizing sensitive data at the source, before it reaches the model, helps you reduce the risk that the model will learn the raw value. But, like any protection technique, it has edges that are worth noting:

• Free text is the largest culprit: Field-level protection does nothing for the names and numbers inside a support transcript or other forms of unstructured data. 

• Determinism has a price: The same plaintext produces the same token every time. This is what maintains data usability across systems and should be implied in any mention of tokenization. A model trained on consistent tokens learns true facts about token identities, which accounts co-occur and which entity did what. This surrounding context could be sensitive information in its own right, that needs to be protected in an appropriate manner. 

Tokenization removes the raw value from the corpus, but it does not make the model forget patterns. Bounding memorization itself is a different control with its own utility bill.

Bear in mind that training isn’t the main way enterprise data reaches a model. Retrieval pipelines embed documents into vectors and hand the model relevant chunks at question time. Both halves of that pipeline are recovery surfaces of their own. 

Embeddings are not one-way. Inversion techniques recover close approximations of the text that produced them, so a vector database of embedded plaintext is best treated as a copy of the plaintext. The retrieved chunks themselves pass through prompts, logs and cache.

Tokenize before embedding and the discipline follows the data into the pipeline. Inverting a vector yields token-bearing text and the retrieval store stops being the softest copy of the most sensitive data.

Attack #2: The model as a cryptanalyst

Here we look at a situation where a threat actor would point a model at the tokens themselves and let it deduce how they were made. The fear comes in two strengths, which need separate answers:

1. What happens if a table is stolen? In this case, the attacker has a sample of tokens, that’s it.

2. What happens if an attacker gets access to plaintext and token pairs?

Let us first evaluate analyzing tokens in isolation. At the highest level, a sample of tokens says nothing about the mechanism because the outputs are built to be indistinguishable from values drawn at random over the field's domain. Studying more of them adds volume, not signal. 

What a sample reveals is policy, not cryptography. It exposes the template: the alphabet, the length and the format. In deployments that preserve fragments of the original data (such as the last four digits of a credit card), those fragments are simply plaintext showing through. A model can extract these fragments faster than any human analyst, but the tokenization system acts as a hard barrier. Without explicit authorization and access to the algorithm, the model cannot read past the plaintext. 

Pairs of plaintext + tokens are the stronger position and the answer is sharper. A mapping where a model could learn from examples, well enough to predict the token for a value it has not seen, would be, by that fact alone, a broken cipher. That is not a bet that models lack cleverness. It says that if data is tokenized before it reaches the model there is no pattern in the data for any learner to find. Analysis behind standards like FF1 and FF3-1 exist to keep it that way.

The picture degrades when determinism is used on a very small domain with low-cardinality fields. In this niche case, the model needs no learning at all. It just needs to remember patterns and models are very good at that. In this scenario, given enough pairs of plaintext and tokens, the permutation can be written down. This is true for deterministic encryption too.

Attack #3: The model as a credentialed caller

This path is the one nobody frames as an attack on tokenization and it is the only one that works. An enterprise AI agent doesn't need to perform complex cryptanalysis or extract secrets from its training weights. If it has authorized access to tokenization grants, it retrieves plaintext through standard system requests. So, any attacker who manipulates the agent's prompts inherits its access privileges.

Prompt injection is the steering: an instruction hidden in a document, a ticket or a webpage the agent reads, telling it to fetch and reveal what its permissions allow. The model became a caller the policy layer authorizes, which is exactly the position the two-lock design assumes is trustworthy.

The response is not new machinery, but the existing machinery applied to a new kind of caller. Agent identities need the same role scoping as human ones, per field and per purpose, with detokenization granted to the narrow workflows that require it rather than to the assistant as a whole. Zero trust applied to AI. 

The logging that makes detokenization visible only works if agent calls are attributed to the agent and to the request that drove them. An over-permissioned agent quietly collapses two locks into one because the policy grant, the lock that was supposed to fail independently, now sits inside the same blast radius as a prompt.

None of the three paths (copy, cryptanalyst, credentialed caller) move a technique to a new spot on the recovery axis. Models stress-test the spot already chosen. Deterministic protection over a low-cardinality field was leaking before AI arrived. The new attention belongs to the operational surface: the free text, the learned associations and the over-permissioned agent.

Conclusion

There are three ways in which AI can be used to attempt to recover protected values. Extraction returns what entered the corpus. If what was entered was tokens, the model reads tokens. A learner finds no mechanism in tokens, only policy. 

The one path that works, the over-permissioned agent, is not an attack on the data protection at all. It’s the age-old story of over-permissioning. Just an ordinary caller with an extraordinary grant, governed the way callers always were, by scoping and a log. 

Models did not change the question this series keeps asking. They changed who asks it, how often and at what speed. A protection technique chosen per field, on purpose, answers them the same way it answered everyone else. To see how these decisions land on your own fields, book time with the Databolt team.