Detects personal data before it leaves the machine, with Indian identifiers built in: Aadhaar (with Verhoeff checksum), Aadhaar Virtual ID, PAN, GSTIN, voter ID, passport, driving licence, CKYC, ration card and NREGA job card numbers, UPI IDs, IFSC, bank accounts, payment cards with expiry dates and verification codes, +91 mobile numbers, ABHA, and names and addresses in Latin and Indic scripts.
The detection rules live in a shared, language-neutral spec, so this package
and the Python package credgate find exactly the same matches. Part of the
CredGate project; early development.
Redact before the prompt reaches the model; the model only ever sees
placeholders such as [PAN] and [PERSON_NAME]:
note <- "Customer name: Priya Nair. PAN ABCPN1234K, mobile +91 98765 43210, wants the EMI date moved."
preview(note, min_confidence = 0.5) # what will be masked, and why
library(credgate); library(ellmer)
safe <- redact(note, min_confidence = 0.5)
chat_ollama(model = "gemma3:1b")$chat(paste("Summarise:", safe))
# vLLM: chat_vllm(base_url = "http://localhost:8000/v1", model = "<served model>")
The vignette vignette("local-llms", package = "credgate") walks through
Ollama, vLLM, data frames and evaluation; runnable scripts are in the
examples folder.
Check the preview. Names are found next to an honorific, relation or field label (
Shri,S/o,Name:,नाम:); a name in running text without such a cue ("Customer Priya Nair called") is not detected yet. Redaction is irreversible; to get the real values back in the model's answer, tokenise instead (see below). An automatic proxy is on the roadmap.
| On 700 held-out synthetic documents | Precision | Recall | F1 | Time |
|---|---|---|---|---|
| credgate | 1.000 | 0.961 | 0.980 | 0.3 s |
| Presidio, India-configured | 0.609 | 0.754 | 0.674 | 11.6 s |
| Presidio, default | 0.565 | 0.498 | 0.529 | 12.1 s |
About the data: the 700 documents are synthetic - KYC forms, bank
messages, call-centre notes, Hinglish chats, Hindi text and hospital
summaries generated by
benchmark/generate_corpus.py
and included in the repository under
benchmark/corpus/.
No real personal data was used.
Types both tools cover, exact span matching, thresholds tuned on a
separate dev split for every system; time is for 1,400 documents. Synthetic
data generated by us, so read with that in mind.
Where Presidio does better. credgate is built for Indian identifiers and Indian-language context, and that is what this benchmark measures. Presidio is broader and more mature:
Full method, per-type results and how to reproduce: benchmark/README.md. The comparison is independent and not affiliated with Microsoft.
CRAN submission of 0.1.0 is under review. Until it is accepted, install the development version from R-universe or GitHub:
# Development version, prebuilt for Windows, macOS and Linux (R-universe):
install.packages("credgate",
repos = c("https://dwarpal.r-universe.dev", "https://cloud.r-project.org"))
# or from GitHub (needs build tools on Windows and macOS):
# install.packages("remotes")
remotes::install_github("dwarpal/credgate", subdir = "pkg/credgate")
With Nix: nix build github:dwarpal/credgate#credgate-r.
library(credgate)
x <- detect_pii("Name: Priya Nair; PAN: ABCPN1234K; UPI: priya.n@oksbi")
x
#> 3 PII matches (values masked; use `x$value` to see them):
#> doc start end type value confidence
#> 1 7 16 person_name XXXXX XXXX 0.80
#> 1 24 33 pan XXXXXX234K 0.97
#> 1 41 53 upi_id XXXXX.X@Xksbi 0.99
x$reasons[2]
#> [1] "matched PAN pattern; context 'PAN'; holder type: individual"
detect_pii() takes a character vector and returns one row per match with
the document index, character positions, type, data class, value, canonical
form (for consistent tokenisation), confidence and reasons. Printing masks
the values.
Context makes generic patterns trustworthy. Pass extra context, such as a
column name, with context:
detect_pii("ABC1234567") # nothing: could be anything
detect_pii("ABC1234567", context = "voter_id") # Voter ID, confidence 0.97
Indic digits, fullwidth characters and zero-width characters are normalised before matching, and positions refer to your original text:
detect_pii("आधार: ५४२२ २९३७ ११८२")$type
#> [1] "aadhaar"
redact() replaces each identifier with a placeholder naming its type;
preview() shows what would be redacted and why, before anything is sent:
redact("Name: Priya Nair; PAN: ABCPN1234K")
#> [1] "Name: [PERSON_NAME]; PAN: [PAN]"
preview("Name: Priya Nair; PAN: ABCPN1234K; order no 878245671885")
#> 3 items would be redacted:
#> Name: [PERSON_NAME]; PAN: [PAN]; order no [AADHAAR]
followed by a table of each placeholder, the masked value, its confidence
and the reasons (here the order number scores only 0.42 because of the
word "order"; use min_confidence to leave such matches alone). Both are
new after 0.1.0: install the GitHub version to use them.
Redaction is one-way. To get the real values back in a model's answer,
tokenise() replaces identifiers with numbered tokens and keeps the
mapping in a vault; restore() puts the values back:
vault <- cg_vault()
tokenise(c(
"Shri Ramesh Kumar, PAN ABCPE1234F, mobile +91 98765 43210",
"pan abcpe1234f called again from 9876543210"
), vault)
#> [1] "Shri [PERSON_NAME_1], PAN [PAN_1], mobile [MOBILE_IN_1]"
#> [2] "pan [PAN_1] called again from [MOBILE_IN_1]"
restore("[PERSON_NAME_1] (PAN [pan_1]) will be called on MOBILE_IN_1.", vault)
#> [1] "Ramesh Kumar (PAN ABCPE1234F) will be called on +91 98765 43210."
The same identifier gets the same token however it is written, so the model
can still tell customers apart. restore() accepts tokens as models tend to
copy them (other case, spaces, missing brackets); strict = TRUE stops on a
token the vault does not know. The vault prints with values masked.
To restore tokens in a later session - or in Python - save the vault to a
file encrypted with a passphrase (at least 12 characters; from the argument,
the CREDGATE_VAULT_PASSPHRASE environment variable, or a prompt):
vault_save(vault, "loans.cgvault") # needs the sodium package
vault <- vault_open("loans.cgvault") # later: same tokens, numbering continues
The passphrase is never stored: without it the file cannot be opened. Keep
*.cgvault files out of version control.
Keyed tokens make the same identifier get the same token everywhere - across sessions, datasets, machines and R or Python - so tokenised tables can still be joined and counted, and parallel workers agree (needs the openssl package):
key <- cg_new_token_key() # store it like a password, e.g. in CREDGATE_TOKEN_KEY
a <- cg_vault(key)
b <- cg_vault(key)
tokenise("PAN ABCPE1234F", a) # "PAN [PAN_bpg46o6x465d4]" (depends on the key)
tokenise("pan abcpe1234f", b) # the same token
vault_merge(a, b) # combine vaults from parallel workers
Whoever holds the key can confirm a guessed value by computing its token,
so keep it out of code and project files. The key is never saved in a vault
file; vault_open(path, key = ...) needs it again.
scan_pii() reports which columns hold PII and how much, without sending
anything anywhere. (New after 0.1.0: until the next CRAN release, install
the GitHub version to use it.) Column names count as context, so a column called
voter_id or cust_pan supports those types:
customers <- data.frame(
cust_pan = c("ABCPE1234F", "AAACR5055K", "XYZPQ9876R"),
notes = c("Called, Aadhaar 542229371182", "no issues", "mobile 9876543210")
)
scan_pii(customers)
#> Scanned 2 columns; PII found in 2: cust_pan and notes
#> column type rate matched whole confidence example
#> cust_pan pan 100% 3 of 3 3 0.97 XXXXXX234F
#> notes aadhaar 33% 1 of 3 0 0.97 XXXXXXXX1182
#> notes mobile_in 33% 1 of 3 0 0.93 XXXXXX3210
whole counts values that are entirely the identifier (an ID column)
rather than containing it (free text). Large columns are sampled
(sample = 10000 values by default).
verhoeff_valid("542229371182") # Aadhaar / VID checksum
luhn_valid("4111 1111 1111 1111") # payment cards
gstin_valid("27AAPFU0939F1ZV") # GSTIN check character
synth_indian_pii(5, seed = 1) # synthetic records for tests and demos
Synthetic values pass the same validation as real ones but are random; use them for tests, never as anonymised data.
register_detector(new_detector(
id = "employee_id", class = "internal", label = "Employee ID",
pattern = bounded("EMP[0-9]{6}")
))
cg_detectors() # all registered detectors
Tables too big for memory are read in chunks. Install the packages you use
(arrow, duckdb, DBI, dbplyr):
scan_pii(arrow::open_dataset("calls.parquet")) # one column at a time
redact_parquet("calls.parquet", "calls_safe.parquet", columns = "notes")
con <- DBI::dbConnect(duckdb::duckdb())
scan_pii(dplyr::tbl(con, "calls")) # any dbplyr table
redact_query(con, "SELECT call_id, notes FROM calls", "calls_safe")
Any DBI database works the same way, for example Postgres:
con <- DBI::dbConnect(RPostgres::Postgres(), dbname = "lending", host = "db.internal")
scan_pii(dplyr::tbl(con, "loan_notes")) # needs dbplyr
redact_query(con, "SELECT id, notes FROM loan_notes", "loan_notes_safe")
redact(), tokenise() and detect_pii() also accept Arrow arrays. See
vignette("large-data") and the
large-data guide.
A policy decides, per destination, what happens to each identifier: allow, redact, tokenise, or block the whole request. The same YAML file works in Python:
policy <- cg_policy("bank-default.yaml")
d <- policy_apply(policy, c("Card 4111 1111 1111 1111 exp 12/27", "CVV 123"), "claude")
d$allowed # FALSE
d$blocked_by # "card-with-cvv-and-expiry"
policy_decide() reports the decision without rewriting; policy_apply()
also rewrites allowed requests (pass vault = when the policy tokenises).
Loading checks everything: an unknown field, tier, action or type is an
error. An example policy is in the repository's
examples/policies/bank-default.yaml.
Policy decisions can be recorded in an append-only, hash-chained log: who sent which kinds of data to which endpoint, under which policy revision and rule - never the text or the values. Editing, deleting or reordering records is detected; with an audit key, identifiers are kept as keyed hashes; card verification codes are never recorded. R and Python share the format.
log <- cg_audit_log("audit.jsonl") # key from CREDGATE_AUDIT_KEY, if set
policy_apply(policy, messages, "claude", audit = log) # recorded, allowed or blocked
audit_verify("audit.jsonl") # intact? and the head hash
audit_read("audit.jsonl") # the records, as a data frame
Wherever credgate shows a detected value (printed matches, preview(),
scan_pii() examples), it masks it:
| Value | Shown as |
|---|---|
| Identifiers (Aadhaar, PAN, mobile, account, card, ...) | last 4 characters kept: XXXX XXXX 1182, XXXXXX234F
|
| Names and addresses | masked in full: XXXXX XXX
|
Keeping the last 4 characters lets you check which value matched, and stays within the display limits of UIDAI's masked Aadhaar and PCI DSS (cards). If your policy allows no part of a value to be shown, mask everything:
options(credgate.mask = "full") # or CREDGATE_MASK=full in ~/.Renviron
Masking changes only what is displayed; detection and redact()
([AADHAAR], [PERSON_NAME]) are unaffected.
Names and addresses are found only next to a cue: an honorific (Shri, Smt,
श्रीमती, or "ji" after the name), a relation (S/o, D/o), a field label
(Name:, पता:) or a house/village keyword followed by a PIN code. A deity
addressed with "ji" ("Ram ji") is taken as a name. A bare 12-digit number may pass
the Aadhaar checksum by chance and is reported with lower confidence.
If you use credgate in your work, please cite it. In R:
citation("credgate")
Apache License 2.0.