Privacy, Data Rights, and Model Security

Definition A dataset license is the set of terms under which a party holding rights in a dataset allows others to use it. It states which operations are permitted, such as copying, modifying, redistributing, commercial use or training models, and on which conditions, such as attribution, sharing derived data under the same terms or deleting copies on request. A license can grant only rights that its issuer holds, so material the source had no authority over is not covered, and the provenance of the data has to be checked. A license is one part of data permission, the question of whether a given party may carry out a given operation on given data. When the data concern people, permission also depends on the basis for using them, which in some settings is consent with a stated scope that can be withdrawn, and on purpose limitation, the principle that data collected for one purpose are not reused for an incompatible purpose without a new basis. Being able to download a file from a public page gives access to it, not any of these permissions.

Definition Personal data is information that relates to an identified or identifiable person. A person is identifiable when they can be singled out or recognized, either directly through an identifier such as a name, an email address or an account number, or indirectly through a combination of other facts together with information an observer could reasonably obtain elsewhere. Fields that identify someone only in combination, such as a region, an age band and a job, are called quasi-identifiers. Personal data can sit in structured fields and inside free text, for example an address or a distinctive description written into a message. Pseudonymous data, in which names are replaced by codes, can still be linked back to people by whoever holds the mapping or by matching other fields. Whether data identify someone depends on what is released, to whom and alongside what else. Legal definitions differ between jurisdictions, and this page describes the idea rather than any particular law.

Definition Memorization is the retention, in a fitted model, of information specific to individual training examples rather than to patterns shared by many of them. In its strongest form the model, or its outputs, reproduce distinctive content of a training example word for word, such as a rare passage of text. Some models memorize by construction, since a fitted k-nearest neighbors model stores its training rows and sharing the model shares the data. Membership inference is the task of deciding, from access to a model or its outputs, whether a given record was in the training data. The observer already holds the record and asks only about its inclusion, which can be sensitive in itself, for example when the training set consists of the patients of one clinic. Repeated records, high model capacity and detailed output access are reasons to test for these risks, but none of them proves that a particular model will disclose a particular record, and a small gap between training and test error does not show that membership inference fails.

Definition Let $M$ be a randomized mechanism that takes a dataset and releases a random output, and fix a neighboring relation between datasets. On this page two datasets are neighbors when one is obtained from the other by adding or removing everything one person contributes, so the relation is symmetric. For $\varepsilon \ge 0$ and $0 \le \delta < 1$, the mechanism $M$ is $(\varepsilon, \delta)$-differentially private if for every pair of neighboring datasets $D$ and $D'$ and every measurable set $S$ of outputs

The probabilities come only from the randomness inside $M$, with both datasets held fixed. The case $\delta = 0$ is called pure differential privacy, and a smaller $\varepsilon$ gives a tighter bound. The additive term $\delta$ is usually chosen far below $1/n$ for a dataset of $n$ people, because a mechanism that publishes each person’s full record independently with probability $\delta$ satisfies the definition with $\varepsilon = 0$. For a real-valued query $f$, the sensitivity

taken over neighboring pairs and assumed finite, is the most one person can change $f$. For $\varepsilon > 0$ and $\Delta f > 0$ the Laplace mechanism releases $f(D) + Z$, where $Z$ has the Laplace density with scale $b = \Delta f / \varepsilon$,

It is $(\varepsilon, 0)$-differentially private. With $a = f(D)$ and $a' = f(D')$, the triangle inequality gives $\lvert z - a' \rvert - \lvert z - a \rvert \le \lvert a - a' \rvert \le \Delta f$ at every output $z$, so the two output densities differ by at most a factor $e^{\Delta f / b} = e^{\varepsilon}$ at every point, and integrating over $S$ gives the inequality. Noise added without this calibration to the sensitivity carries no guarantee. Differential privacy limits what a release reveals about one person’s participation. It does not stop inferences about a person drawn from patterns in other people’s data, and it does not make an unauthorized use permitted.

Definition A privacy budget is a declared total $(\varepsilon, \delta)$ that bounds the privacy loss of everything released from a protected dataset, under one stated neighboring relation and one stated accounting method. Two results do the accounting at the basic level. Basic sequential composition says the following. Suppose mechanisms $M_1, \dots, M_k$ are applied to the same dataset, each $M_i$ is $(\varepsilon_i, \delta_i)$-differentially private under that relation, and each uses fresh randomness independent of the others, where $M_i$ may be chosen after seeing earlier outputs as long as it is $(\varepsilon_i, \delta_i)$-differentially private for every possible value of them. Then releasing all $k$ outputs together is $(\varepsilon_{\text{tot}}, \delta_{\text{tot}})$-differentially private with

Post-processing says that any function of a differentially private output that does not access the private data again, even a randomized function using independent randomness, keeps the same $(\varepsilon, \delta)$. Refined accounting methods can give smaller totals in suitable settings. Every release counts, including repeated training runs, model selection and diagnostics computed from the protected data. A budget is not a number of people who may safely be exposed, and a calculation for one count does not certify a trained model as private, which needs a mechanism designed for training and a full accounting.

Definition Data poisoning is the deliberate insertion or alteration of training data, labels or other learning inputs, such as fine-tuning sets or user feedback, by someone who wants to change the fitted model, either its behavior overall or its behavior on inputs the attacker chooses. An adversarial input is an input crafted at prediction time, when the model is already fixed, to make it produce an unwanted output, often through a change that looks small or irrelevant to a person. Poisoning acts on learning and adversarial inputs act on prediction, so they call for different controls. Against poisoning these include tracking the provenance of data, restricting who can write training data and labels, validating changes and inspecting unusual shifts in values or labels. Against adversarial inputs they include validating the structure of inputs, testing the model against relevant manipulations and routing doubtful cases to a fallback or to human review. Neither ordinary noise nor a differential privacy guarantee replaces these controls. Privacy limits what a release reveals about participation, while integrity concerns how manipulation changes behavior, and a workflow can satisfy one and fail the other.