Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning

2.2.1 Attacker’s Goal.

The goal of a poisoning attack can be defined in terms of the intended security violation, and the attack and error specificity, as detailed below.

Security Violation. It defines the security violation caused by the attack, which can be: (i) an integrity violation, if malicious activities evade detection without compromising normal system operation; (ii) an availability violation, if normal system functionality is compromised, causing a denial of service for legitimate users; or (iii) a privacy violation, if the attacker aims to obtain private information about the system itself, its users, or its data.

Attack Specificity. It determines which samples are subject to the attack. It can be: (i) sample-specific (targeted), if a specific set of sample(s) is targeted by the attack; or (ii) sample-generic (indiscriminate), if any sample can be affected.

Error Specificity. It determines how the attack influences the model’s predictions. It can be: (i) error-specific, if the attacker aims to have a sample misclassified as a specific class; or (ii) error-generic, if the attacker attempts to have a sample misclassified as any class different from the true class.

2.2.2 Attacker’s Knowledge.

The attacker may get to know some details about the target system, including information about: (i) the (clean) training data \(\mathcal {D}\), (ii) the ML model \(\mathcal {M}\) being used, and (iii) the test data \(\mathcal {T}\). The first component considers how much knowledge the attacker has on the training data. The second component refers to the ability of the attacker to access the target model, including its internal (trained) parameters \(\boldsymbol {\theta }\), but also additional information such as hyperparameters, initialization, and the training algorithm. The third component specifies if the attacker knows in advance (or has access to) the samples that should be misclassified at test time. Although not explicitly mentioned in previous work, we have found that the knowledge of test samples is crucial for some attacks to work as expected. Clearly, attacks that are designed to work on specific test instances are not expected to generalize to different test samples (e.g., to other samples belonging to the same class). Depending on the combination of the previously defined properties, we can define two main attack settings, as detailed below.

White-box Attacks. The attacker has complete knowledge about the targeted system. Although not always representative of practical cases, this setting allows us to perform a worst-case analysis, and it is particularly helpful for evaluating defenses.

Black-box Attacks. Black-box attacks can be subdivided into two main categories: black-box transfer attacks and black-box query attacks. Although generally referred to as a black-box attack, black-box transfer attacks assume that the attacker has partial knowledge of the training data and/or the target model. In particular, the attacker is assumed to be able to collect a surrogate dataset and use it to train a surrogate model approximating the target. Then, white-box attacks can be computed against the surrogate model and subsequently transferred against the target model. Under some mild conditions, such attacks have been shown to transfer successfully to the target model with high probability [45]. It is also worth remarking that black-box query attacks can also be staged against a target model by only sending input queries to the model and observing its predictions to iteratively refine the attack without exploiting any additional knowledge [31, 132, 167]. However, to date, most of the poisoning attacks staged against learning algorithms in black-box settings exploit surrogate models and attack transferability.

2.2.3 Attacker’s Capability.

The attacker’s capability is defined in terms of how the attacker can influence the learning setting and on the data perturbation that can be applied to training and/or test samples.

Influence on Learning Setting. The three learning settings described in Section 2.1 open the door towards different data poisoning attacks. In both training-from-scratch (TS) and fine-tuning (FT) scenarios, the attacker alters a subset of the training dataset collected and used by the victim to train or fine-tune the machine learning model. Conversely, in the model-training (MT) scenario, as first hypothesized by Gu et al. [70], the attacker acts as a malicious third-party trainer or as a man-in-the-middle, controlling the training process. The attacker tampers with the training procedure and returns to the victim user a model that behaves according to their goal. The advantage for the attacker is the victim will never be aware of the training dataset actually used. However, to keep their attack stealthy, the attacker must ensure that the provided model retains high prediction accuracy, making sure to pass the validation phase without suspicion from the victim user. The attacker’s knowledge, discussed in Section 2.2.2, is defined depending on the setting under consideration. In the model-training and training-from-scratch settings, \(\mathcal {D}^\prime\) and \(\mathcal {M}\) refer to the training data and algorithm used for training the model from random initialization of its weights. Conversely, in the fine-tuning setting, \(\mathcal {D}^\prime\) and \(\mathcal {M}\) refer to the fine-tuning dataset and learning algorithm, respectively.

Data Perturbation. Staging a poisoning attack requires the attacker to manipulate a given fraction (p) of the training data. In some cases, i.e., in backdoor attacks, the attacker is also required to manipulate the test samples that are under their control by adding an appropriate trigger to activate the previously implanted backdoor at test time. More specifically, poisoning attacks can alter a fraction of the training labels and/or apply a (different) perturbation to each of the training (poisoning) samples. If the attack only modifies the training labels, but it does not perturb any training sample, then it is often referred to as a label-flip poisoning attack. Conversely, if the training labels are not modified (e.g., if they are validated or assigned by human experts or automated labeling procedures), then the attacker can stage a so-called clean-label poisoning attack. Such attacks only slightly modify the poisoning samples, using imperceptible perturbations that preserve the original semantics of the input samples along with their class labels [148]. We define the strategies used to manipulate training and test data in poisoning attacks in the next section.

2.2.4 Attack Strategy.

The attack strategy defines how the attacker manipulates data to stage the desired poisoning attack. Both indiscriminate and targeted poisoning attacks only alter the training data, while backdoor attacks also require embedding the trigger within the test samples to be misclassified. We revise the corresponding data manipulation strategies in the following.

Training Data Perturbation (\(\boldsymbol {\delta }\)). Two main categories of perturbation have been used to mount poisoning attacks. The former includes perturbations that are found by solving an optimization problem, either formalized as a bilevel (BL) programming problem or as a feature-collision (FC) problem. The latter involves the manipulation of training samples in targeted and backdoor poisoning attacks such that they collide with the target samples in the given representation space to induce misclassification of such target samples in an attacker-chosen class. When it comes to backdoor attacks, three main types of triggers exist, which can be applied to training samples to implant the backdoor during learning: patch triggers (T\(^P\)), which consist of replacing a small subset of contiguous input features with a patch pattern in the input sample; functional triggers (T\(^F\)), which are embedded into the input sample via a blending function; and semantical triggers (T\(^S\)), which perturb the given input while preserving its semantics (e.g., modifying face images by adding sunglasses or altering the face expression, but preserving the user identity). The choice of this strategy plays a fundamental role, since it influences the computational effort, effectiveness, and stealthiness of the attack. More concretely, the trigger strategies are less computationally demanding, as they do not require optimizing the perturbation, but the attack may be less effective and easier to detect. Conversely, an optimized approach can enhance the effectiveness and stealthiness of the attack at the cost of being more computationally demanding. In Figure 2, we give some examples of patch, functional, and semantical triggers, one example of a poisoning attack optimized with bilevel programming, and one example of a clean-label feature-collision attack.

Fig. 2.

View Figure

Test Data Perturbation (\(\boldsymbol {t}\)). During operation, i.e., at test time, the attacker can submit malicious samples to exploit potential vulnerabilities that were previously implanted during model training via a backdoor attack. In particular, as we will see in Section 3.3, backdoor attacks are activated when a specific trigger \(\boldsymbol {t}\) is present in the test samples. Normally, the test-time trigger is required to exactly match the trigger implanted during training, thus including patch, functional, and semantical triggers.

2.3.1 Defender’s Goal.

The defender’s goal is to preserve the integrity, availability, and privacy of their ML model, i.e., to mitigate any kind of security violation that might be caused by an attack. The defender thus adopts appropriate countermeasures to alleviate the effect of possible attacks without significantly affecting the behavior of the model for legitimate users.

2.3.2 Defender’s Knowledge and Capability.

The defender’s knowledge and capability determine in which learning setting a defense can be applied. We identify four aspects that influence how the defender can operate to protect the model: (i) having access to the (poisoned) training data \(\mathcal {D}^\prime\) and to (ii) a separate, clean validation set \(\mathcal {V}\) and (iii) having control on the training procedure \(\mathcal {W}\) and on (iv) the model’s parameters \(\boldsymbol {\theta }\). We will see in more detail how these assumptions are matched to each defense in Section 4.

2.3.3 Defense Strategy.

The defense strategy defines how the defender operates to protect the system from malicious attacks before deployment (i.e., at training time) and after the model’s deployment (i.e., at test time). We identify six distinct categories of defenses:

These defenses essentially work by either (i) cleaning the data or (ii) modifying the model. In the former case, the defender aims to sanitize training or test data. Training data sanitization and test data sanitization are two strategies adopted, respectively, at training and at test time to mitigate the influence of data poisoning attacks. Alternatively, the defender can act directly on the model, by (i) identifying possible internal vulnerabilities and removing/fixing components that lead to anomalous behavior/classifications or by (ii) changing the training procedure to make the model less susceptible to training data manipulations. The first approach is employed in model inspection, trigger reconstruction, and model sanitization defensive mechanisms. The second approach, instead, includes algorithms that operate at the training level to implement robust training mechanisms.

3.1.1 Label-flip Poisoning.

The most straightforward strategy to stage poisoning attacks against ML is label-flip, originally proposed by Biggio et al. [15]. The adversary does not perturb the feature values, but they mislabel a subset of samples in the training dataset, compromising the performance accuracy of ML models such as Support Vector Machines (SVMs). Beyond that, Xiao et al. [190] showed that random flips could have far-from-optimal performance, which nevertheless would require solving an NP-hard optimization problem. Due to its intractability, heuristic strategies have been proposed by Xiao et al. [190], and later by Xiao et al. [189], to efficiently approximate the optimal formulation.

3.1.2 Bilevel Poisoning.

In this case, the attacker manipulates both the training samples and their labels. The pioneering work in this direction was proposed by Biggio et al. [16], where a gradient-based indiscriminate poisoning attack is exploited against SVMs. They exploited implicit differentiation to derive the gradient required to optimize the poisoning samples by their iterative algorithm. Until convergence, the poisoning samples are iteratively updated following the implicit gradient, directing towards maximization of the model’s validation error. Mathematically speaking, this idea corresponds to treating the poisoning task as a bilevel optimization problem: (1) \(\begin{equation} \max _{\boldsymbol {\delta }\in \Delta } \;\;\;\; L (\mathcal {V}, \mathcal {M}, \boldsymbol {\theta }^\star), \end{equation}\) (2) \(\begin{equation} {\rm s.t.} \;\;\;\; \boldsymbol {\theta }^\star \in \mathop{\text{arg min}}\limits_{\theta} \, \mathcal {L} (\mathcal {D} \cup \mathcal {D}_p ^{\boldsymbol {\delta }}, \mathcal {M}, \boldsymbol {\theta }), \end{equation}\) with \(\Delta\) being the set of admissible manipulation of the training samples that preserve the constraints imposed by the attackers (e.g., \(\ell _p\), or box-constraints).⁷ We define with \(\mathcal {D}_p = \lbrace (\boldsymbol {x}_i, y_i)\rbrace _{i=1}^{n}\) the training data controlled by the attacker, before any perturbation is applied, being \(y_i\) the pristine label of sample \(\boldsymbol {x}_i\) and n the number of samples in \(\mathcal {D}_p\). We then denote with \(\mathcal {D}_p ^{\boldsymbol {\delta }}\) the corresponding poisoning dataset manipulated according to the perturbation parameter \(\boldsymbol {\delta }\). The attacker optimizes the perturbation \(\boldsymbol {\delta }\) (applied to the poisoning samples \(\mathcal {D}_p\)) to increase the error/loss L of the target model \(\mathcal {M}\) on the clean validation samples \(\mathcal {V}\). Our formulation in Equations (1) and (2) encompass both dirty or clean-label attacks according to the nature of \(\mathcal {D}_p ^{\boldsymbol {\delta }}\). For example, we can define \(\mathcal {D}_p ^{\boldsymbol {\delta }} = \lbrace (\boldsymbol {x}_i + \boldsymbol {\delta }_i, y_i^\prime)\rbrace _{i=1}^{n}\),⁸ being \(y_i^\prime\) the poisoning label chosen by the attacker, with \(y_i^\prime = y_i\) for a clean-label attack and \(y_i^\prime \ne y_i\) for a dirty-label attack. Solving this bilevel optimization is challenging, since the inner and the outer problems in Equations (1) and (2) have conflicting objectives. More concretely, the inner objective is a regularized empirical risk minimization, while the outer one is empirical risk maximization, both considering data from the same distribution. A similar approach was later generalized in Xiao et al. [188] and Frederickson et al. [58] to target feature selection algorithms (i.e., LASSO, ridge regression, and elastic net). Subsequent work tried to analyze the robustness of ML models when the attacker has limited knowledge about the training dataset or the victim’s classifier. In this scenario, the most investigated methodology is given by the transferability of the attack [45, 116, 153]. The attacker crafts the poisoning samples using surrogate datasets and/or models and then transfers the attack to another target model. This approach has proven effective for corrupting logistic classifiers [45], algorithmic fairness [153], and differentially private learners [116]. More details about the transferability of poisoning attacks are reported in Section 3.5.

Differently from previous work, Cinà et al. [42] observed that a simple heuristic strategy, together with a variable reduction technique, can reach noticeable results against linear classifiers, with increased computational efficiency. More concretely, the authors showed how previous gradient-based approaches can be affected by several factors (e.g., loss landscape) that degrade their performance in terms of computation time and attack efficiency.

Although effective, the aforementioned poisoning attacks have been designed to fool models with a relatively small number of parameters. More recently, Muñoz-González et al. [124] showed that devising poisoning attacks against larger models, such as convolutional neural networks, can be computationally and memory-demanding. To this end, Muñoz-González et al. [124] pioneered the idea to adapt hyperparameter optimization methods, which aims to solve bilevel programming problems more efficiently in the context of poisoning attacks. The authors indeed proposed a back-gradient descent technique to optimize poisoning samples, drastically reducing the attack complexity. The underlying idea is to back-propagate the gradient of the objective function to the poisoning samples while learning the poisoned model. However, they assume the objective function is sufficiently smooth to trace the gradient backward correctly. Accordingly with the results in Reference [124], Yang et al. [194] showed that computing the analytical or estimated gradient of the validation loss in Equation (1) with respect to the poisoning samples can be as well computational and query expensive. Another way explored in Yang et al. [194] was to train a generative model from which the poisoning samples are generated, thus increasing the generation rate.

3.1.3 Bilevel Poisoning (Clean-label).

Previous work examined in Section 3.1.2 assumes that the attacker has access to a small percentage of the training data and can alter both features and labels. Similar attacks have been staged by assuming that the attacker can control a much larger fraction of the training set, while only slightly manipulating each poisoning sample to preserve its class label, i.e., performing a clean-label attack. This idea was introduced by Mei and Zhu [121], who considered manipulating the whole training set to arbitrarily define the importance of individual features on the predictions of convex learners. More recently, DeepConfuse [53] and Fowl et al. [55] proposed novel techniques to mount clean-label poisoning attacks against DNNs. In Reference [53], the attacker trains a generative model, similarly to Reference [194], to craft clean-label poisoning samples that can compromise the victim’s model. Inspired by recent developments proposed in Reference [62], Fowl et al. [55] used a gradient alignment optimization technique to alter the training data imperceptibly, but diminishing the model’s performance. Even though Feng et al. [53] and Fowl et al. [55] can target DNNs, the attacker is assumed to perturb a high fraction of samples in the training set. We do believe that this is a very demanding setting for poisoning attacks. In fact, such attacks are often possible because ML is trained on data collected in the wild (e.g., labeled through tools such as a mechanical Turk) or crowdsourced from multiple users; thus, it would be challenging for attackers in many applications to realistically control a substantial fraction of these training data. In conclusion, the quest for scalable, effective, and practical indiscriminate poisoning attacks on DNNs is still open. Accordingly, it remains also unclear whether DNNs can be significantly subverted by such attacks in practical settings.

3.2.1 Bilevel Poisoning.

In Section 3.1.2, we reviewed the work in Muñoz-González et al. [124]. In addition to indiscriminate poisoning, the authors formulated targeted poisoning attacks as: (3) \(\begin{equation} \min _{\boldsymbol {\delta }\in \Delta } \;\;\;\; L (\mathcal {V}, \mathcal {M}, \boldsymbol {\theta }^\star) + L (\mathcal {V}_t, \mathcal {M},\boldsymbol {\theta }^\star), \end{equation}\) (4) \(\begin{equation} \text{s.t.} \;\;\;\; \boldsymbol {\theta }^\star \in \mathop{\arg min }\limits_{\boldsymbol {\theta }} \, \mathcal {L} (\mathcal {D} \cup \mathcal {D}_p ^{\boldsymbol {\delta }},\mathcal {M}, \boldsymbol {\theta }). \end{equation}\) Within this formulation, the attacker optimizes the perturbation \(\boldsymbol {\delta }\) on the poisoning samples \(\mathcal {D}_p\) to have a set of target (validation) samples \(\mathcal {V}_t\) misclassified, while preserving the accuracy on the clean (validation) samples in \(\mathcal {V}\). It is worth remarking here that the attack is optimized on a set of validation samples and then evaluated on a separate set of test samples. The underlying rationale is that the attacker can not typically control the specific realization of the target instances at test time (e.g., if images are acquired from a camera sensor, then the environmental and acquisition conditions can not be controlled), and the attack is thus expected to generalize correctly to that case.

A similar attack was introduced by Koh and Liang [92], to show the equivalence between gradient-based (bilevel) poisoning attacks and influence functions, i.e., functions defined in the area of robust statistics that identify the most relevant training points influencing specific predictions. Notably, these authors were the first to consider the fine-tuning (FT) scenario in their experiments, training the classification function f (i.e., an SVM with the RBF kernel) on top of a feature representation \(\phi\) extracted from an internal layer of a DNN. Although these two bilevel optimization strategies have been proven effective, they remain too computationally demanding to be applied to DNNs.

Jagielski et al. [84] showed how to generalize targeted poisoning attacks to an entire subpopulation in the data distribution while reducing the computational cost. To create subpopulations, the attacker selects data samples by matching their features or clustering them in feature space. The poisoning attack can be performed either by label flipping or linearizing the influence function to approximate the poisoning gradients, thus reducing the computational cost of the attack. Muñoz-González et al. [124] and Jagielski et al. [84] define a more ambitious goal for the attack compared to Koh and Liang [92], as their attacks aim to generalize to all samples coming from the target distribution or the given subpopulation. Specifically, the attack by Koh and Liang [92] is tailored for misleading the model only for some specific test samples, which means considering the test set \(\mathcal {T}\) rather than a validation set \(\mathcal {V}_t\) in Equation (3). However, the cost of the attack by Muñoz-González et al. [124] is quite high, due to need of solving a bilevel problem, while the attack by Jagielski et al. [84] is faster, but it does not achieve the same success rate on all subpopulations.

3.2.2 Feature Collision (Clean-label).

This category of attacks is based on a heuristic strategy named feature collision, suited to the so-called fine-tuning scenario, which avoids the need of solving a complex bilevel problem to optimize poisoning attacks. In particular, PoisonFrog [148] was the first work proposing this idea, which can be formalized as: (5) \(\begin{eqnarray} \min _{\boldsymbol {\delta }\in \Delta } & \Vert \phi (\boldsymbol {x} +\boldsymbol {\delta }) - \phi (\boldsymbol {z})\Vert _2^2. \end{eqnarray}\) This attack amounts to creating a poisoning sample \(\boldsymbol {x} + \boldsymbol {\delta }\) that collides with the target test sample \(\boldsymbol {z} \in \mathcal {T}\) in the feature space, so the fine-tuned model predicts \(\boldsymbol {z}\) according to the poisoning label associated with \(\boldsymbol {x}\). To this end, the adversary leverages the feature extractor \(\phi\) to minimize the distance of the poisoning sample with the target in the feature space. Moreover, the authors observed that, due to the complexity and nonlinear behavior of \(\phi\), even poisoning samples coming from different distributions can be slightly perturbed in the input space to match the feature representation of the target sample \(\boldsymbol {z}\). To make the poisoning sample look realistic in input space and implement a clean-label attack, the adversarial perturbation \(\boldsymbol {\delta }\in \Delta\) is bounded by the attacker in its \(\ell _p\) norm [148] (e.g., \(\Vert \boldsymbol {\delta }\Vert _2 \le \epsilon\)). Such box constraint can also be implemented as a soft constraint, as originally done by Shafahi et al. [148].⁹ Similarly, Guo and Liu [72] adopted feature collision to stage the attack, but they extended the attack’s objective function to further increase the poisoning effectiveness. Nevertheless, although this strategy turns out to be effective, it assumes that the feature extractor is fixed and that it is not updated during the fine-tuning process. Moreover, StringRay [158], ConvexPolytope [212], and BullseyePolytope [2] observed that when reducing the attacker’s knowledge, the poisoning effectiveness decreases. These works showed that feature collision is not practical if the attacker does not know exactly the details of the feature extractor, as the embedding of poisoning samples may not be consistent across different feature extractors. To mitigate these difficulties, ConvexPolytope [212] and BullseyePolytope [2] optimize the poisoning samples against ensemble models, constructing a convex polytope around the target samples to enhance the effectiveness of the attack. The underlying idea is that constructing poisoning samples against ensemble models may improve the attack transferability. The authors further optimize the poisoning samples by establishing a strong connection among all the layers and the embeddings of the poisoning samples, partially overcoming the assumption that the feature extractor \(\phi\) remains fixed.

All these approaches have the property of creating clean-label samples, as first proposed in Shafahi et al. [148], to stay undetected even when the class labels of training points are validated by humans. This is possible, as these attacks are staged against deep models, since for these models, small (adversarial) perturbations of samples in the input space correspond to large changes in their feature representations.

3.2.3 Bilevel Poisoning (Clean-label).

Although feature collision attacks are effective, they may not result in the optimal accuracy, and they do not minimize the number of poisoned points to change the model’s prediction on a single test point. Moreover, they assume that the training process is not significantly changing the feature embedding. Indeed, when the whole model is trained from scratch, these strategies may not work properly, as poisoning samples can be embedded differently. Recent developments, including MetaPoison [80] and the work by Geiping et al. [62], tackle the targeted poisoning attack in the training-from-scratch (TS) scenario, while ensuring the clean-label property. These approaches are derived from the bilevel formulation in Equations (3) and (4), but they exploit distinct and more scalable approaches to target DNNs and optimize the attack directly against the test samples \(\mathcal {T}\) as done in Reference [92]. More concretely, MetaPoison [80] uses a meta-learning algorithm, as done by Muñoz-González et al. [124], to decrease the computational complexity of the attack. They further enhance the transferability of their attack by optimizing the poisoning samples against an ensemble of neural networks, trained with different hyperparameter configurations and algorithms (e.g., weight initialization, number of epochs). Geiping et al. [62] craft poisoning samples to maximize the alignment between the inner loss and the outer loss in Equations (3) and (4). The authors observed that matching the gradient direction of malicious examples is an effective strategy for attacking DNNs trained from scratch, even on large training datasets. Although modern feature collision or optimized strategies are emerging with notable results for targeted attacks, their performance, especially in black-box settings, still demands further investigation.

3.3.1 Trigger Poisoning.

Earlier work in backdoor attacks considered three main families of backdoor triggers, i.e., patch, functional, and semantical triggers, as discussed below.

Patch. The first threat vector of attack for backdoor poisoning has been investigated in BadNets [70]. The authors considered the case where the user outsources the training process of a DNN to a third-party service, which maliciously alters the training dataset to implant a backdoor in the model. To this end, the attacker picks a random subset of the training data, blends the backdoor trigger into them, and changes their corresponding labels according to an attacker-chosen class. A similar idea has been investigated further in LatentBackdoor [196] and TrojanNN [110], where the backdoor trigger is designed to maximize the response of selected internal neurons, thus reducing the training data needed to plant the trigger. Additionally, LatentBackdoor [196] designed the trigger to survive even if the last layers are fine-tuned with novel clean data, while TrojanNN [110] does not need access to the training data, as a reverse-engineering procedure is applied to create a surrogate dataset. All these attacks assume that the trigger is always placed in the same position, limiting their application against specific defense strategies [5, 28, 155]. To overcome this issue, BaN [143] introduced different backdoor attacks where the trigger can be attached in various locations of the input image. The underlying idea was to force the model to learn the backdoor trigger and make it location-invariant.

Functional. The patch strategy is based on the idea that poisoning samples repeatedly present a fixed pattern as a trigger, which may, however, be detected upon human validation of training samples (in the TS and FT scenarios, at least). In contrast, a functional trigger represents a stealthier strategy, as the corresponding trigger perturbation is slightly spaced throughout the image or changes according to the input. Some works assume to slightly perturb the entire image so those small variations are not detectable by humans, but evident enough to mislead the model. In WaNET [129] warping functions are used to generate invisible backdoor triggers (see Figure 2). Moreover, the authors enforced the model to distinguish the backdoor warping functions among other pristine ones. In Li et al. [99] steganography algorithms are used to hide the trigger into the training data. Specifically, the attacker replaces the least significant bits to contain the binary string representing the trigger. In DFST [36], style transfer generative models are exploited to generate and blend the trigger. However, the aforementioned poisoning approaches assume that the attacker can change the labeling process and that no human inspection is done on the training data. This assumption is then relaxed by Barni et al. [8] and Liu et al. [111], where clean-label backdoor poisoning attacks are considered; in particular, Liu et al. [111] used natural reflection effects as trigger to backdoor the system, while Barni et al. [8] used an invisible sinusoidal signal as backdoor trigger (see Figure 2). More practical scenarios, where the attacker is assumed to have limited knowledge, have been investigated by Chen et al. [33] and Zhong et al. [211]. In these two works, the authors used the idea of blending fixed patterns to backdoor the model. In the former approach, Chen et al. [33] assume that the attacker blends image patterns into the training data and tunes the blend ratio to create almost invisible triggers, while impacting the backdoor’s effectiveness. In the latter, Zhong et al. [211] assume that an invisible grid pattern is generated to increase the pixel’s intensity, and its effectiveness is tested in the TS and FT settings.

Semantical. The semantical strategy incorporates the idea that backdoor triggers should be feasible and stealthy. For example, Sarkar et al. [145] used facial expressions or image filters (e.g., old-age, smile) as backdoor triggers against real-world facial recognition systems. At training time, the backdoor trigger is injected into the training data to cause the model to associate a smile filter with the authorization of a user. At test time, the attacker can use the same filter to mislead classification. Similarly, Chen et al. [33] and Wenger et al. [179] tried to poison face-recognition systems by blending physically implementable objects (e.g., sunglasses, earrings) as triggers.

3.3.2 Bilevel Poisoning.

Trigger-based strategies assume that the attacker uses a predefined perturbation to mount the attack. However, an alternative strategy for the attacker is to learn the trigger/perturbation itself to enhance the backdoor effectiveness. To this end, even backdoor poisoning can be formalized as a bilevel optimization problem: (6) \(\begin{equation} \min _{\boldsymbol {\delta }\in \Delta } \;\;\;\; L (\mathcal {V}, \mathcal {M}, \boldsymbol {\theta }^\star) + L (\mathcal {V}_t ^{\boldsymbol {t}}, \mathcal {M}, \boldsymbol {\theta }^\star),\\ \end{equation}\) (7) \(\begin{equation} \text{s.t.} \;\;\;\; \boldsymbol {\theta }^\star \in \mathop{\text{arg min}}\limits_{\theta} \, \mathcal {L} (\mathcal {D} \cup \mathcal {D}_p ^{\boldsymbol {\delta }},\mathcal {M}, \boldsymbol {\theta }). \end{equation}\)

Here, the attacker optimizes the training perturbation \(\boldsymbol {\delta }\) for poisoning samples in \(\mathcal {D}_p\) to mislead the model’s prediction for validation samples \(\mathcal {V}_t\) containing the backdoor trigger \(\boldsymbol {t}\). In contrast to indiscriminate and targeted attacks (in Sections 3.1 and 3.2), the attacker injects the backdoor trigger in the validation samples \(\boldsymbol {t}\) to cause misclassifications. Additionally, as for targeted poisoning, the error on \(\mathcal {V}\) is minimized to preserve the system’s functionality.

One way to address this bilevel formulation is to craft optimal poisoning samples using generative models [48, 101, 128], as also done in Reference [194] for indiscriminate poisoning. Nguyen and Tran [128] trained the generative model with a loss that enforces the diversity and noninterchangeable of the trigger, while LIRA’s [48] generator is trained to enforce effectiveness and invisibility of the triggers. Conversely, Li et al. [101] used a generative neural network steganography technique to embed a backdoor string into poisoning samples. Another way is to perturb training samples with adversarial noise, as done by Li et al. [99] and Zhong et al. [211]. More concretely, in the former approach, the trigger maximizes the response of specific internal neurons, and a regularization term is introduced in the objective function to make the backdoor trigger invisible. In the latter work, the attacker looks for the minimum universal perturbation that pushes any input towards the decision boundary of a target class. The attacker can use this invisible perturbation trigger on any image, inducing the model to misclassify the target class.

3.3.3 Feature Collision (Clean-label).

The backdoor trigger visibility influences the stealthiness of the attack. A backdoor trigger that is too obvious can be easily spotted when the dataset is inspected [142]. However, Hidden Trigger [142] introduced the idea of using the feature collision strategy, seen in Section 3.2.2 and formulated in Equation (5), to hide the trigger into natural target samples. Specifically, the attacker first injects a random patch trigger into the training set, and then each poisoning sample is masked via feature collision. The resulting poisoning images are visually indistinguishable from the target and have a consistent label (i.e., they are clean-label), while the test samples with the patch trigger will collide with the poisoning samples in feature space, ensuring that the attack works as expected.

Although the work in Reference [142] implements an effective and stealthy clean-label attack, it is applicable only in the feature extractor \(\phi\) is not updated. Such a limitation is mitigated by Turner et al. [169], who exploit a surrogate latent space, rather than \(\phi\), to interpolate the backdoor samples, hiding the training-time trigger. Moreover, the attacker can tune the trigger visibility at test time to enhance the attack’s effectiveness.

3.3.4 Bilevel Poisoning (Clean-label).

Inspired by recent success of the gradient-alignment technique in Reference [62] for targeted poisoning, Souri et al. [156] exploited the same bilevel-descending strategy to stage clean-label backdoor poisoning attacks in the training-from-scratch scenario. Similarly to Saha et al. [142], the training and the test data perturbations are different, enhancing the stealthiness of the attack and making it stronger against existing defenses.

3.4.1 Unrealistic Threat Models.

The first challenge we formulate here questions some of the threat models considered in previous work. The reason is that such threat models are not well representative of what may happen in many real-world scenarios for the attackers. They are valuable, because they allow system designers to test the system’s robustness under worst-case scenarios, but their practicability and effectiveness against realistic production systems are unknown. To give an accurate estimate of how poisoning attacks are effective against ML production systems, we should consider assumptions that are less favorable to the attacker. For example, Fowl et al. [55] and Feng et al. [53] assume that the attacker controls almost the entire training dataset to effectively mount an indiscriminate poisoning attack against DNNs. While this may happen in certain hypothesized situations, it is also not quite surprising that a poisoning attack works if the attacker controls a large fraction of the training set. We believe that poisoning attacks that assume that only a small fraction of the training points can be controlled by the attacker are more realistic and, therefore, viable against real production systems. We refer the reader to a similar discussion in the context of federated learning poisoning in Reference [150].

Another limitation of threat models considered for poisoning attacks is that, in some cases, exact knowledge of the test samples is implicitly assumed. For example, References [148] and [62] optimize a targeted poisoning attack to induce misclassification of few specific test samples. In particular, the attack is both optimized and tested using the same test samples, differently from work that optimizes the poisoning samples using validation data and then tests the attack impact on a separate test set [16, 124]. This evaluation setting clearly enables the attack to reach higher success rates, but at the same time, there is no guarantee that the attack will generalize even to minor variations of the considered test samples, questioning its applicability outside of settings in which the attacker has exact knowledge of the test inputs. For instance, the attack may not work as expected in physical domains, where images are acquired by a camera under varying illumination and environmental conditions. In such cases, it is indeed clear that the attacker can not know beforehand the specific realization of the test sample, as they do not control the acquisition conditions. On a similar note, only a few studies on backdoor poisoning have considered real-world scenarios where external factors (such as lighting, camera orientation, etc.) can alter the trigger. Indeed, as done in References [148] and [62], most papers consider digital applications where the implanted trigger is nearly unaltered. In conclusion, although some recent works seem to have improved the effectiveness of poisoning attacks, their assumptions are often not representative of the actual production system or the attacker’s settings, limiting their applicability only in the proposed context.

3.4.2 Computational Complexity of Poisoning Attacks.

The second challenge we discuss here is related to the solution of the bilevel programming problem used to optimize poisoning attacks. The problem, as analyzed by Muñoz-González et al. [124], is that solving the bilevel formulation with a gradient-based approach requires computing and inverting the Hessian matrix associated to the equilibrium conditions of the inner learning problem, which scales cubically in time and quadratically in space with respect to the number of model’s parameters. Even if one may exploit rank-one updates to the Hessian matrix, and Hessian-vector products coupled with conjugate descent to speed up the computation of required gradients, the approach remains too computationally demanding to attack modern deep models, where the number of parameters is on the order of millions. Nevertheless, it is also true that that solving the bilevel problem is expected to improve the effectiveness of the attack and its stealthiness against defenses. For example, the bilevel strategy approach is the only one at the state-of-the-art that allows mounting an effective attack in the training-from-scratch (TS) setting. Other heuristic approaches, e.g., feature collision, have been shown to be totally ineffective if the feature extractor \(\phi\) is updated during training [62]. For backdoor poisoning, the recent developments in the literature show that bilevel-inspired attacks are more effective and can better counter existing defenses [48, 128, 156]. Thus, tackling the complexity of the bilevel poisoning problem remains a relevant open challenge to ensure a fairer and scalable evaluation of modern deep models against such attacks.

4.7.1 Inconsistent Defense Settings.

The assumptions on the defender’s knowledge and capabilities reflect what is required to deploy a defense. In indiscriminate defenses, or robust training and training data sanitization in general, these are very homogeneous. When it comes to model inspection, trigger reconstruction, model sanitization, and test data sanitization, there is a larger variation in both the defender’s knowledge and capabilities. In particular, we lack understanding on the effect of individual capabilities or knowledge, for example, not having direct access to the model when provided as a service and interacting via queries. More work is required that enables comparison across approaches here and that sheds light on the individual components of the defense setting.

4.7.2 Insufficient Defense Evaluations.

In Table 4, we match poisoning attack strategies and defenses by reporting in each cell the defense papers that evaluate against the corresponding attack strategy. In some cases, indicated with a cross (✗), a defense of this strategy is not possible, as there is no trigger to reconstruct (indiscriminate or targeted) or the test data is not altered by the attacker and can thus not be sanitized (indiscriminate attacks). Furthermore, Table 4 shows that the amount of defenses per attack strategies varies greatly. Whereas for backdoor attacks using patch triggers there are around 50 defenses, only 11 defenses have been considered against semantic triggers, 1 against bilevel targeted attacks [58], 1 against bilevel patch backdoor attacks [195], and none against indiscriminate clean label bilevel attacks. With only a few defenses [163, 214], there is also a shortage of model inspection and sanitization defenses when no trigger manifests in the model.

Beyond this shortage, there is a need to thoroughly test existing defenses using adaptive attacks, which are depicted in Table 5. Adaptive attacks are tailored to circumvent one or several defenses. In other words, the attack identifies essential components like, for example, a threshold within a defense and adapts the poisoning points to be below this threshold. For example, Koh et al. [93] constrain the indiscriminate poisons features so several points are in close vicinity to avoid outlier detection. In the case of backdoors, Shokri et al. [152] regularize the trigger to be less detectable within the network. Tang et al. [163] and Lin et al. [105] employ different strategies to make training data with trigger more similar to benign data. Yet, as visible in Table 5, current adaptive backdoor attacks tend to break the same defenses. More work is thus needed to understand all defenses’ limitations through adaptive attacks, even though systematizing the design of such attacks and automating the corresponding evaluations is not trivial. To this end, it may be interesting to design indicators of failure that automate the identification of faulty, non-adaptive evaluations for poisoning attacks, as recently shown in Reference [136] for adversarial examples.

4.7.3 Overly Specialized Defenses.

Furthermore, few defenses (only roughly one-sixth) have been evaluated against different kinds of triggers. Only one defense in test data sanitization [200] and two defenses in trigger reconstruction [78, 109] have been evaluated against more than one trigger type. There are three defenses for each training data sanitization [75, 149, 186], model sanitization [100, 199, 200], and robust training [22, 61, 102]. In model inspection, five [71, 151, 155, 163, 193] defenses test on more than one attack type. There are even more general defenses that are able to handle multiple poisoning attacks, such as indiscriminate, targeted, and backdoor attacks, as, for example, Geiping et al. [61] and Hong et al. [77] show.

7.1.1 Attack Timeline.

The attack timeline is shown in Figure 5. To the best of our knowledge, the first example of indiscriminate poisoning was developed in 2006 by Perdisci et al. [134], Barreno et al. [9], and Newsome et al. [127] in the computer security area. Such attacks, as well as subsequent attacks in the same area [90, 141], were based on heuristic approaches to mislead application-specific ML models, and there was not a unifying mathematical formulation describing them. It was only later, in 2012, that indiscriminate poisoning against machine learning was formulated for the first time as a bilevel optimization [190] to compute optimal label-flip poisoning attacks. Since then, indiscriminate poisoning has been studied under two distinct settings, i.e., assuming either (i) that a small fraction of training samples can be largely perturbed [16, 45, 124]; or (ii) that all training points can be slightly perturbed [53, 55, 121].

Targeted and backdoor poisoning attacks only appeared in 2017, and interestingly, they both started from different strategies. Targeted poisoning started with the bilevel formulation in Koh and Liang [92], but evolved in more heuristic approaches, such as feature collision [72, 148, 212]. Only recently, targeted poisoning attacks were reformulated as bilevel problems, given the limitation of the aforementioned heuristic approaches [62, 80]. Backdoor poisoning started with the adoption of patch [70, 110] and functional [111, 128] triggers. However, in the last years, such heuristic choices have been put aside, and backdoor attacks are getting closer to the idea of formulating them in terms of a bilevel optimization, not only to enhance their effectiveness, but also their ability to bypass detection [142, 156].

The historical development of the three types of attacks is primarily aimed at solving or mitigating as much as possible the challenges highlighted in Section 3.4, i.e., (i) considering more realistic threat models and (ii) designing more effective and scalable poisoning attacks. In particular, recent developments in attacks seek to improve the applicability of their threat models by tampering with the training data as little as possible (e.g., a few points altered with invisible perturbations) to evade defenses and by considering more practical settings (e.g., training-from-scratch). Moreover, more recent poisoning attacks aim to tackle the computational complexity and time required to solve the bilevel problem, not only to improve attack scalability but also their ability to stay undetected against current defenses. In Section 7.2, we more thoroughly discuss these challenges, along with some possible future research directions to address them.

7.1.2 Defense Timeline.

The defense timeline is shown in Figure 6. The first defenses, training data sanitization and robust training variants, were introduced 2008 and 2009 in a security context [15, 43, 125]. Following works in training data sanitization were based on outlier detection, and mitigated backdoor [168], indiscriminate [96], and targeted [58] attacks. To train robustly, Biggio et al. [13] showed in 2011 that regularization can serve as a defense, a finding recently confirmed for backdoors [25]. In 2019, differential privacy was shown to be able to mitigate poisoning attacks [77, 116]. This connection to privacy underlines the need to study poisoning also in relation to other ML security issues, as we will discuss in Section 7.2.4. The remaining kinds of defenses are characterized by more diverse threat models, as we discussed in Section 4.7.1. The type of attack mitigated is, however, less diverse and focuses mainly on backdoors, as explained in Section 4.7.3. We start with model inspection approaches, which were first introduced by Chen et al. [28] and were based on outlier detection on latent representations. In 2020, Kolouri et al. [94] generalized the backdoor inspections to be model-independent using a meta-classifier. Recently, Zhu et al. [214] introduced a search-based approach to determine whether a model suffers from targeted poisoning. The latter approach also proposed how to sanitize the model. The first defenses for such model sanitization against backdoors were trigger-agnostic and based on fine-tuning [107, 112] and later on, data augmentation [200]. Another possibility, introduced by Wang et al. [173], is to retrain a model based on a reconstructed trigger. Wang et al. [173] introduced the idea of reconstruction a trigger in 2019. They generated a trigger based on optimization of a pattern that causes backdoor behavior, e.g., misclassification of many samples when added to them. More recent approaches improve trigger reconstruction by considering distributions over triggers [138], not individual patterns. A reconstructed trigger can also serve to inspect a model [173] or serve to sanitize test data [173]. However, the first approaches to sanitize test data in 2017 were based on outlier detection to inspect the model inspection and sanitize training data. Analogous to model inspection, initial works relied on latent, model-specific features [112], whereas later works from 2020 use model-agnostic input transformations [104].

One historical development that is highly relevant but left out in both timeline figures is the study of adaptive attacks against defenses to assess their robustness, as discussed in Section 4.7.2. We elaborate on this challenge in Section 7.2.3.

7.2.1 Considering Realistic Threat Models.

One pertinent challenge arising from the discussion on poisoning attacks in Section 3.4.1 demands considering more realistic threat models and attack scenarios, as also recently pointed out in Reference [150]. While assessing machine learning models in real-world settings is not straightforward [154], the need to develop realistic threat models is still an open question in machine learning security and has so far only received recognition for test-time attacks [63]. Here, we define some guidelines that can serve as a basis for future work that wants to assess the real safety impact of poisoning versus real applications. First, limit the attacker’s knowledge of the target system and their capacity to tamper during training. For example, an attack that assumes only a small percentage of control over the training set can be broadly applied. Second, develop more stealthy poisoning strategies to avoid detection against defenses. Some attack strategies, e.g., patch trigger or feature collision, are computationally efficient, but several defensive countermeasures exist to detect them (see Table 4). Finally, evaluating poisoning attacks against real-world applications and making them adaptive to the presence of a defender. Therefore, we invite the research community to evaluate poisoning attacks with more realistic or less favorable assumptions for the attacker, which also take into account the specific application domain.

7.2.2 Designing More Effective and Scalable Poisoning Attacks.

The other challenge we highlighted in Section 3.4.2 is the computational complexity of poisoning attacks when relying on bilevel optimization. However, the same limitation is also encountered in other research domains such as hyperparameter optimization and meta-learning, which naturally are formulated within the mathematical framework of bilevel programming [57]. More concretely, the former is the process of determining the optimal combination of hyperparameters that maximizes the performance of an underlying learning algorithm. However, meta-learning encompasses feature selection, algorithm selection, learning to learn, or ensemble learning, to which the same reasoning applies. Having formulated poisoning attacks within the bilevel framework (see Section 3.6) hints that strategies developed to speed up the optimization of bilevel programs involved in meta-learning or hyperparameter optimization taks can be adapted to facilitate the development of novel scalable attacks. In principle, by imagining poisoning samples as the attacker-controlled learning hyperparameters, we could apply the approaches proposed in these two fields to mount an attack. Notably, we find some initial works connecting these two fields with data poisoning. For example, Shen et al. [151] rely in their approach on a k-arms technique, a technique similar to bandits, as done by Jones et al. [87]. Further, Muñoz-González et al. [124] exploited the back-gradient optimization technique proposed in References [49, 117], originally proposed for hyperparameter optimization, and subsequently, Huang et al. [80] inherited the same approach, making the attack more effective against deep neural networks. Apart from the work just mentioned, the connection between the two fields and poisoning is still currently under-investigated, and other ideas could still be explored. For example, the optimization proposed by Reference [114] can further reduce runtime complexity and memory usage even when dealing with millions of hyperparameters. Or another way might be to move away from gradient-based approaches and consider gradient-free approaches, thus overcoming the complexity of the inverting the Hessian matrix seen in Section 3.4.2. In the area of gradient-free methods, the most straightforward way is to use grid or random search [11], which can be sped up using reinforcement learning [98]. Also, Bayesian optimization has been used, given a few sampled points from the objective and constraint functions, to approximate the target function [87]. Last but not least, evolutionary algorithms [198] as well as particle swarm optimization [113] have shown to be successful.

In conclusion, we consider these two fields as possible future research directions to find more effective and scalable poisoning attacks for assessing ML robustness in practice.

7.2.3 Systematizing and Improving Defense Evaluations.

Regardless of future attacks, we need to systematize and understand the limits of existing (and future) defenses better. As we have seen in Section 6, there is no coherent benchmark for defenses. Such a benchmark exposes flawed evaluations and assesses the robustness of a defense per se or in relation to other defenses (taking into account the defense’s setting, as discussed in Section 4.7.2). Jointly with benchmarks, evaluation guidelines, as discussed for ML evasion by Carlini et al. [24], help to improve defense evaluation. More specifically, these guidelines can encompass knowledge when attacks fail and why, similar to work on evasion attack failure [136]. Crucial in this context is also, as discussed in Section 4.7.2, to expand our understanding of adaptive attacks.

An orthogonal question is how to increase existing knowledge about tradeoffs between, for example, attack strength and stealthiness for indiscriminate attacks [58] or backdoors [33, 143, 169]. Further tradeoffs relate clean accuracy and accuracy under the poisoning attack by hyperparameter tuning. More concretely, Demontis et al. [45] and Cinà et al. [40] showed that more regularized classifiers tend to resist better to poisoning attacks, at the cost of slightly reducing clean accuracy. Ideally, impossibility results further increase our knowledge about hard limitations. To the best of our knowledge, the only impossibility results provided thus far for subpopulation poisoning attacks can be found in Reference [84], showing that it is impossible to defend poisoning attacks that target only a fraction of the data. Expanding our knowledge about tradeoffs and impossibilities will help to design and configure effective defenses.

7.2.4 Designing Generic Defenses against Multiple Attacks.

Defenses against poisoning attacks are often designed to target specific types of attacks (as discussed in the Section 4.7.3), leading to a limitation where they may not be effective against new attack types. Therefore it is important for defense mechanisms to be versatile and capable of detecting a wide range of potential poisoning attacks. Some defenses, however, do evaluate several poisoning attacks [61, 77, 151] or even different ML security threats such as backdoors and evasion [160] or poisoning and privacy [77].

In addition to creating more robust defenses, such interdisciplinary works also increase our understanding of how poisoning interferes with non-poisoning ML attacks [27, 178]. One attack is evasion, where a small perturbation is added to a sample at test time to force the model to misclassify an output. Evasion is closely related, but different from backdoors, which add a fixed perturbation at training time, causing an upfront known vulnerability at test time. Only a few works study evasion and poisoning together. For example, Sun et al. [160] introduce a defense against both backdoors and adversarial examples. Furthermore, Fowl et al. [56] show that adversarial examples with the original labels are strong poisons at training time. In the opposite direction, Weng et al. [178] find that if backdoor accuracy is high, evasion tends to be less successful and vice versa. Furthermore, Mehra et al. [120] study poisoning of certified evasion defenses: Using poisoning, they decrease the certified radius and accuracy. Two works, namely, by Manoj and Blum [118] and Goldwasser et al. [65], relate evasion and backdoors in a theoretical way. Both share rigid assumptions, however, Manoj and Blum [118] show an impossibility results in terms of non-existence of backdoor for some natural learning problems. Goldwasser et al. [65], however, show that backdoor detection might be impossible. In relation to privacy or intellectual property, poisoning can be used to increase the information leakage from training data at test time on collaborative learning [27]. Privacy can further be a defense against poisoning [22, 77], or poisoning can be a tool to obtain [55]. Summarizing, there is little knowledge on how poisoning interacts with other attacks. More work is needed to understand to understand the relationship between these different threats and secure machine learning models against them simultaneously.