Detect adversarial inputs using Amazon SageMaker Model Monitor and Amazon SageMaker Debugger

Analysis over the previous few years has proven that machine studying (ML) fashions are weak to adversarial inputs, the place an adversary can craft inputs to strategically alter the mannequin’s output (in image classification, speech recognition, or fraud detection). For instance, think about you will have deployed a mannequin that identifies your staff primarily based on photographs of their faces. As demonstrated within the whitepaper Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition, malicious staff could apply refined however rigorously designed modifications to their picture and idiot the mannequin to authenticate them as different staff. Clearly, such adversarial inputs—particularly if there are a big quantity of them—can have a devastating enterprise impression.

Ideally, we wish to detect every time an adversarial enter is distributed to the mannequin to quantify how adversarial inputs are impacting your mannequin and enterprise. To this finish, a large class of strategies analyze particular person mannequin inputs to test for adversarial conduct. Nevertheless, energetic analysis in adversarial ML has led to more and more subtle adversarial inputs, a lot of that are recognized to make detection ineffective. The rationale for this shortcoming is that it’s troublesome to attract conclusions from a person enter as as to whether it’s adversarial or not. To this finish, a current class of strategies focuses on distributional-level checks by analyzing a number of inputs at a time. The important thing thought behind these new strategies is that contemplating a number of inputs at a time allows extra highly effective statistical evaluation that isn’t doable with particular person inputs. Nevertheless, within the face of a decided adversary with deep information of the mannequin, even these superior detection strategies can fail.

Nevertheless, we will defeat even these decided adversaries by offering the protection strategies with further data. Particularly, as a substitute of simply the analyzing mannequin inputs, analyzing the latent representations collected from the intermediate layers in a deep neural community considerably strengthens the protection.

On this publish, we stroll you thru detect adversarial inputs utilizing Amazon SageMaker Mannequin Monitor and Amazon SageMaker Debugger for a picture classification mannequin hosted on Amazon SageMaker.

To breed the totally different steps and outcomes listed on this publish, clone the repository detecting-adversarial-samples-using-sagemaker into your Amazon SageMaker pocket book occasion and run the pocket book.

Detecting adversarial inputs

We present you detect adversarial inputs utilizing the representations collected from a deep neural community. The next 4 photographs present the unique coaching picture on the left (taken from the Tiny ImageNet dataset) and three photographs produced by the Projected Gradient Descent (PGD) assault [1] with totally different perturbation parameters ϵ. The mannequin used right here was ResNet18. The ϵ parameter defines the quantity of adversarial noise added to the photographs. The unique picture (left) is accurately predicted as class 67 (goose). The adversarially modified photographs 2, 3, and 4 are incorrectly predicted as class 51 (mantis) by the ResNet18 mannequin. We are able to additionally see that photographs generated with small ϵ are perceptually indistinguishable from the unique enter picture.

Subsequent, we create a set of regular and adversarial photographs and use t-Distributed Stochastic Neighbor Embedding (t-SNE [2]) to visually evaluate their distributions. t-SNE is a dimensionality discount methodology that maps high-dimensional knowledge right into a 2- or three-dimensional house. Every knowledge level within the following picture presents an enter picture. Orange knowledge factors current the traditional inputs taken from the check set, and blue knowledge factors point out the corresponding adversarial photographs generated with an epsilon of 0.003. If regular and adversarial inputs are distinguishable, then we might anticipate separate clusters within the t-SNE visualization. As a result of each belong to the identical cluster, which means that a detection approach that focuses solely on adjustments within the mannequin enter distribution can’t distinguish these inputs.

Let’s take a better have a look at the layer representations produced by totally different layers within the ResNet18 mannequin. ResNet18 consists of 18 layers; within the following picture, we visualize the t-SNE embeddings for the representations for six of those layers.

Because the previous determine reveals, pure and adversarial inputs change into extra distinguishable for deeper layers of the ResNet18 mannequin.

Primarily based on these observations, we use a statistical methodology that measures distinguishability with speculation testing. The tactic consists of a two-sample test utilizing maximum mean discrepancy (MMD). MMD is a kernel-based metric for measuring the similarity between two distributions producing the information. A two-sample check takes two units that comprise inputs drawn from two distributions, and determines whether or not these distributions are the identical. We evaluate the distribution of inputs noticed within the coaching knowledge and evaluate it with the distribution of the inputs obtained throughout inference.

Our methodology makes use of these inputs to estimate the p-value utilizing MMD. If the p-value is larger than a user-specific significance threshold (5% in our case), we conclude that each distributions are totally different. The edge tunes the trade-off between false positives and false negatives. The next threshold, equivalent to 10%, decreases the false unfavourable charge (there are fewer instances when each distributions had been totally different however the check failed to point that). Nevertheless, it additionally leads to extra false positives (the check signifies each distributions are totally different even when that isn’t the case). However, a decrease threshold, equivalent to 1%, leads to fewer false positives however extra false negatives.

As a substitute of making use of this methodology solely on the uncooked mannequin inputs (photographs), we use the latent representations produced by the intermediate layers of our mannequin. To account for its probabilistic nature, we apply the speculation check 100 instances on 100 randomly chosen pure inputs and 100 randomly chosen adversarial inputs. Then we report the detection charge as the share of assessments that resulted in a detection occasion based on our 5% significance threshold. The upper detection charge is a stronger indication that the 2 distributions are totally different. This process provides us the next detection charges:

• Layer 1: 3%
• Layer 4: 7%
• Layer 8: 84%
• Layer 12: 95%
• Layer 14: 100%
• Layer 15: 100%

Within the preliminary layers, the detection charge is somewhat low (lower than 10%), however will increase to 100% within the deeper layers. Utilizing the statistical check, the tactic can confidently detect adversarial inputs in deeper layers. It’s usually enough to easily use the representations generated by the penultimate layer (the final layer earlier than the classification layer in a mannequin). For extra subtle adversarial inputs, it’s helpful to make use of representations from different layers and combination the detection charges.

Resolution overview

Within the earlier part, we noticed detect adversarial inputs utilizing representations from the penultimate layer. Subsequent, we present automate these assessments on SageMaker by utilizing Mannequin Monitor and Debugger. For this instance, we first practice a picture classification ResNet18 mannequin on the tiny ImageNet dataset. Subsequent, we deploy the mannequin on SageMaker and create a customized Mannequin Monitor schedule that runs the statistical check. Afterwards, we run inference with regular and adversarial inputs to see how efficient the tactic is.

Seize tensors utilizing Debugger

Throughout mannequin coaching, we use Debugger to seize representations generated by the penultimate layer, that are used afterward to derive details about the distribution of regular inputs. Debugger is a characteristic of SageMaker that lets you seize and analyze data equivalent to mannequin parameters, gradients, and activations throughout mannequin coaching. These parameter, gradient, and activation tensors are uploaded to Amazon Easy Storage Service (Amazon S3) whereas the coaching is in progress. You may configure guidelines that analyze these for points equivalent to overfitting and vanishing gradients. For our use case, we solely wish to seize the penultimate layer of the mannequin (.*avgpool_output) and the mannequin outputs (predictions). We specify a Debugger hook configuration that defines an everyday expression for the layer representations to be collected. We additionally specify a save_interval that instructs Debugger to gather this knowledge through the validation part each 100 ahead passes. See the next code:

from sagemaker.debugger import DebuggerHookConfig, CollectionConfig

debugger_hook_config = DebuggerHookConfig(
      collection_configs=[ 
          CollectionConfig(
                name="custom_collection",
                parameters=.*ResNet_output",
                             "eval.save_interval": "100" )])

Run SageMaker coaching

We cross the Debugger configuration into the SageMaker estimator and begin the coaching:

import sagemaker 
from sagemaker.pytorch import PyTorch

function = sagemaker.get_execution_role()

pytorch_estimator = PyTorch(entry_point="practice.py",
                            source_dir="code",
                            function=function,
                            instance_type="ml.p3.2xlarge",
                            instance_count=1,
                            framework_version='1.8',
                            py_version='py3',
                            hyperparameters = {'epochs': 25, 
                                               'learning_rate': 0.001},
                            debugger_hook_config=debugger_hook_config
                           )
pytorch_estimator.match()

Deploy a picture classification mannequin

After the mannequin coaching is full, we deploy the mannequin as an endpoint on SageMaker. We specify an inference script that defines the model_fn and transform_fn features. These features specify how the mannequin is loaded and the way incoming knowledge must be preprocessed to carry out the mannequin inference. For our use case, we allow Debugger to seize related knowledge throughout inference. Within the model_fn operate, we specify a Debugger hook and a save_config that specifies that for every inference request, the mannequin inputs (photographs), the mannequin outputs (predictions), and the penultimate layer are recorded (.*avgpool_output). We then register the hook on the mannequin. See the next code:

def model_fn(model_dir):
    
    #create mannequin    
    mannequin = create_and_load_model(model_dir)
    
    
    #hook configuration
    tensors_output_s3uri = os.environ.get('tensors_output')
    
    #seize layers for each inference request
    save_config = smd.SaveConfig(mode_save_configs={
        smd.modes.PREDICT: smd.SaveConfigMode(save_interval=1),
    })
   
    #configure Debugger hook
    hook = smd.Hook(
        tensors_output_s3uri,
        save_config=save_config,
        include_regex='.*avgpool_output|.*ResNet_output_0|*ResNet_input',
    )
    
    #register hook
    hook.register_module(mannequin) 
    
    #set mode
    hook.set_mode(modes.PREDICT)
    
    return mannequin

Now we deploy the mannequin, which we will do from the pocket book in two methods. We are able to both name pytorch_estimator.deploy() or create a PyTorch mannequin that factors to the mannequin artifact information in Amazon S3 which were created by the SageMaker coaching job. On this publish, we do the latter. This permits us to cross in surroundings variables into the Docker container, which is created and deployed by SageMaker. We’d like the surroundings variable tensors_output to inform the script the place to add the tensors which are collected by SageMaker Debugger throughout inference. See the next code:

from sagemaker.pytorch import PyTorchModel

sagemaker_model = PyTorchModel(
    model_data=pytorch_estimator.model_data,
    function=function,
    source_dir="code",
    entry_point="inference.py",
    env={
          'tensors_output': f's3://{sagemaker_session.default_bucket()}/data_capture/inference',
        },
    framework_version='1.8',
    py_version='py3',
)

Subsequent, we deploy the predictor on an ml.m5.xlarge occasion kind:

predictor = sagemaker_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.xlarge",
    data_capture_config=data_capture_config,
    deserializer=sagemaker.deserializers.JSONDeserializer(),
)

Create a customized Mannequin Monitor schedule

When the endpoint is up and operating, we create a personalized Mannequin Monitor schedule. It is a SageMaker processing job that runs on a periodic interval (equivalent to hourly or each day) and analyzes the inference knowledge. Mannequin Monitor gives a pre-configured container that analyzes and detects knowledge drift. In our case, we wish to customise it to fetch the Debugger knowledge and run the MMD two-sample check on the retrieved layer representations.

To customise it, we first outline the Mannequin Monitor object, which specifies on which occasion kind these jobs are going to run and the situation of our customized Mannequin Monitor container:

from sagemaker.model_monitor import ModelMonitor

monitor = ModelMonitor(
    base_job_name="ladis-monitor",
    function=function,
    image_uri=processing_repository_uri,
    instance_count=1,
    instance_type="ml.m5.massive",
    env={ 'training_data':f'{pytorch_estimator.latest_job_debugger_artifacts_path()}', 
          'inference_data': f's3://{sagemaker_session.default_bucket()}/data_capture/inference'},
)

We wish to run this job on an hourly foundation, so we specify CronExpressionGenerator.hourly() and the output areas the place evaluation outcomes are uploaded to. For that we have to outline ProcessingOutput for the SageMaker processing output:

from sagemaker.model_monitor import CronExpressionGenerator, MonitoringOutput
from sagemaker.processing import ProcessingInput, ProcessingOutput

#inputs and outputs for scheduled monitoring job
vacation spot = f's3://{sagemaker_session.default_bucket()}/data_capture/outcomes'
processing_output = ProcessingOutput(
    output_name="outcome",
    supply="/choose/ml/processing/outcomes",
    vacation spot=vacation spot,
)
output = MonitoringOutput(supply=processing_output.supply, vacation spot=processing_output.vacation spot)

#create schedule
monitor.create_monitoring_schedule(
    output=output,
    endpoint_input=predictor.endpoint_name,
    schedule_cron_expression=CronExpressionGenerator.hourly(),
)

Let’s look nearer at what our customized Mannequin Monitor container is operating. We create an evaluation script, which hundreds the information captured by Debugger. We additionally create a trial object, which allows us to entry, question, and filter the information that Debugger saved. With the trial object, we will iterate over the steps saved through the inference and coaching phases trial.steps(mode).

First, we fetch the mannequin outputs (trial.tensor("ResNet_output_0")) in addition to the penultimate layer (trial.tensor_names(regex=".*avgpool_output")). We do that for the inference and validation phases of coaching (modes.EVAL and modes.PREDICT). The tensors from the validation part function an estimation of the traditional distribution, which we then use to match the distribution of inference knowledge. We created a category LADIS (Detecting Adversarial Enter Distributions through Layerwise Statistics). This class gives the related functionalities to carry out the two-sample check. It takes the listing of tensors from the inference and validation phases and runs the two-sample check. It returns a detection charge, which is a price between 0–100%. The upper the worth, the extra doubtless that the inference knowledge follows a special distribution. Moreover, we compute a rating for every pattern that signifies how doubtless a pattern is adversarial and the highest 100 samples are recorded, in order that customers can additional examine them. See the next code:

import LADIS
import sample_selection

#entry tensors saved throughout coaching
trial = create_trial("s3://xxx/coaching/debug-output/")

#iterate over validation steps saved by Debugger throughout coaching
for step in trial.steps(mode=modes.EVAL):
       
   #get mannequin outputs
   tensor = trial.tensor("ResNet_output_0").worth(step, mode=modes.EVAL)
   prediction = np.argmax(tensor)
   val_predictions.append(prediction)
   
   #get outputs from penultimate layer 
   for layer in trial.tensor_names(regex=".*avgpool_output"):
      tensor = trial.tensor(layer).worth(step, mode=modes.EVAL)])
      val_pen_layer[layer].append(tensor)
      
#entry tensors saved throughout inference
trial = create_trial("s3://xxx/data_capture/inference/")

#iterate over inference steps saved by Debugger
for step in trial.steps(mode=modes.PREDICT):
       
   #get mannequin outputs
   tensor = trial.tensor("ResNet_output_0").worth(step, mode=modes.PREDICT)
   prediction = np.argmax(tensor)
   inference_predictions.append(prediction)
    
   #get penultimate layer
   for layer in trial.tensor_names(regex=".*avgpool_output"):
      tensor = trial.tensor(layer).worth(step, mode=modes.PREDICT)])
      inference_pen_layer[layer].append(tensor)


#create LADIS object 
ladis = LADIS.LADIS(val_pen_layer, val_predictions, 
                    inference_pen_layer, inference_predictions)

#run MMD check
detection_rate = ladis.get_detection_rate(layers=[0], mix=True)

#decide how a lot every pattern contribute to the detection
for index in vary(len(query_latent['avgpool_output_0'])):
    
    stats.append(sample_selection.compute_ME_stat(val_latent['avgpool_output_0', 
                            inference_pen_layer['avgpool_output_0'],
                            inference_pen_layer['avgpool_output_0'][index]))

#discover prime 100 samples that had been probably the most impactful for detection
samples = sorted(stats)[:100]

Take a look at in opposition to adversarial inputs

Now that our customized Mannequin Monitor schedule has been deployed, we will produce some inference outcomes.

First, we run with knowledge from the holdout set after which with adversarial inputs:

test_dataset = datasets.CIFAR10('knowledge/cifar10', practice=False, obtain=True, remodel=None)

#run inference loop over holdout dataset
for index, (picture, label) in enumerate(zip(test_dataset.knowledge, test_dataset.targets)):

    #predict
    outcome = predictor.predict(picture)

We are able to then test the Mannequin Monitor show in Amazon SageMaker Studio or use Amazon CloudWatch logs to see if a problem was discovered.

Subsequent, we use the adversarial inputs in opposition to the mannequin hosted on SageMaker. We use the check dataset of the Tiny ImageNet dataset and apply the PGD assault, which introduces perturbations on the pixel degree such that the mannequin doesn’t acknowledge appropriate courses. Within the following photographs, the left column reveals two unique check photographs, the center column reveals their adversarially perturbed variations, and the suitable column reveals the distinction between each photographs.

Now we will test the Mannequin Monitor standing and see that a number of the inference photographs had been drawn from a special distribution.

Outcomes and person motion

The customized Mannequin Monitor job determines scores for every inference request, which signifies how doubtless the pattern is adversarial based on the MMD check. These scores are gathered for all inference requests. Their rating with the corresponding Debugger step quantity is recorded in a JSON file and uploaded to Amazon S3. After the Mannequin Monitoring job is full, we obtain the JSON file, retrieve step numbers, and use Debugger to retrieve the corresponding mannequin inputs for these steps. This permits us to examine the photographs that had been detected as adversarial.

The next code block plots the primary two photographs which were recognized because the almost certainly to be adversarial:

#entry inference knowledge
trial = create_trial(f"s3://{sagemaker_session.default_bucket()}/data_capture/inference")
steps = trial.steps(mode=modes.PREDICT)

#load constraint_violations.json file generated by customized ModelMonitor
outcomes = monitor.latest_monitoring_constraint_violations().body_dict)

for index in vary(2):
    # get outcomes: step and rating
    step = outcomes['violations'][index]['description']['Step']
    rating = spherical( outcomes['violations'][index]['description']['Score'],3)
    
    # get enter picture
    picture = trial.tensor('ResNet_input_0').worth(step, mode=modes.PREDICT)[0,:,:,:]
    
    # get predicted class
    predicted = np.argmax(trial.tensor('ResNet_output_0').worth(step, mode=modes.PREDICT))
    
    # visualize picture 
    plot_image(picture, predicted)

In our instance check run, we get the next output. The jellyfish picture was incorrectly predicted as an orange, and the camel picture as a panda. Clearly, the mannequin failed on these inputs and didn’t even predict the same picture class, equivalent to goldfish or horse. For comparability, we additionally present the corresponding pure samples from the check set on the suitable aspect. We are able to observe that the random perturbations launched by the attacker are very seen within the background of each photographs.

The customized Mannequin Monitor job publishes the detection charge to CloudWatch, so we will examine how this charge modified over time. A major change between two knowledge factors could point out that an adversary was making an attempt to idiot the mannequin at a particular timeframe. Moreover, you may as well plot the variety of inference requests being processed in every Mannequin Monitor job and the baseline detection charge, which is computed over the validation dataset. The baseline charge is normally near 0 and solely serves as a comparability metric.

The next screenshot reveals the metrics generated by our check runs, which ran three Mannequin Monitoring jobs over 3 hours. Every job processes roughly 200–300 inference requests at a time. The detection charge is 100% between 5:00 PM and 6:00 PM, and drops afterwards.

Moreover, we will additionally examine the distributions of representations generated by the intermediate layers of the mannequin. With Debugger, we will entry the information from the validation part of the coaching job and the tensors from the inference part, and use t-SNE to visualise their distribution for sure predicted courses. See the next code:

import seaborn as sns
from sklearn.manifold import TSNE


#compute TSNE embeddings
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)
embedding = tsne.fit_transform(np.concatenate((val_penultimate_layer, inference_penultimate_layer)))

# plot outcomes
sns.scatterplot(x=embedding[:,0], y= embedding[:,1], hue=labels, alpha=0.6, palette=sns.color_palette(None, len(np.distinctive(labels))), legend="full")
plt.determine(figsize=(10,5))

In our check case, we get the next t-SNE visualization for the second picture class. We are able to observe that the adversarial samples are clustered in another way than the pure ones.

Abstract

On this publish, we confirmed use a two-sample check utilizing most imply discrepancy to detect adversarial inputs. We demonstrated how one can deploy such detection mechanisms utilizing Debugger and Mannequin Monitor. This workflow lets you monitor your fashions hosted on SageMaker at scale and detect adversarial inputs robotically. To be taught extra about it, take a look at our GitHub repo.

References

[1] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. In the direction of deep studying fashions immune to adversarial assaults. In Worldwide Convention on Studying Representations, 2018.

[2] Laurens van der Maaten and Geoffrey Hinton. Visualizing knowledge utilizing t-SNE. Journal of Machine Studying Analysis, 9:2579–2605, 2008. URL http://www.jmlr.org/papers/v9/vandermaaten08a.html.

Concerning the Authors

Nathalie Rauschmayr is a Senior Utilized Scientist at AWS, the place she helps clients develop deep studying functions.

Yigitcan Kaya is a fifth 12 months PhD scholar at College of Maryland and an utilized scientist intern at AWS, engaged on safety of machine studying and functions of machine studying for safety.

Bilal Zafar is an Utilized Scientist at AWS, engaged on Equity, Explainability and Safety in Machine Studying.

Sergul Aydore is a Senior Utilized Scientist at AWS engaged on Privateness and Safety in Machine Studying