Minimize the production impact of ML model updates with Amazon SageMaker shadow testing

Amazon SageMaker now permits you to evaluate the efficiency of a brand new model of a mannequin serving stack with the presently deployed model previous to a full manufacturing rollout utilizing a deployment security observe referred to as shadow testing. Shadow testing might help you determine potential configuration errors and efficiency points earlier than they affect end-users. With SageMaker, you don’t must put money into constructing your shadow testing infrastructure, permitting you to give attention to mannequin growth. SageMaker takes care of deploying the brand new model alongside the present model serving manufacturing requests, routing a portion of requests to the shadow model. You may then evaluate the efficiency of the 2 variations utilizing metrics akin to latency and error charge. This offers you larger confidence that manufacturing rollouts to SageMaker inference endpoints gained’t trigger efficiency regressions, and helps you keep away from outages because of unintentional misconfigurations.

On this publish, we display this new SageMaker functionality. The corresponding pattern pocket book is offered on this GitHub repository.

Overview of answer

Your mannequin serving infrastructure consists of the machine studying (ML) mannequin, the serving container, or the compute occasion. Let’s take into account the next eventualities:

• You’re contemplating selling a brand new mannequin that has been validated offline to manufacturing, however wish to consider operational efficiency metrics, akin to latency, error charge, and so forth, earlier than making this choice.
• You’re contemplating modifications to your serving infrastructure container, akin to patching vulnerabilities or upgrading to newer variations, and wish to assess the affect of those modifications previous to promotion to manufacturing.
• You’re contemplating altering your ML occasion and wish to consider how the brand new occasion would carry out with reside inference requests.

The next diagram illustrates our answer structure.

For every of those eventualities, choose a manufacturing variant you wish to take a look at towards and SageMaker routinely deploys the brand new variant in shadow mode and routes a replica of the inference requests to it in actual time throughout the identical endpoint. Solely the responses of the manufacturing variant are returned to the calling utility. You may select to discard or log the responses of the shadow variant for offline comparability. Optionally, you’ll be able to monitor the variants via a built-in dashboard with a side-by-side comparability of the efficiency metrics. You should utilize this functionality both via SageMaker inference update-endpoint APIs or via the SageMaker console.

Shadow variants construct on prime of the manufacturing variant functionality in SageMaker inference endpoints. To reiterate, a manufacturing variant consists of the ML mannequin, serving container, and ML occasion. As a result of every variant is impartial of others, you’ll be able to have totally different fashions, containers, or occasion sorts throughout variants. SageMaker permits you to specify auto scaling insurance policies on a per-variant foundation to allow them to scale independently primarily based on incoming load. SageMaker helps as much as 10 manufacturing variants per endpoint. You may both configure a variant to obtain a portion of the incoming site visitors by setting variant weights, or specify the goal variant within the incoming request. The response from the manufacturing variant is forwarded again to the invoker.

A shadow variant(new) has the identical elements as a manufacturing variant. A user-specified portion of the requests, referred to as the site visitors sampling proportion, is forwarded to the shadow variant. You may select to log the response of the shadow variant in Amazon Easy Storage Service (Amazon S3) or discard it.

Word that SageMaker helps a most of 1 shadow variant per endpoint. For an endpoint with a shadow variant, there could be a most of 1 manufacturing variant.

After you arrange the manufacturing and shadow variants, you’ll be able to monitor the invocation metrics for each manufacturing and shadow variants in Amazon CloudWatch below the AWS/SageMaker namespace. All updates to the SageMaker endpoint are orchestrated utilizing blue/inexperienced deployments and happen with none loss in availability. Your endpoints will proceed responding to manufacturing requests as you add, modify, or take away shadow variants.

You should utilize this functionality in certainly one of two methods:

• Managed shadow testing utilizing the SageMaker Console – You may leverage the console for a guided expertise to handle the end-to-end journey of shadow testing. This allows you to setup shadow checks for a predefined period of time, monitor the progress via a reside dashboard, clear up upon completion, and act on the outcomes.
• Self-service shadow testing utilizing the SageMaker Inference APIs – In case your deployment workflow already makes use of create/replace/delete-endpoint APIs, you’ll be able to proceed utilizing them to handle Shadow Variants.

Within the following sections, we stroll via every of those eventualities.

State of affairs 1 – Managed shadow testing utilizing the SageMaker Console

In the event you want to select SageMaker to handle the end-to-end workflow of making, managing, and performing on the outcomes of the shadow checks, think about using the Shadow checks’ functionality within the Inference part of the SageMaker Console. As acknowledged earlier, this allows you to setup shadow checks for a predefined period of time, monitor the progress via a reside dashboard, presents clear up choices upon completion, and act on the outcomes. To study extra, please go to the shadow checks part of our documentation for a step-by-step walkthrough of this functionality.

Pre-requisites

The fashions for manufacturing and shadow have to be created on SageMaker. Please seek advice from the CreateModel API right here.

Step 1 – Create a shadow take a look at

Navigate to the Inference part of the left navigation panel of the SageMaker console after which select Shadow checks. This may take you to a dashboard with all of the scheduled, working, and accomplished shadow checks. Click on ‘create a shadow take a look at’. Enter a reputation for the take a look at and select subsequent.

This may take you to the shadow take a look at settings web page. You may select an present IAM function or create one which has the AmazonSageMakerFullAccess IAM coverage hooked up. Subsequent, select ‘Create a brand new endpoint’ and enter a reputation (xgb-prod-shadow-1). You may add one manufacturing and one shadow variant related to this endpoint by clicking on ‘Add’ within the Variants part. You may choose the fashions you’ve created within the ‘Add Mannequin’ dialog field. This creates a manufacturing or variant. Optionally, you’ll be able to change the occasion sort and rely related to every variant.

All of the site visitors goes to the manufacturing variant andit responds to invocation requests. You may management a portion of the requests that’s routed to the shadow variant by altering the Visitors Sampling Proportion.

You may management the period of the take a look at from one hour to 30 days. If unspecified, it defaults to 7 days. After this era, the take a look at is marked full. If you’re working a take a look at on an present endpoint, it will likely be rolled again to the state previous to beginning the take a look at upon completion.

You may optionally seize the requests and responses of the Shadow variant utilizing the Knowledge Seize choices. If left unspecified, the responses of the shadow variant are discarded.

Step 2 – Monitor a shadow take a look at

You may view the listing of shadow checks by navigating to the Shadow Exams part below Inference. Click on on the shadow take a look at created within the earlier step to view the small print of a shadow take a look at and monitor it whereas it’s in progress or after it has accomplished.

The Metrics part gives a comparability of the important thing metrics and gives overlaid graphs between the manufacturing and shadow variants, together with descriptive statistics. You may evaluate invocation metrics akin to ModelLatency and Invocation4xxErrors in addition to occasion metrics akin to CPUUtilization and DiskUtilization.

Step 3 – Promote the Shadow variant to the brand new manufacturing variant

Upon evaluating, you’ll be able to both select to advertise the shadow variant to be the brand new manufacturing variant or take away the shadow variant. For each these choices, choose ‘Mark Full’ on the highest of the web page. This presents you with an choice to both promote or take away the shadow variant.

In the event you select to advertise, you can be taken to a deployment web page, the place you’ll be able to affirm the variant settings previous to deployment. Previous to deployment, we suggest sizing your shadow variants to have the ability to deal with 100% of the invocation site visitors. If you’re not utilizing shadow testing to guage alternate occasion sorts or sizes, you should use the select the ‘retain manufacturing variant settings. In any other case, you’ll be able to select to ‘retain shadow variant settings. In the event you select this selection, please be sure that your site visitors sampling is ready at 100%. Alternatively, you’ll be able to specify the occasion sort and rely when you want to override these settings.

When you affirm the deployment, SageMaker will provoke an replace to your endpoint to advertise the shadow variant to the brand new manufacturing variant. As with SageMaker all updates, your endpoint will nonetheless be operational through the replace.

State of affairs 2: Shadow testing utilizing SageMaker inference APIs

This part covers the right way to use the prevailing SageMaker create/replace/delete-endpoint APIs to deploy shadow variants.

For this instance, we’ve two XGBoost fashions that symbolize two totally different variations of the fashions which were pre-trained. mannequin.tar.gz is the mannequin presently deployed in manufacturing. model2 is the newer mannequin, and we wish to take a look at its efficiency when it comes to operational metrics akin to latency earlier than deciding to make use of it in manufacturing. We deploy model2 as a shadow variant of mannequin.tar.gz. Each pre-trained fashions are saved within the public S3 bucket s3://sagemaker-sample-files. We firstdownload the modelour native compute occasion after which add to S3.

The fashions on this instance are used to foretell the likelihood of a cellular buyer leaving their present cellular operator. The dataset we use is publicly obtainable and was talked about within the e-book Discovering Knowledge in Data by Daniel T. Larose. These fashions had been skilled utilizing the XGB Churn Prediction Notebook in SageMaker. You can even use your personal pre-trained fashions, through which case you’ll be able to skip downloading from s3://sagemaker-sample-files and duplicate your personal fashions on to mannequin/ folder.

!aws s3 cp s3://sagemaker-sample-files/fashions/xgb-churn/xgb-churn-prediction-model.tar.gz mannequin/
!aws s3 cp s3://sagemaker-sample-files/fashions/xgb-churn/xgb-churn-prediction-model2.tar.gz mannequin/

Step 1 – Create fashions

We add the mannequin information to our personal S3 bucket and create two SageMaker fashions. See the next code:

model_url = S3Uploader.add(
    local_path="mannequin/xgb-churn-prediction-model.tar.gz",
    desired_s3_uri=f"s3://{bucket}/{prefix}",
)
model_url2 = S3Uploader.add(
    local_path="mannequin/xgb-churn-prediction-model2.tar.gz",
    desired_s3_uri=f"s3://{bucket}/{prefix}",
from sagemaker import image_uris
image_uri = image_uris.retrieve("xgboost", boto3.Session().region_name, "0.90-1")
image_uri2 = image_uris.retrieve("xgboost", boto3.Session().region_name, "0.90-2")

model_name = f"DEMO-xgb-churn-pred-{datetime.now():%Y-%m-%d-%H-%M-%S}"
model_name2 = f"DEMO-xgb-churn-pred2-{datetime.now():%Y-%m-%d-%H-%M-%S}"

resp = sm.create_model(
    ModelName=model_name,
    ExecutionRoleArn=function,
    Containers=[{"Image": image_uri, "ModelDataUrl": model_url}],
)

resp = sm.create_model(
    ModelName=model_name2,
    ExecutionRoleArn=function,
    Containers=[{"Image": image_uri2, "ModelDataUrl": model_url2}],
)

Step 2 – Deploy the 2 fashions as manufacturing and shadow variants to a real-time inference endpoint

We create an endpoint config with the manufacturing and shadow variants. The ProductionVariants and ShadowProductionVariants are of specific curiosity. Each these variants have ml.m5.xlarge cases with 4 vCPUs and 16 GiB of reminiscence, and the preliminary occasion rely is ready to 1. See the next code:

ep_config_name = f"Shadow-EpConfig-{datetime.now():%Y-%m-%d-%H-%M-%S}"
production_variant_name = "manufacturing"
shadow_variant_name = "shadow"
create_endpoint_config_response = sm.create_endpoint_config(
    EndpointConfigName=ep_config_name,
    ProductionVariants=[
    # Type: Array of ProductionVariant (https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html) objects
      { 
         "VariantName": shadow_variant_name,
        {
            "VariantName": production_variant_name,
            "ModelName": model_name,
            "InstanceType": "ml.m5.xlarge",
            "InitialInstanceCount": 2,
            "InitialVariantWeight": 1,
        }
    ],
     # Kind: Array of ShadowProductionVariants 
    ShadowProductionVariants = [
         "ModelName": model_name2,
         "InitialInstanceCount": 1,
         "InitialVariantWeight": 0.5,
         "InstanceType": "ml.m5.xlarge" 
      }
   ]
)

Lastly, we create the manufacturing and shadow variant:

endpoint_name = f"xgb-prod-shadow-{datetime.now():%Y-%m-%d-%H-%M-%S}"
create_endpoint_api_response = sm.create_endpoint(
                                    EndpointName=endpoint_name,
                                    EndpointConfigName=ep_config_name,
                                )

Step 3 – Invoke the endpoint for testing

After the endpoint has been efficiently created, you’ll be able to start invoking it. We ship about 3,000 requests in a sequential method:

def invoke_endpoint(endpoint_name, wait_interval_sec=0.01, should_raise_exp=False):
    with open("test_data/test-dataset-input-cols.csv", "r") as f:
        for row in f:
            payload = row.rstrip("n")
            strive:
                for i in vary(10): #ship the identical payload 10 instances for testing function
                    response = sm_runtime.invoke_endpoint(
                        EndpointName=endpoint_name, ContentType="textual content/csv", Physique=payload
                    )
            besides Exception as e:
                print("E", finish="", flush=True)
                if should_raise_exp:
                    elevate e

invoke_endpoint(endpoint_name)

Step 4 – Evaluate metrics

Now that we’ve deployed each the manufacturing and shadow fashions, let’s evaluate the invocation metrics. For an inventory of invocation metrics obtainable for comparability, seek advice from Monitor Amazon SageMaker with Amazon CloudWatch. Let’s begin by evaluating invocations between the manufacturing and shadow variants.

The InvocationsPerInstance metric refers back to the variety of invocations despatched to the manufacturing variant. A fraction of those invocations, specified within the variant weight, are despatched to the shadow variant. The invocation per occasion is calculated by dividing the overall variety of invocations by the variety of cases in a variant. As proven within the following charts, we will affirm that each the manufacturing and shadow variants are receiving invocation requests in keeping with the weights specified within the endpoint config.

Subsequent, let’s evaluate the mannequin latency (ModelLatency metric) between the manufacturing and shadow variants. Mannequin latency is the time taken by a mannequin to reply as considered from SageMaker. We will observe how the mannequin latency of the shadow variant compares with the manufacturing variant with out exposing end-users to the shadow variant.

We anticipate the overhead latency (OverheadLatency metric) to be comparable throughout manufacturing and shadow variants. Overhead latency is the interval measured from the time SageMaker receives the request till it returns a response to the shopper, minus the mannequin latency.

Step 5- Promote your shadow variant

To advertise the shadow mannequin to manufacturing, create a brand new endpoint configuration with present ShadowProductionVariant as the brand new ProductionVariant and take away the ShadowProductionVariant. This may take away the present ProductionVariant and promote the shadow variant to grow to be the brand new manufacturing variant. As at all times, all SageMaker updates are orchestrated as blue/inexperienced deployments below the hood, and there’s no lack of availability whereas performing the replace.

Optionally, you’ll be able to leverage Deployment Guardrails if you wish to use all-at-once site visitors shifting and auto rollbacks throughout your replace.

promote_ep_config_name = f"PromoteShadow-EpConfig-{datetime.now():%Y-%m-%d-%H-%M-%S}"

create_endpoint_config_response = sm.create_endpoint_config(
    EndpointConfigName=promote_ep_config_name,
    ProductionVariants=[
        {
            "VariantName": shadow_variant_name,
            "ModelName": model_name2,
            "InstanceType": "ml.m5.xlarge",
            "InitialInstanceCount": 2,
            "InitialVariantWeight": 1.0,
        }
    ],
)
print(f"Created EndpointConfig: {create_endpoint_config_response['EndpointConfigArn']}")

update_endpoint_api_response = sm.update_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=promote_ep_config_name,
)

wait_for_endpoint_in_service(endpoint_name)

sm.describe_endpoint(EndpointName=endpoint_name)

Step 6 – Clear Up

If you don’t plan to make use of this endpoint additional, it is best to delete the endpoint to keep away from incurring further costs and clear up different sources created on this weblog.

dsm.delete_endpoint(EndpointName=endpoint_name)
sm.delete_endpoint_config(EndpointConfigName=ep_config_name)
sm.delete_endpoint_config(EndpointConfigName=promote_ep_config_name)
sm.delete_model(ModelName=model_name)
sm.delete_model(ModelName=model_name2)

Conclusion

On this publish, we launched a brand new functionality of SageMaker inference to check the efficiency of recent model of a mannequin serving stack with the presently deployed model previous to a full manufacturing rollout utilizing a deployment security observe referred to as shadow testing. We walked you thru the benefits of utilizing shadow variants and strategies to configure the variants with an end-to-end example. To study extra about shadow variants, seek advice from shadow checks documentation.

In regards to the Authors

Raghu Ramesha is a Machine Studying Options Architect with the Amazon SageMaker Service group. He focuses on serving to clients construct, deploy, and migrate ML manufacturing workloads to SageMaker at scale. He makes a speciality of machine studying, AI, and laptop imaginative and prescient domains, and holds a grasp’s diploma in Laptop Science from UT Dallas. In his spare time, he enjoys touring and pictures.

Qingwei Li is a Machine Studying Specialist at Amazon Internet Providers. He obtained his Ph.D. in Operations Analysis after he broke his advisor’s analysis grant account and didn’t ship the Nobel Prize he promised. Presently he helps clients within the monetary service and insurance coverage trade construct machine studying options on AWS. In his spare time, he likes studying and instructing.

Qiyun Zhao is a Senior Software program Growth Engineer with the Amazon SageMaker Inference Platform group. He’s the lead developer of the Deployment Guardrails and Shadow Deployments, and he focuses on serving to clients to handle ML workloads and deployments at scale with excessive availability. He additionally works on platform structure evolutions for quick and safe ML jobs deployment and working ML on-line experiments relaxed. In his spare time, he enjoys studying, gaming and touring.

Tarun Sairam is a Senior Product Supervisor for Amazon SageMaker Inference. He’s all for studying in regards to the newest traits in machine studying and serving to clients leverage them. In his spare time, he enjoys biking, snowboarding, and enjoying tennis.