Text summarization with Amazon SageMaker and Hugging Face

On this publish, we present you easy methods to implement some of the downloaded Hugging Face pre-trained fashions used for textual content summarization, DistilBART-CNN-12-6, inside a Jupyter pocket book utilizing Amazon SageMaker and the SageMaker Hugging Face Inference Toolkit. Primarily based on the steps proven on this publish, you possibly can strive summarizing textual content from the WikiText-2 dataset managed by fast.ai, obtainable on the Registry of Open Data on AWS.

World knowledge volumes are rising at zettabyte scale as corporations and customers broaden their use of digital merchandise and on-line companies. To higher perceive this rising knowledge, machine studying (ML) pure language processing (NLP) methods for textual content evaluation have developed to handle use circumstances involving textual content summarization, entity recognition, classification, translation, and extra. AWS affords pre-trained AWS AI companies that may be built-in into purposes utilizing API calls and require no ML expertise. For instance, Amazon Comprehend can carry out NLP duties reminiscent of customized entity recognition, sentiment evaluation, key phrase extraction, subject modeling, and extra to assemble insights from textual content. It may carry out textual content evaluation on all kinds of languages for its numerous options.

Textual content summarization is a useful approach in understanding giant quantities of textual content knowledge as a result of it creates a subset of contextually significant data from supply paperwork. You’ll be able to apply this NLP approach to longer-form textual content paperwork and articles, enabling faster consumption and simpler doc indexing, for instance to summarize name notes from conferences.

Hugging Face is a well-liked open-source library for NLP, with over 49,000 pre-trained fashions in additional than 185 languages with help for various frameworks. AWS and Hugging Face have a partnership that enables a seamless integration by way of SageMaker with a set of AWS Deep Studying Containers (DLCs) for coaching and inference in PyTorch or TensorFlow, and Hugging Face estimators and predictors for the SageMaker Python SDK. These capabilities in SageMaker assist builders and knowledge scientists get began with NLP on AWS extra simply. Processing texts with transformers in deep studying frameworks reminiscent of PyTorch is often a posh and time-consuming process for knowledge scientists, usually resulting in frustration and lack of effectivity when growing NLP initiatives. The rise of AI communities like Hugging Face, mixed with the ability of ML companies within the cloud like SageMaker, speed up and simplify the event of those textual content processing duties. SageMaker helps you construct, prepare, deploy, and operationalize Hugging Face fashions.

Textual content summarization overview

You’ll be able to apply textual content summarization to determine key sentences inside a doc or determine key sentences throughout a number of paperwork. Textual content summarization can produce two kinds of summaries: extractive and abstractive. Extractive summaries don’t include any machine-generated textual content and are a set of essential sentences chosen from the enter doc. Abstractive summaries include new human-readable phrases and sentences generated by the textual content summarization mannequin. Most textual content summarization programs are primarily based on extractive summarization as a result of correct abstractive textual content summarization is tough to realize.

Hugging Face has over 400 pre-trained state-of-the-art text summarization models available, implementing completely different mixtures of NLP methods. These fashions are skilled on completely different datasets, uploaded and maintained by know-how corporations and members of the Hugging Face neighborhood. You’ll be able to filter the fashions by most downloaded or most appreciated, and straight load them when utilizing the summarization pipeline Hugging Face transformer API. The Hugging Face transformer simplifies the NLP implementation course of in order that high-performance NLP fashions will be fine-tuned to ship textual content summaries, with out requiring intensive ML operation data.

Hugging Face textual content summarization fashions on AWS

SageMaker affords enterprise analysts, knowledge scientists, and MLOps engineers a alternative of instruments to design and function ML workloads on AWS. These instruments give you quicker implementation and testing of ML fashions to realize your optimum outcomes.

From the SageMaker Hugging Face Inference Toolkit, an open-source library, we define three other ways to implement and host Hugging Face textual content summarization fashions utilizing a Jupyter pocket book:

• Hugging Face summarization pipeline – Create a Hugging Face summarization pipeline utilizing the “summarization” process identifier to make use of a default textual content summarization mannequin for inference inside your Jupyter pocket book. These pipelines summary away the complicated code, providing novice ML practitioners a easy API to rapidly implement textual content summarization with out configuring an inference endpoint. The pipeline additionally permits the ML practitioner to pick out a selected pre-trained mannequin and its related tokenizer. Tokenizers put together textual content to be prepared as an enter for the mannequin by splitting textual content into phrases or subwords, which then are transformed to IDs by way of a lookup desk. For simplicity, the next code snippet gives for the default case when utilizing pipelines. The DistilBART-CNN-12-6 mannequin is without doubt one of the most downloaded summarization fashions on Hugging Face and is the default model for the summarization pipeline. The final line calls the pre-trained mannequin to get a abstract for the handed textual content given the supplied two arguments.

  from transformers import pipeline
  
  summarizer = pipeline("summarization")
  summarizer("An apple a day, retains the physician away", min_length=5, max_length=20)

• SageMaker endpoint with pre-trained mannequin – Create a SageMaker endpoint with a pre-trained mannequin from the Hugging Face Model Hub and deploy it on an inference endpoint, such because the ml.m5.xlarge occasion within the following code snippet. This methodology permits skilled ML practitioners to rapidly choose particular open-source fashions, fine-tune them, and deploy the fashions onto high-performing inference situations.

  from sagemaker.huggingface import HuggingFaceModel
  from sagemaker import get_execution_role
  
  function = get_execution_role()
  
  # Hub Mannequin configuration. https://huggingface.co/fashions
  hub = {
    'HF_MODEL_ID':'sshleifer/distilbart-cnn-12-6',
    'HF_TASK':'summarization'
  }
  
  # create Hugging Face Mannequin Class
  huggingface_model = HuggingFaceModel(
      transformers_version='4.17.0',
      pytorch_version='1.10.2',
      py_version='py38',
      env=hub,
      function=function,
  )
  
  # deploy mannequin to SageMaker Inference
  predictor = huggingface_model.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")

• SageMaker endpoint with a skilled mannequin – Create a SageMaker mannequin endpoint with a skilled mannequin saved in an Amazon Easy Storage Service (Amazon S3) bucket and deploy it on an inference endpoint. This methodology permits skilled ML practitioners to rapidly deploy their very own fashions saved on Amazon S3 onto high-performing inference situations. The mannequin itself is downloaded from Hugging Face and compressed, after which will be uploaded to Amazon S3. This step is demonstrated within the following code snippet:

  from sagemaker.huggingface import HuggingFaceModel
  from sagemaker import get_execution_role
  
  function = get_execution_role()
  
  # create Hugging Face Mannequin Class
  huggingface_model = HuggingFaceModel(
      transformers_version='4.17.0',
      pytorch_version='1.0.2',
      py_version='py38',
      model_data="s3://my-trained-model/artifacts/mannequin.tar.gz",
      function=function,
  )
  
  # deploy mannequin to SageMaker Inference
  predictor = huggingface_model.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")

AWS has a number of sources obtainable to help you in deploying your ML workloads. The Machine Studying Lens of the AWS Properly Architected Framework recommends ML workloads greatest practices, together with optimizing sources and lowering value. These advisable design ideas be certain that nicely architected ML workloads on AWS are deployed to manufacturing. Amazon SageMaker Inference Recommender helps you choose the appropriate occasion to deploy your ML fashions at optimum inference efficiency and value. Inference Recommender hastens mannequin deployment and reduces time to market by automating load testing and optimizing mannequin efficiency throughout ML situations.

Within the subsequent sections, we reveal easy methods to load a skilled mannequin from an S3 bucket and deploy it to an acceptable inference occasion.

Conditions

For this walkthrough, you need to have the next stipulations:

Load the Hugging Face mannequin to SageMaker for textual content summarization inference

Use the next code to obtain the Hugging Face pre-trained textual content summarization mannequin DistilBART-CNN-12-6 and its tokenizer, and save them domestically in SageMaker to your Jupyter pocket book listing:

from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

PRE_TRAINED_MODEL_NAME='sshleifer/distilbart-cnn-12-6'

mannequin = BartForConditionalGeneration.from_pretrained(PRE_TRAINED_MODEL_NAME, cache_dir=hf_cache_dir)
mannequin.save_pretrained('./fashions/bart_model/')

tokenizer = BartTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME)
tokenizer.save_pretrained('./fashions/bart_tokenizer/')

Compress the saved textual content summarization mannequin and its tokenizer into tar.gz format and add the compressed mannequin artifact to an S3 bucket:

! tar -C fashions/ -czf mannequin.tar.gz code/ bart_tokenizer/ bart_model/
from sagemaker.s3 import S3Uploader

file_key = 'mannequin.tar.gz'
model_artifact = S3Uploader.add(file_key,'s3://my-trained-model/artifacts')

Choose an inference Docker container image to carry out the textual content summarization inference. Outline the Linux OS, PyTorch framework, and Hugging Face Transformer model and specify the Amazon Elastic Compute Cloud (Amazon EC2) occasion sort to run the container.

The Docker picture is out there within the Amazon Elastic Container Registry (Amazon ECR) of the identical AWS account, and the hyperlink for that container picture is returned as a URI.

from sagemaker.image_uris import retrieve

deploy_instance_type="ml.m5.xlarge"

pytorch_inference_image_uri = retrieve('huggingface',
                                       area=area,
                                       model='4.6.1',
                                       instance_type=deploy_instance_type,
                                       base_framework_version='pytorch1.8.1',
                                       image_scope="inference")

Outline the textual content summarization mannequin to be deployed by the chosen container picture performing inference. Within the following code snippet, the compressed mannequin uploaded to Amazon S3 is deployed:

from sagemaker.huggingface.mannequin import HuggingFaceModel
from sagemaker import get_execution_role

function = get_execution_role()

# create Hugging Face Mannequin Class
huggingface_model = HuggingFaceModel(
   model_data="s3://my-trained-model/artifacts/mannequin.tar.gz", # path to your skilled sagemaker mannequin
   image_uri=pytorch_inference_image_uri,
   function=function, # iam function with permissions to create an Endpoint
   transformers_version="4.6.1", # transformers model used
)

# deploy mannequin to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1, 
   instance_type="ml.m5.xlarge"
)

Check the deployed textual content summarization mannequin on a pattern enter:

# instance request, it is advisable outline "inputs"
knowledge = {
   "textual content": "Digital camera - You might be awarded a SiPix Digital Digital camera! name 09061221066 fromm landline. Supply inside 28 days."
}

# request
predictor.predict(knowledge)

Use Inference Recommender to guage the optimum EC2 occasion for the inference process

Subsequent, create a number of payload samples of enter textual content in JSON format and compress them right into a single payload file. These payload samples are utilized by the Inference Recommender to match inference efficiency between completely different EC2 occasion varieties. Every of the pattern payloads should match the JSON format proven earlier. You may get examples from the WikiText-2 dataset managed by fast.ai, obtainable on the Registry of Open Data on AWS.

Add the compressed textual content summarization mannequin artifact and the compressed pattern payload file to the S3 bucket. We uploaded the mannequin in an earlier step, however for readability we embody the code to add it once more:

bucket = sagemaker.Session().default_bucket()

prefix = "sagemaker/inference-recommender"

model_archive_name = "mannequin.tar.gz"
payload_archive_name = "payload.tar.gz"

sample_payload_url = sagemaker.Session().upload_data(
    payload_archive_name, bucket=bucket, key_prefix=prefix + "/inference"
)
model_url = sagemaker.Session().upload_data(
    model_archive_name, bucket=bucket, key_prefix=prefix + "/mannequin"
)

Evaluation the checklist of ordinary ML fashions obtainable on SageMaker throughout widespread mannequin zoos, reminiscent of NLP and pc imaginative and prescient. Choose an NLP mannequin to carry out the textual content summarization inference:

import boto3
import pandas as pd

inference_client = boto3.shopper("sagemaker", area)

list_model_metadata_response = inference_client.list_model_metadata()

domains = []
frameworks = []
framework_versions = []
duties = []
fashions = []

for model_summary in list_model_metadata_response["ModelMetadataSummaries"]:
    domains.append(model_summary["Domain"])
    duties.append(model_summary["Task"])
    fashions.append(model_summary["Model"])
    frameworks.append(model_summary["Framework"])
    framework_versions.append(model_summary["FrameworkVersion"])

knowledge = {
    "Area": domains,
    "Process": duties,
    "Framework": frameworks,
    "FrameworkVersion": framework_versions,
    "Mannequin": fashions,
}

df = pd.DataFrame(knowledge)

pd.set_option("show.max_rows", None)
pd.set_option("show.max_columns", None)
pd.set_option("show.width", 1000)
pd.set_option("show.colheader_justify", "heart")
pd.set_option("show.precision", 3)

show(df.sort_values(by=["Domain", "Task", "Framework", "FrameworkVersion"]))

The next instance makes use of the bert-base-cased NLP mannequin. Register the textual content summarization mannequin into the SageMaker mannequin registry with the appropriately recognized area, framework, and process from the earlier step. The parameters for this instance are proven initially of the next code snippet.

Observe the vary of EC2 occasion varieties to be evaluated by Inference Recommender beneath SupportedRealtimeInferenceInstanceTypes within the following code. Be sure that the service limits for the AWS account enable the deployment of a lot of these inference nodes.

ml_domain = "NATURAL_LANGUAGE_PROCESSING"
ml_task = "FILL_MASK"
model_name = "bert-base-cased"
dlc_uri = pytorch_inference_image_uri
framework = 'PYTORCH'
framework_version='1.6.0'

inference_client = boto3.shopper("sagemaker", area)

model_package_group_name = uuid.uuid1()

model_pacakge_group_response = inference_client.create_model_package_group(
    ModelPackageGroupName=str(model_package_group_name), ModelPackageGroupDescription="description"
)

model_package_version_response = inference_client.create_model_package(
    ModelPackageGroupName=str(model_package_group_name),
    ModelPackageDescription="InferenceRecommenderDemo",
    Area=ml_domain,
    Process=ml_task,
    SamplePayloadUrl=sample_payload_url,
    InferenceSpecification={
        "Containers": [
            {
                "ContainerHostname": "huggingface-pytorch",
                "Image": dlc_uri,
                "ModelDataUrl": model_url,
                "Framework": framework,
                "FrameworkVersion": framework_version,
                "NearestModelName": model_name,
                "Environment": {
                    "SAGEMAKER_CONTAINER_LOG_LEVEL": "20",
                    "SAGEMAKER_PROGRAM": "inference.py",
                    "SAGEMAKER_REGION": region,
                    "SAGEMAKER_SUBMIT_DIRECTORY": model_url,
                },
            },
        ],
        "SupportedRealtimeInferenceInstanceTypes": [
            "ml.t2.xlarge",
            "ml.c5.xlarge",
            "ml.m5.xlarge",
            "ml.m5d.xlarge",
            "ml.r5.xlarge",
            "ml.inf1.xlarge",
        ],
        "SupportedContentTypes": [
            "application/json",
        ],
        "SupportedResponseMIMETypes": ["application/json"],
    },
)

Create an Inference Recommender default job utilizing the ModelPackageVersion ensuing from the earlier step. The uuid Python library is used to generate a novel identify for the job.

from sagemaker import get_execution_role

shopper = boto3.shopper("sagemaker", area)

function = get_execution_role()
default_job = uuid.uuid1()
default_response = shopper.create_inference_recommendations_job(
    JobName=str(default_job),
    JobDescription="Job Description",
    JobType="Default",
    RoleArn=function,
    InputConfig={"ModelPackageVersionArn": model_package_version_response["ModelPackageArn"]},
)

You may get the standing of the Inference Recommender job by working the next code:

inference_recommender_job = shopper.describe_inference_recommendations_job(
        JobName=str(default_job)
)

When the job standing is COMPLETED, evaluate the inference latency, runtime, and different metrics of the EC2 occasion varieties evaluated by the Inference Recommender default job. Choose the acceptable node sort primarily based in your use case necessities.

knowledge = [
    {**x["EndpointConfiguration"], **x["ModelConfiguration"], **x["Metrics"]}
    for x in inference_recommender_job["InferenceRecommendations"]
]
df = pd.DataFrame(knowledge)
df.drop("VariantName", inplace=True, axis=1)
pd.set_option("max_colwidth", 400)
df.head()

Conclusion

SageMaker affords a number of methods to make use of Hugging Face fashions; for extra examples, try the AWS Samples GitHub. Relying on the complexity of the use case and the necessity to fine-tune the mannequin, you possibly can choose the optimum manner to make use of these fashions. The Hugging Face pipelines could be a good start line to rapidly experiment and choose appropriate fashions. When it is advisable customise and parameterize the chosen fashions, you possibly can obtain the fashions and deploy them to personalised inference endpoints. To fine-tune the mannequin extra for a selected use case, you’ll want to coach the mannequin after downloading it.

NLP fashions typically, together with textual content summarization fashions, carry out higher after being skilled on a dataset that’s particular for the use case. The MLOPs and mannequin monitoring options of SageMaker make it possible for the deployed mannequin continues to carry out inside expectations. On this publish, we used Inference Recommender to guage one of the best suited occasion sort to deploy the textual content summarization mannequin. These suggestions can optimize efficiency and value in your ML use case.

In regards to the Authors

Dr. Nidal AlBeiruti is a Senior Options Architect at Amazon Internet Providers, with a ardour for machine studying options. Nidal has over 25 years of expertise working in quite a lot of international IT roles at completely different ranges and verticals. Nidal acts as a trusted advisor for a lot of AWS clients to help and speed up their cloud adoption journey.

Darren Ko is a Options Architect primarily based in London. He advises UK and Eire SMB clients on rearchitecting and innovating on the cloud. Darren is excited about purposes constructed with serverless architectures and he’s enthusiastic about fixing sustainability challenges with machine studying.