AI/ML-driven actionable insights and themes for Amazon third-party sellers using AWS

The Amazon Worldwide Vendor Progress (ISG) crew runs the CSBA (Buyer Service by Amazon) program that helps over 200,000 third-party Service provider Fulfilled Community (MFN) sellers. Amazon name facilities facilitate lots of of hundreds of cellphone calls, chats, and emails going between the customers and Amazon MFN sellers. The big quantity of contacts creates a problem for CSBA to extract key info from the transcripts that helps sellers promptly deal with buyer wants and enhance buyer expertise. Due to this fact, it’s crucial to routinely uncover insights from these transcripts, carry out theme detection to investigate a number of buyer conversations, and routinely current a set of themes that point out the highest causes for buyer contact, in order that the client issues are addressed in the proper method and as quickly as potential.

This submit presents an answer that makes use of a workflow and AWS AI and machine studying (ML) companies to supply actionable insights primarily based on these transcripts. We use a number of AWS AI/ML companies, reminiscent of Contact Lens for Amazon Join and Amazon SageMaker, and make the most of a mixed structure. This resolution is examined with ISG utilizing a small quantity of knowledge samples. On this submit, we talk about the thought course of, constructing this resolution, and the end result from the take a look at. We imagine the teachings discovered and our journey offered right here could allow you to by yourself journey.

Operational panorama and enterprise workflow

The next determine reveals the really helpful operational panorama with stakeholders and enterprise workflow for ISG in order that sellers can keep near their prospects anytime, wherever. The patron contacts Buyer Service by means of a contact middle platform and engages with the Buyer Service Affiliate (CSA). Then the transcripts of contacts turn into out there to CSBA to extract actionable insights by means of tens of millions of buyer contacts for the sellers, and the info is saved within the Vendor Information Lake. Sellers use the Amazon Vendor Central portal to entry the analytics outcomes and take motion to rapidly and successfully deal with buyer issues.

Answer overview

The next diagram reveals the structure reflecting the workflow operations into AI/ML and ETL (extract, remodel, and cargo) companies.

The workflow steps are as follows:

1. We use Amazon Join as a cloud contact middle for consumer-CSA interactions. Contact Lens for Amazon Join generates name and chat transcripts; derives contact abstract, analytics, categorization of associate-customer interplay, and difficulty detection; and measures buyer sentiments.
2. Contact Lens then shops analytics information into an Amazon Easy Storage Service (Amazon S3) bucket for long-term retention.
3. Amazon Kinesis Information Streams collects and transfers the high-throughput analytics information, processed by AWS Lambda, and injects and shops the info into an intermediate S3 bucket. At this stage, the info accommodates name and chat transcripts, sentiment scores, detected points, and classes.
4. It triggers the Lambda capabilities to ingest the info stream, extract the requested information fields, and set off inference of customized ML analyses by AWS AI/ML companies, on high of Contact Lens outcomes.On this evaluation, Contact Lens supplies correct sentiment scores measuring buyer satisfaction on consumer-CSA interactions. Contact Lens guidelines assist us categorize identified points within the contact middle. At this stage, ISG needed to supply extra insights to the vendor by detecting the theme by means of discovering beforehand unknown points in seller-specific calls, carried out resolutions, and particular key phrases. Right here, a non-deep studying mannequin was skilled and run on SageMaker, the main points of which will probably be defined within the following part.
5. After the AI/ML-based analytics, all actionable insights are generated after which saved within the Vendor Information Lake. The insights are shared on the Vendor Central Portal for the worldwide sellers to pinpoint the basis trigger and take immediate motion.

Within the following sections, we dive deeper into the AI/ML resolution and its parts.

Information labeling

On this part, we describe our strategy for information labeling to determine the contact motive and determination, and our methodology for key phrases extraction for the sellers to carry out root trigger evaluation.

Contact motive and determination labeling

To detect the contact motive from transcripts by ML, we utilized seven Standardized Subject Codes (SICs) as the info labels from the pattern information supplied by ISG crew:

• Contacted vendor to request cancelation
• Monitoring reveals delivered however cargo not acquired
• Cargo undeliverable
• Cargo not delivered previous supply date
• Cargo in transit to buyer
• Request Return Mailing Label (RML)
• Merchandise non-returnable

The contact motive labels might be additional prolonged by including the beforehand unknown points to the vendor; nonetheless, these points had not been outlined within the SIC. Not like the contact motive, the contact decision doesn’t have a label related to the transcripts. The decision classes had been specified by the ISG crew, and the resolutions wanted to be labeled primarily based on these classes. Due to this fact, we utilized Amazon SageMaker Floor Reality to create or replace labels for every contact.

Floor Reality supplies a knowledge labeling service that makes it straightforward to label information, and provides you the choice to make use of human annotators by means of Amazon Mechanical Turk, third-party distributors, or your personal personal workforce. For this resolution, the ISG crew outlined for classes for contact decision in over 140 transcript paperwork, which had been labeled by Amazon Mechanical Turk contractors:

• Full refund – 69 data
• Contact vendor – 52 data
• Partial refund – 15 data
• Different – 4 data

It solely took a few hours for the contractors to finish the multi-label textual content classification contact middle decision labeling for the 140 paperwork, and have them reviewed by the client. Within the subsequent step, we construct the multi-class classification fashions, then predict the contact motive and determination from the brand new name and chat transcripts coming from the customer support.

Key phrases for the basis trigger evaluation

One other problem is to extract the key phrases from the transcripts that may information the MFN sellers on particular actions. For this instance, the vendor must seize the important thing info reminiscent of product info, crucial timeline, drawback particulars, and refund provided by the CSA, which will not be clear. Right here we constructed a customized key phrases extraction mannequin in SageMaker utilizing the RAKE (Speedy Automated Key phrase Extraction) algorithm, following the method proven within the following determine. RAKE is a domain-independent key phrase extraction algorithm that determines key phrases by analyzing the frequency of phrase look and its co-occurrence with different phrases within the textual content.

After the usual doc preprocessing, RAKE detects probably the most related key phrases and phrases from the transcript paperwork. The output is listed as follows:

[('im amazons chat helper .. im', 0.08224299065420558),

('jun 23 .. could', 0.041588785046728964), <== timeline

('original payment method please', 0.04112149532710279), <== resolution: refund

('amazon gift card balance', 0.04112149532710279), <== resolution: refund

('previous conversation .. let', 0.04018691588785046),

('faulty pieces would like', 0.036448598130841114), <== call reason: faulty piece

('nice day !:)..', 0.025233644859813078),

('dual fuel gas', 0.025233644859813078), <== call reason: product info

('customer service hub', 0.025233644859813078),

('5 business days', 0.025233644859813078), <== timeline

('try .. got', 0.02476635514018691),

('right place ..', 0.023364485981308407),

('item .. let', 0.023364485981308407),

('youd like help', 0.02242990654205607),

('think would help', 0.02242990654205607),

('search help pages', 0.02242990654205607),

('gbc1793w ). order', 0.02242990654205607), <== call reason: product info

('moment .. ok', 0.021962616822429903),

('charcoal combo grill', 0.021028037383177565), <== call reason: product info

('call back ..', 0.021028037383177565),

('yes please anything', 0.020093457943925228),

('chat ..', 0.014953271028037382),

('would love', 0.014018691588785043),

('looks like', 0.014018691588785043),

('bent pieces', 0.013084112149532708), <== call reason: faulty details

This method captured key phrases with high relevance scores on the critical information such as timeline (“June 23”), refund resolution (“Amazon gift card,” “in 5 business days”), product information (“charcoal combo grill,” “dual fuel gas,” “gbc1793w”) and problem details (“faulty piece,” “bent pieces”). These insights not only tell the seller that this customer has been taken care of by getting a refund, but also guide the seller to further investigate the gas grill product defect and avoid having similar issues for other customers.

Text classification model training

Contact Lens generated transcripts, contact summary, and sentiments for call and chat samples collected from ISG Customer Service. Throughout the testing, the transcription and sentiment scores were accurate as expected. Along with known issues, the ISG team also looks for detecting unknown issues from transcripts to meet the seller-specific needs such as delivery problems, product defects, the resolutions provided by the contact, and issues or key phrases leading to a return or refund.

To address this challenge, we extended our tests through custom models on SageMaker. Our experience pointed to “bag-of-words” based, more conventional (non-deep learning) models using SageMaker based on the size of the dataset and samples.

We performed the contact reason classification modeling following the three steps on SageMaker as shown in the following figure.

The steps are as follows:

1. Preprocessing – We used the NLTK library to lower the cases; remove punctuation, tags, markups, and white space trailing; and filter single letters, numeric values, and customized lists of stop words.
2. Vectorization – We used the TF-IDF (Term Frequency-Inverse Document Frequency) method to convert the processed document into a matrix of TF-IDF features. The method quantifies the importance and relevance of words and phrases in a document with a collection of documents (corpus), then generates the features in numeric values to represent how important a word is to the document in the corpus. For this solution, we tested with specifying 750 and 1,500 features.
3. Multi-class classification – We generated a seven-class classification model using a vectorized feature list and SIC labels. We utilized 90% of the samples for training and 10% for validation.

We tested three algorithms aiming to obtain the best-performing model:

• First, we used the SageMaker Linear Learner algorithm with default hyperparameters and performed 10 epochs, and reached 71% accuracy for the testing set.
• Next, we used the SageMaker built-in XGBoost algorithm, and ran automatic hyperparameter optimization (HPO) tuning on four parameters (eta, alpha, min_child_weight, max_depth), which gave us 71% accuracy for the testing set.
• Finally, we worked with AutoGluon’s AutoML framework on SageMaker, which performs automatic modeling and hyperparameter selection with multiple models ensembling and multiple layers stacking. The framework trained 13 models and generated the final ensemble model yielding 74% accuracy for the testing set. We also tested by increasing the number of TF-IDF vectorizer features to 1,500; with the AutoGluon model, the classification accuracy on testing set can be further improved to 82%.

For our model training through AutoGluon, we used the MultilabelPredictor method from the AutoGluon library. This predictor performs multi-label prediction for tabular data. We used the sample notebook from AWS samples on GitHub. We used the same notebook by starting with importing AutoGluon libraries and defining the class for MultilabelPredictor(). To save space, we don’t show those lines in the following code snippet; you can copy/paste that part from the sample notebook. We employed the training in the file train.csv in our S3 bucket (your_path_to_s3/train.csv), specified the column used for label, and performed model training through MultilabelPredictor.

train_data = TabularDataset(‘your_path_to_s3/train.csv’)
subsample_size = 106                                                    # the sample size for training
train_data = train_data.sample(n=subsample_size, random_state=0)
labels = [‘label’]                                                      # column to foretell primarily based on the others
problem_types = ['multiclass']                                          # kind of every prediction drawback
save_path = ‘your_save_path_to_results’                                 # the trail to your s3 bucket for outcomes to retailer
time_limit = 60                                                         # variety of seconds to coach the TabularPredictor for every label

multi_predictor = MultilabelPredictor(labels=labels, problem_types=problem_types, path=save_path)
multi_predictor.match(train_data, time_limit=time_limit)

The next desk lists the AI/ML companies and fashions, and summarizes the accuracy.

The next charts summarize the accuracy for the pattern set primarily based on quantity of options.

Within the following charts, we noticed that the fashions of the choice tree with a gradient boosting machine, reminiscent of LGB, XGBoost, and Random Forest, had been higher decisions for any such drawback for each the 750-feature fashions and 1,500-feature fashions. The neural internet mannequin is ranked decrease among the many 13 fashions, which confirmed our expectation that deep studying won’t be appropriate for our case.

Conclusion

With AWS AI/ML companies, we are able to present correct and environment friendly contact motive and make contact with decision detection and different actionable insights for Amazon Worldwide Vendor Progress. MFN sellers can use these insights to raised perceive shopper issues, and take efficient actions to resolve Amazon customers’ points, whereas additionally optimizing their course of and prices.

You’ll be able to tailor the answer on your contact middle by creating your personal customized mannequin in SageMaker, and feeding the decision and chat transcripts for coaching and inference. You would additionally apply this resolution for normal theme detection to investigate buyer conversations in your contact middle.

In regards to the Authors

Yunfei Bai is a Senior Options Architect at AWS. With the background in AI/ML, Information Science and Analytics, Yunfei helps prospects undertake AWS companies to ship enterprise outcomes. He designs AI/ML and Information Analytics options that overcome advanced technical challenges and drive strategic targets. Yunfei is a PhD in Digital and Electrical Engineering . Outdoors of labor, Yunfei enjoys studying and music.

Burak Gozluklu is a Principal ML Specialist Options Architect situated in Boston, MA. Burak has +15 years of business expertise in simulation modeling, information science and ML know-how. He helps world prospects adopting AWS applied sciences and particularly, AI/ML options to realize their enterprise targets. Burak has a PhD in Aerospace Eng. from METU, MS in Techniques Engineering and post-doc on system dynamics from MIT in Cambridge, MA. Burak is enthusiastic about yoga and meditation.

Chelsea Cai is a Senior Product Supervisor at Amazon’s Worldwide Vendor Progress (ISG) group, the place she works for Buyer Service by Amazon service (CSBA) serving to 3P sellers enhance their customer support/CX by means of Amazon CS know-how and worldwide organizations. In her spare time, she likes philosophy, psychology, swimming, mountaineering, good meals, and spending time together with her household and buddies.

Abhishek Kumar is a Senior Product Supervisor at Amazon’s Worldwide Vendor Progress (ISG) group, the place he develops software program platforms and purposes to assist world 3P sellers handle their Amazon enterprise. In his free time, Abhishek enjoys touring, studying Italian, and exploring European cultures and cuisines together with his prolonged Italian household.