For a while I have been researching and trying on “Machine Learning” and “Natural Language Processing” topics. During my researches and tries, I also think about how and where I can implement these topics in the domain I am in. (At the end of the day, to see the nice impacts of an idea, which we implemented, on the end users, makes us as a developer happy, right?)
In this article, we will perform sentiment analysis on product comments in an e-commerce company using .NET Core and Azure Text Analytics API. Our goal is with the sentiment analysis to ensure that an end-user can have an idea without having read the comments about a product.
First of all, in a nutshell I want to talk about what sentiment analysis is.
In short sentiment analysis is:
We can say sentiment analysis is the process of determining positive or negative opinions from a text.
It is also known as idea mining, which examines the thought or attitude of a speaker. Especially in today’s technology age, with the rapid progress of machine learning, great works are carried out on sentiment analysis. If we look at the past (e.g 10 years ago), we can see that the sentiment analysis of the markets is done in the forex companies, and the buy and sell transactions are carried out in accordance with these analyses. (Forex player knowns)
Nowadays, companies use services such as sentiment analysis on social media to find out what people think about their products.
For example, we usually share what we eat or drink or how we feel on twitter or instagram. These sharing actions may seem like simple things, but many companies are able to analyze/process and determine which products or which direction they need to follow accordingly.
In this article, we will use the sentiment analysis in order to help the next user to choose and buy products more quickly. To be able to do this, we will discover feelings of other users about products.
There are many different methods to be able to do the sentiment analysis. For example, you can create your own sentiment analyser using Python’s VADER NLTK package (I’m currently working on it, maybe I can write an article about it) or you can choose a cloud provider API to saving time.
Okay, in this article we will use Azure Text Analytics API to perform the sentiment analysis.
So what is the Azure Text Analytics API and what does it offer us?
Azure Text Analytics API is a cloud-based service that allows us to perform advanced natural language processing over raw text. Especially when “time to market” become important, using cloud services such as the Azure Text Analytics API saves speed and time.
Azure Text Analytics API has a 4 core function as like below:
NOTE: Choosing free tier, we can use it up to 5000 transactions per month.
To use the Text Analytics API, let’s enter the “AI + Machine Learning” tab in the Azure marketplace and select the “Text Analytics“.
Then, we need to fill following fields and click the “create” button.
API is ready now.
We can access “endpoint” and “key” informations of the Azure Text Analytics API on the following overview screen to be able to use the next steps of the this article.
Well let’s assume we are working in an e-commerce company, and users can write comments about products, which they bought. I think, to be able to read product comments before we buy it, is important functionality in terms of both end-user and the company.
Well, if we could show an average end-user score to the end-users for each product by performing sentiment analysis on all the product comments instead of reading all product comments, wouldn’t that be perfect? Thus both end-users are not wasting much their time by reading all the comments and we may have the opportunity to convert the end-users visits into sales quickly.
Then let’s code!
First create a .NET Core “webapi” project called “SentimentAnalysisWithNETCoreExample” as like below.
dotnet new webapi -n SentimentAnalysisWithNETCoreExample
Then let’s include the “Microsoft.EntityFrameworkCore” package to the project with the following command.
dotnet add package Microsoft.EntityFrameworkCore
Now we can define our domain models.
Let’s create a folder called “Models“, and create another folder called “Domain” in the “Models” folder.
-> SentimentAnalysisWithNETCoreExample --> Models ---> Domain
In the “Domain” folder, create a class called “Product” as like below.
using System.Collections.Generic; namespace SentimentAnalysisWithNETCoreExample.Models.Domain { public class Product { public Product() { this.Comments = new List<Comment>(); } public int ID { get; set; } public string Name { get; set; } public string Category { get; set; } public double Price { get; set; } public double CustomerRating { get; set; } public ICollection<Comment> Comments { get; set; } } }
The important point here is the “CustomerRating” property. We will fill the value of this property with an average score result obtained as a result of the sentiment analysis of the product’s comments.
Now let’s create another class called “Comment” into the “Domain” folder.
namespace SentimentAnalysisWithNETCoreExample.Models.Domain { public class Comment { public int ID { get; set; } public int ProductID { get; set; } public string Content { get; set; } public double? SentimentScore { get; set; } } }
We will fill the value of the “SentimentScore” property with each comment’s own sentiment score.
Now we can create data context and sample dataset. Let’s create a new folder in the root directory called “Data“, and then add a class inside that called “ProductDBContext” as like below.
using Microsoft.EntityFrameworkCore; using SentimentAnalysisWithNETCoreExample.Models.Domain; namespace SentimentAnalysisWithNETCoreExample.Data { public class ProductDBContext : DbContext { public ProductDBContext(DbContextOptions<ProductDBContext> options) : base(options) { } public DbSet<Product> Products { get; set; } public DbSet<Comment> Comments { get; set; } protected override void OnModelCreating(ModelBuilder modelBuilder) { #region Products var product_1 = new Product() { ID = 1, Name = "Samsung Note 9", Category = "Cep Telefonu", Price = 600 }; var product_2 = new Product() { ID = 2, Name = "Samsung Galaxy 9", Category = "Cep Telefonu", Price = 550 }; modelBuilder.Entity<Product>().HasData(product_1, product_2); #endregion #region Comments var comment_1 = new Comment() { ID = 1, ProductID = 1, Content = "Çok harika bir telefon. Çok memnunum. Herkese tavsiye ederim." }; var comment_2 = new Comment() { ID = 2, ProductID = 1, Content = "Güzel telefon, gayet kullanışlı." }; var comment_3 = new Comment() { ID = 3, ProductID = 1, Content = "Hızlı telefon, beğendim." }; var comment_4 = new Comment() { ID = 4, ProductID = 1, Content = "Bataryası çok iyi dayanıyor tavsiye ederim." }; var comment_5 = new Comment() { ID = 5, ProductID = 2, Content = "Fena değil. Biraz ısınıyor. Daha iyi olabilirdi." }; var comment_6 = new Comment() { ID = 6, ProductID = 2, Content = "Kasasını beğenmedim. Elden çok kayıyor." }; var comment_7 = new Comment() { ID = 7, ProductID = 2, Content = "Memnunum, kamerası çok iyi." }; var comment_8 = new Comment() { ID = 8, ProductID = 2, Content = "Kamerasının gece çekimini beğenmedim." }; modelBuilder.Entity<Comment>().HasData(comment_1, comment_2, comment_3, comment_4, comment_5, comment_6, comment_7, comment_8); #endregion base.OnModelCreating(modelBuilder); } } }
We created data context by inheriting from the “DbContext” class in a standard way. We added the “Products” and “Comments” dbsets in the “ProductDBContext” class. To have a sample dataset, we added a few products and comments in the “OnModelCreating” method.
Now, before start coding the business services, let’s declare request & response models.
To do this, we need to create “Requests” and “Responses” folders in the “Models” folder. Than let’s add “GetSentimentAnalysisRequest” and “GetSentimentAnalysisRequestItem” classes in the “Requests” folder as like below.
using System.Collections.Generic; namespace SentimentAnalysisWithNETCoreExample.Models.Requests { public class GetSentimentAnalysisRequest { public GetSentimentAnalysisRequest() { Documents = new List<GetSentimentAnalysisRequestItem>(); } public IList<GetSentimentAnalysisRequestItem> Documents { get; set; } } }
namespace SentimentAnalysisWithNETCoreExample.Models.Requests { public class GetSentimentAnalysisRequestItem { public string Language { get; set; } public string Id { get; set; } public string Text { get; set; } } }
We will use the “GetSentimentAnalysisRequest” and “GetSentimentAnalysisRequestItem” classes to call the Azure Text Analytics API‘s sentiment endpoint.
The sentiment endpoint expects a request like below:
{ "documents": [ { "language": "comment lang", "id": "comment id", "text": "comment" }, { "language": "comment lang", "id": "comment id", "text": "comment" } ] }
NOTE: There are a NuGet package to use Azure Text Analytics API. But currently it’s in preview state, and support .NET Standard 1.4. Therefore, we will implement our request models and services to use on .NET Core 2.1.
We will create response models for the sentiment endpoint in the “Responses” folder which we created.
So let’s create “GetSentimentAnalysisResponse” and “GetSentimentAnalysisResponseItem” classes in the “Responses” folder as like below.
using System.Collections.Generic; namespace SentimentAnalysisWithNETCoreExample.Models.Responses { public class GetSentimentAnalysisResponse { public IList<GetSentimentAnalysisResponseItem> Documents { get; set; } } }
namespace SentimentAnalysisWithNETCoreExample.Models.Responses { public class GetSentimentAnalysisResponseItem { public string Id { get; set; } public double? Score { get; set; } } }
Now we created the response models to get the sentiment results and we can start to implement the service.
To do that, let’s create a new folder called “Services” in the root directory and define an interface called “ITextAnalyticsService“.
using System.Threading.Tasks; using SentimentAnalysisWithNETCoreExample.Models.Requests; using SentimentAnalysisWithNETCoreExample.Models.Responses; namespace SentimentAnalysisWithNETCoreExample.Services { public interface ITextAnalyticsService { Task<GetSentimentAnalysisResponse> GetSentimentAsync(GetSentimentAnalysisRequest request); } }
After that, let’s create another folder called “Implementations” in the “Services” folder and inside create a new class called “TextAnalyticsService“, and then implement “ITextAnalyticsService” interface as like below.
using System.Net; using System.Net.Http; using System.Threading.Tasks; using Microsoft.Extensions.Configuration; using Newtonsoft.Json; using SentimentAnalysisWithNETCoreExample.Models.Requests; using SentimentAnalysisWithNETCoreExample.Models.Responses; namespace SentimentAnalysisWithNETCoreExample.Services.Implementations { public class TextAnalyticsService : ITextAnalyticsService { private readonly IHttpClientFactory _httpClientFactory; private readonly IConfiguration _configuration; public TextAnalyticsService(IHttpClientFactory httpClientFactory, IConfiguration configuration) { _httpClientFactory = httpClientFactory; _configuration = configuration; } public async Task<GetSentimentAnalysisResponse> GetSentimentAsync(GetSentimentAnalysisRequest request) { HttpClient client = _httpClientFactory.CreateClient("TextAnalyticsAPI"); var sentimentResponse = await client.PostAsJsonAsync( requestUri: _configuration.GetValue<string>("TextAnalyticsAPISentimentResourceURI"), value: request); GetSentimentAnalysisResponse getSentimentAnalysisResponse = null; if(sentimentResponse.StatusCode == HttpStatusCode.OK) { getSentimentAnalysisResponse = await sentimentResponse.Content.ReadAsAsync<GetSentimentAnalysisResponse>(); } return getSentimentAnalysisResponse; } } }
Actually, we implemented the “TextAnalyticsService” in a simple way. Let’s take a look.
We created the HttpClientFactory via the “TextAnalyticsAPI” key with named-client approach. Then we performed the POST operation by retrieving the sentiment resource URI of the Azure Text Analytics API through the “IConfiguration” service. If the operation is successfully completed, we are mapping the response with the “GetSentimentAnalysisResponse” model which we created before.
Now, let’s define “IProductCommentService” interface under the “Services” folder to implement product comments operations as follows.
using System.Collections.Generic; using System.Threading.Tasks; using SentimentAnalysisWithNETCoreExample.Models.Domain; namespace SentimentAnalysisWithNETCoreExample.Services { public interface IProductCommentService { Task<List<Comment>> GetComments(int productId); Task<List<Comment>> CalculateCommentsSentimentScoreAsync(List<Comment> comments); } }
Then, create a class called “ProductCommentService” in the “Implementations” folder under the “Services” folder, and implement the “IProductCommentService” interface as like below.
using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; using Microsoft.EntityFrameworkCore; using SentimentAnalysisWithNETCoreExample.Data; using SentimentAnalysisWithNETCoreExample.Models.Domain; using SentimentAnalysisWithNETCoreExample.Models.Requests; using SentimentAnalysisWithNETCoreExample.Models.Responses; namespace SentimentAnalysisWithNETCoreExample.Services.Implementations { public class ProductCommentService : IProductCommentService { private const string LANG = "tr"; private readonly ProductDBContext _productDBContext; private readonly ITextAnalyticsService _textAnalyticsService; public ProductCommentService(ProductDBContext productDBContext, ITextAnalyticsService textAnalyticsService) { _productDBContext = productDBContext; _textAnalyticsService = textAnalyticsService; } public async Task<List<Comment>> GetComments(int productId) { List<Comment> comments = await _productDBContext.Comments.Where(c => c.ProductID == productId) .ToListAsync(); return comments; } public async Task<List<Comment>> CalculateCommentsSentimentScoreAsync(List<Comment> comments) { var getSentimentAnalysisRequest = new GetSentimentAnalysisRequest(); comments.ForEach(comment => { var multiLanguageInput = new GetSentimentAnalysisRequestItem() { Language = LANG, Id = comment.ID.ToString(), Text = comment.Content }; getSentimentAnalysisRequest.Documents.Add(multiLanguageInput); }); GetSentimentAnalysisResponse getSentimentAnalysisResponse = await _textAnalyticsService.GetSentimentAsync(getSentimentAnalysisRequest); if (getSentimentAnalysisResponse != null && getSentimentAnalysisResponse.Documents.Count > 0) { // Add sentiment analysis result to the comments foreach (GetSentimentAnalysisResponseItem getSentimentAnalysisResponseItem in getSentimentAnalysisResponse.Documents) { Comment comment = comments.FirstOrDefault(c => c.ID == Convert.ToInt32(getSentimentAnalysisResponseItem.Id)); comment.SentimentScore = getSentimentAnalysisResponseItem.Score; } } return comments; } } }
We are getting related product comments from database with the “GetCommentsAsync” method. In the “CalculateCommentsSentimentScoreAsync” method, we are calculating sentiment scores of comments with the service of Azure Text Analytics API which we created. If not any errors occur while calling the API, then we are mapping sentiment scores of the comments.
Now, we need another service where we can perform operations related to products.
First of all, we need to define product response model in the “Models/Responses” folder, that we created before, in order to not expose the domain model to the outside.
Let’s define “GetProductResponse” and “GetProductCommentResponse” classes in the “Responses” folder as follows.
using System.Collections.Generic; namespace SentimentAnalysisWithNETCoreExample.Models.Responses { public class GetProductResponse { public int ID { get; set; } public string Name { get; set; } public string Category { get; set; } public double Price { get; set; } public double CustomerRating { get; set; } public List<GetProductCommentResponse> Comments { get; set; } } }
namespace SentimentAnalysisWithNETCoreExample.Models.Responses { public class GetProductCommentResponse { public string Content { get; set; } } }
After defining the models, let’s create another interface called “IProductService” in the “Services” folder.
using System.Collections.Generic; using System.Threading.Tasks; using SentimentAnalysisWithNETCoreExample.Models.Domain; using SentimentAnalysisWithNETCoreExample.Models.Responses; namespace SentimentAnalysisWithNETCoreExample.Services { public interface IProductService { Task<GetProductResponse> GetProductAsync(int id); } }
Then, create a class named “ProductService” in the “Services/Implementations” folder and implement as follows.
using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; using Microsoft.EntityFrameworkCore; using SentimentAnalysisWithNETCoreExample.Data; using SentimentAnalysisWithNETCoreExample.Models.Domain; using SentimentAnalysisWithNETCoreExample.Models.Responses; namespace SentimentAnalysisWithNETCoreExample.Services.Implementations { public class ProductService : IProductService { private readonly ProductDBContext _productDBContext; private readonly IProductCommentService _productCommentService; public ProductService(ProductDBContext productDBContext, IProductCommentService productCommentService) { _productDBContext = productDBContext; _productCommentService = productCommentService; } public async Task<GetProductResponse> GetProductAsync(int id) { Product product = await _productDBContext.Products.Where(p => p.ID == id) .FirstOrDefaultAsync(); GetProductResponse productResponse = null; if (product != null) { productResponse = new GetProductResponse(); productResponse.ID = product.ID; productResponse.Name = product.Name; productResponse.Category = product.Category; productResponse.Price = product.Price; List<Comment> comments = await _productCommentService.GetComments(id); productResponse.Comments = comments.Select(c => new GetProductCommentResponse() { Content = c.Content }).ToList(); if (comments.Count > 0) { List<Comment> commentsWithSentimentAnalysis = await _productCommentService.CalculateCommentsSentimentScoreAsync(comments); productResponse.CustomerRating = await CalculateProductCustomerRatingScore(commentsWithSentimentAnalysis); } } return productResponse; } private async Task<double> CalculateProductCustomerRatingScore(List<Comment> comments) { double sentimentScores = 0; double customerRating = 0; comments.ForEach(_comment => { sentimentScores += _comment.SentimentScore.Value; }); customerRating = (sentimentScores / comments.Count()); return await Task.FromResult(customerRating); } } }
If we look at the “GetProductAsync” method that we created, we are getting the related product comments with the “IProductCommentService” which we injected.
If related product comments are not null, we are calculating sentiment scores of related comments using the “CalculateCommentsSentimentScoreAsync” method in the “IProductCommentService“.
Then, we are using the “CalculateProductCustomerRatingScoreAsync” private method to get an average score of how users feel (positive/negative) about the product.
Finally we have completed the sections of defining and implementing services.
Now let’s define a controller called “ProductsController” under the “Controllers” folder as follows.
using System.Collections.Generic; using System.Threading.Tasks; using Microsoft.AspNetCore.Mvc; using SentimentAnalysisWithNETCoreExample.Models.Domain; using SentimentAnalysisWithNETCoreExample.Models.Responses; using SentimentAnalysisWithNETCoreExample.Services; namespace SentimentAnalysisWithNETCoreExample.Controllers { [Route("api/[controller]")] [ApiController] public class ProductsController : ControllerBase { private readonly IProductService _productService; public ProductsController(IProductService productService) { _productService = productService; } [HttpGet("{id}")] public async Task<ActionResult> GetProduct(int id) { GetProductResponse product = await _productService.GetProductAsync(id); if (product != null) { return Ok(product); } else { return NotFound(); } } } }
In the above controller class, after injecting the “IProductService” interface, we exposed a GET endpoint. Now we have an endpoint that we can get products by “id“.
Now let’s update the “Startup” class in order to provide the necessary actions such as injection as like below.
using System; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Hosting; using Microsoft.AspNetCore.Mvc; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using SentimentAnalysisWithNETCoreExample.Data; using SentimentAnalysisWithNETCoreExample.Services; using SentimentAnalysisWithNETCoreExample.Services.Implementations; namespace SentimentAnalysisWithNETCoreExample { public class Startup { public Startup(IConfiguration configuration) { Configuration = configuration; } public IConfiguration Configuration { get; } // This method gets called by the runtime. Use this method to add services to the container. public void ConfigureServices(IServiceCollection services) { services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1); services.AddHttpClient("TextAnalyticsAPI", c => { c.BaseAddress = new Uri(Configuration.GetValue<string>("TextAnalyticsAPIBaseAddress")); c.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", Configuration.GetValue<string>("TextAnalyticsAPIKey")); }); //Services services.AddTransient<ITextAnalyticsService, TextAnalyticsService>(); services.AddTransient<IProductService, ProductService>(); services.AddTransient<IProductCommentService, ProductCommentService>(); //Data services.AddDbContext<ProductDBContext>(option => option.UseInMemoryDatabase("ProductDBContext")); } // This method gets called by the runtime. Use this method to configure the HTTP request pipeline. public void Configure(IApplicationBuilder app, IHostingEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } else { app.UseHsts(); } using (var serviceScope = app.ApplicationServices.CreateScope()) { var context = serviceScope.ServiceProvider.GetService<ProductDBContext>(); context.Database.EnsureCreated(); } app.UseHttpsRedirection(); app.UseMvc(); } } }
If we look at the above code block, we injected HttpClient with Azure subscription key and the “TextAnalyticsAPI” name. Then we injected the “ITextAnalyticsService“, “IProductService” and “IProductCommentService” interfaces.
We specified DbContext as an in-memory. In the “Configure” method, we have ensured the initialization of sample dataset over the DbContext.
Now let’s add the corresponding configuration keys to the “appsettings” json file as follows.
{ "Logging": { "LogLevel": { "Default": "Warning" } }, "AllowedHosts": "*", "TextAnalyticsAPIBaseAddress" : "https://westcentralus.api.cognitive.microsoft.com", "TextAnalyticsAPISentimentResourceURI" : "text/analytics/v2.0/sentiment", "TextAnalyticsAPIKey" : "YOUR_KEY" }
The values of the “TextAnalyticsAPIBaseAddress“, “TextAnalyticsAPISentimentResourceURI” and “TextAnalyticsAPIKey” keys that we have added above can be found in the Text Analytics resource, that we have created through the Azure Portal in the beginning section of this article.
Now we are ready to test!
First, let’s run the API with the “dotnet run” command. Then, let’s get the first sample product, which we have prepared with positive comments via the “https://localhost:5001/api/products/1” endpoint.
If we look at the above response, we can see that we get an average “customerRating” score based on 4 comment’s sentiment results. With this result, which is evaluated in the range of “0” to “1“, we can say that the users have reviewed this product with an average rate of 94% as positively.
Now let’s take a look at the result of the second sample product that contains some negative comments.
In this scenario, as a result of sentiment analysis, the response has an average “customerRating” rate of 51%, since the product has some negative comments.
As I mentioned in the beginning of this article, you can create your own sentiment analyzer with different languages and tools such as Python’s VADER NLTK package. If you wish, you can also benefit from ready-to-use timesaving cloud providers. Within the scope of this article, we have tried to look at how we can benefit sentiment analysis on product reviews on an e-commerce platform using the Azure Text Analytics API.
https://github.com/GokGokalp/AzureTextAnalyticsAPI-Sentiment-Sample
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/quickstarts/csharp
https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/overview
{:tr} Makalenin ilk bölümünde, Software Supply Chain güvenliğinin öneminden ve containerized uygulamaların güvenlik risklerini azaltabilmek…
{:tr}Bildiğimiz gibi modern yazılım geliştirme ortamında containerization'ın benimsenmesi, uygulamaların oluşturulma ve dağıtılma şekillerini oldukça değiştirdi.…
{:tr}Bildiğimiz gibi bir ürün geliştirirken olabildiğince farklı cloud çözümlerinden faydalanmak, harcanacak zaman ve karmaşıklığın yanı…
{:tr}Bazen bazı senaryolar vardır karmaşıklığını veya eksi yanlarını bildiğimiz halde implemente etmekten kaçamadığımız veya implemente…
{:tr}Bildiğimiz gibi microservice architecture'ına adapte olmanın bir çok artı noktası olduğu gibi, maalesef getirdiği bazı…
{:tr}Bir önceki makale serisinde Dapr projesinden ve faydalarından bahsedip, local ortamda self-hosted mode olarak .NET…
View Comments
Teşekkürler, faydalı bir makale olmuş.
Ben teşekkür ederim.
Asp.NET'te yazılan bir ERP projesine doğal dil işleme eklenmek istense, ve azurenin yapıtğı olayı farklı bir bakış acısı ile yazılması öngörülse,
Bu algoritmayı sadece ar-ge olabilecek kısımlarının Pyhton ya da Core tabanlı olarak
hangisinde yazılması daha isabetli olur. Bu noktada görüşünüzü alabilir miyim.
Merhaba ar-ge kapsamında değerlendirmek istiyorsanız, tabi ki bu logic'i kendiniz istediğiniz bir dil ile yazmanız isabetli olacaktır. Bu konuda Python diyebilirim, hem community hemde kullanabileceğiniz library'ler konusunda daha zengin bir seçim olacaktır.
Teşekkürler.
Merhaba. Gece 3 te okudum. Uykumdan ettin :) kalktım denedim. Güzel bir anlatım olmuş. Teşekkürler
Beğenmene sevindim. Teşekkür ederim. :)
Güzel paylaşım teşekkürler
Python‘ın VADER NLTK paketi hakkında bir makale yazmayı düşünüyor musunuz ?
Ve bu paket hakkında önerebileceğiniz bir kaynak var mı ?
Merhaba vakit bulabilirsem bir kaç örnek ekleyeceğim.