Elasticsearch Series 04 – Building Autocomplete API with Completion Suggester in ASP.NET Core

When we look at many popular commercial websites such as Amazon, Netflix and eBay, we can clearly see that autocomplete(search suggestion) boxes are important for the companies.

We know that a good search result is also important for end users. In terms of commercial websites, when we are able to direct the end user to the right product or category quickly, this situation will affect sales rates positively.

To be able to see related suggestions while we are writing, isn’t it great?

And I decided to write an autosuggest sample using Elasticsearch – Completion Suggester and .NET Core according to e-mails which followers requested.

There are several different ways to implement autocomplete/suggest feature in elasticsearch such as “ngrams“, “prefix queries” and “completion suggester“. Also, different implementations have some tradeoffs such as resulting and indexing speeds, document sizes etc… For “autocomplete/search-as-you-type” functionality example in this article, I will try to show how we can implement autocomplete feature in a most performant way(I think) using “completion suggester” feature.

Completion Suggester

I think resulting speed is important when it’s about to give an instant feedback to end users with a good suggestion result. At this point, completion suggester works differently from other implementation ways. All combinations to be suggested, need to be indexed on elasticsearch with a “completion” type mapping. Completion suggester uses an in-memory data structure called FST(Finite-state transducer) to provide fast lookup operation.
Thus, it can perform prefix lookup operation faster than other term-based queries.

Let’s take a look at the example on the elastic engineering blog to understand it working logic better. Suppose that “hotel“, “marriot“, “mercure“, “munchen” and “munich” words are on a FST.

On the in-memory graph above, suggester performs a matching process from left to right according to the text input which entered by a user. For example, when the user enters “h” as an input, this word will be completed immediately because the only possible matching option is “hotel“. If the user enters “m“, this time the suggester will list all words which start with “m“.

The disadvantage of completion suggester that matching process always starts from left to right as like above. For example “Sam” text will be matched with “Samsung Note 8” not with “Note 8 Samsung“. In such cases, term-based queries work more efficiently. However, as I mentioned above, we can also perform the match operation of “Note 8 Samsung” word by indexing all related combinations to be suggested on a single suggest output with completion suggester. I will mention this in an example later.

It is also possible to specify “Fuzzy Matching” with completion suggester and “Weights” for scoring operations.

Scenario

Let’s assume we are working at an e-commerce company. Product owner, responsible for search domain, asked us to create an autocomplete feature that gives a result close to real-time based on the “brand” and “product name” fields.

First we need to feed items, that we want to suggest on elasticsearch.

1) Creation of Mapping and Index

We have to create a mapping, which has completion type, to use completion suggester. For that, first create a .NET Core class library called “Autocomplete.Business.Objects” and include “NEST” library using NuGet package manager.

We created this library as separately, because we will use the models which we define here, in both feeder application and autocomplete API.

First, define “Product” and “ProductSuggestResponse” models as follows.

using Nest;

namespace Autocomplete.Business.Objects
{
    public class Product
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public CompletionField Suggest {get;set;}
    }
}

using System.Collections.Generic;

namespace Autocomplete.Business.Objects
{
    public class ProductSuggestResponse
    {
        public IEnumerable<ProductSuggest> Suggests { get; set; }
    }

    public class ProductSuggest
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public double Score { get; set; }  
    }
}

We will use the “Suggest” property in the “Product” model for texts that we want to suggest during autocomplete.

Create a new .NET Core class library called “Autocomplete.Business“, then include the “Autocomplete.Business.Objects” and “NEST” libraries. After that let’s define an interface called “IAutocompleteService“.

using System.Collections.Generic;
using System.Threading.Tasks;
using Autocomplete.Business.Objects;

namespace Autocomplete.Business
{
    public interface IAutocompleteService
    {
        Task<bool> CreateIndexAsync(string indexName);
        Task IndexAsync(string indexName, List<Product> products);
        Task<ProductSuggestResponse> SuggestAsync(string indexName, string keyword);
    }
}

and implement it as follows.

using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Autocomplete.Business.Objects;
using Nest;

namespace Autocomplete.Business
{
    public class AutocompleteService : IAutocompleteService
    {
        readonly ElasticClient _elasticClient;

        public AutocompleteService(ConnectionSettings connectionSettings)
        {
            _elasticClient = new ElasticClient(connectionSettings);
        }

        public async Task<bool> CreateIndexAsync(string indexName)
        {
            var createIndexDescriptor = new CreateIndexDescriptor(indexName)
                .Mappings(ms => ms
                          .Map<Product>(m => m
                                .AutoMap()
                                .Properties(ps => ps
                                    .Completion(c => c
                                        .Name(p => p.Suggest))))

                         );

            if (_elasticClient.IndexExists(indexName.ToLowerInvariant()).Exists)
            {
                _elasticClient.DeleteIndex(indexName.ToLowerInvariant());
            }

            ICreateIndexResponse createIndexResponse = await _elasticClient.CreateIndexAsync(createIndexDescriptor);

            return createIndexResponse.IsValid;
        }

        public async Task IndexAsync(string indexName, List<Product> products)
        {
            await _elasticClient.IndexManyAsync(products, indexName);
        }

        public async Task<ProductSuggestResponse> SuggestAsync(string indexName, string keyword)
        {
            ISearchResponse<Product> searchResponse = await _elasticClient.SearchAsync<Product>(s => s
                                     .Index(indexName)
                                     .Suggest(su => su
                                          .Completion("suggestions", c => c
                                               .Field(f => f.Suggest)
                                               .Prefix(keyword)
                                               .Fuzzy(f => f
                                                   .Fuzziness(Fuzziness.Auto)
                                               )
                                               .Size(5))
                                             ));

            var suggests = from suggest in searchResponse.Suggest["suggestions"]
                              from option in suggest.Options
                              select new ProductSuggest
                              {
                                    Id = option.Source.Id,
                                    Name = option.Source.Name,
                                    SuggestedName = option.Text,
                                    Score = option.Score
                              };

            return new ProductSuggestResponse
            {
                Suggests = suggests
            };
        }
    }
}

If we look at the “CreateIndexAsync” method above, we did mapping process of “Product” model. Also, we specified “Suggest” property in the “Product” model as completion field. At this point, default “simple” analyzer is used as the analyzer. Simple analyzer divides all text into terms as lower case. If you want, it is possible to replace the analyzer with the “Analyzer” method on the completion.

The “SuggestAsync” method will be used during autocomplete. Actually, we specified that we will perform completion operation with the “Suggest” field. Also, we set text input that user will enter, to the completion using the “Prefix” method. I guess, “Fuzzy” is great feature while autocomplete process. Sometimes we can make simple typo mistakes when we write, right? 🙂

And the last one is, we mapped “Text” and “Score” properties of suggestion options which are come from “searchResponse“, to “ProductSuggest” model.

NOTE: It is also possible to access original document of suggested text from “Source” property.

Finally, we can start to develop console application which will feed suggestion documents to elasticsearch. So, let’s create a .NET Core console application project called “Autocomplete.Feed“, then include the “Autocomplete.Business.Objects” and “Autocomplete.Business” libraries.

First of all, we need an elasticsearch instance to test. So let’s run the following command on Docker.

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.1.2

and change the “Program.cs” class with the following code blog.

using System;
using System.Collections.Generic;
using Autocomplete.Business;
using Autocomplete.Business.Objects;
using Nest;

namespace Autocomplete.Feed
{
    class Program
    {
        static void Main(string[] args)
        {
            List<Product> products = new List<Product>();

            products.Add(new Product()
            {
                Id = 1,
                Name = "Samsung Galaxy Note 8",
                Suggest = new CompletionField()
                {
                    Input = new [] { "Samsung Galaxy Note 8", "Galaxy Note 8", "Note 8" }
                }
            });

            products.Add(new Product()
            {
                Id = 2,
                Name = "Samsung Galaxy S8",
                Suggest = new CompletionField()
                {
                    Input = new[] { "Samsung Galaxy S8", "Galaxy S8", "S8" }             
                }
            });

            products.Add(new Product()
            {
                Id = 3,
                Name = "Apple Iphone 8",
                Suggest = new CompletionField()
                {
                    Input = new[] { "Apple Iphone 8", "Iphone 8" }
                }
            });

            products.Add(new Product()
            {
                Id = 4,
                Name = "Apple Iphone X",
                Suggest = new CompletionField()
                {
                    Input = new[] { "Apple Iphone X", "Iphone X" }
                }
            });

            products.Add(new Product()
            {
                Id = 5,
                Name = "Apple iPad Pro",
                Suggest = new CompletionField()
                {
                    Input = new[] { "Apple iPad Pro", "iPad Pro" }
                }
            });

            var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200"));
            IAutocompleteService autocompleteService = new AutocompleteService(connectionSettings);
            string productSuggestIndex = "product_suggest";

            bool isCreated = autocompleteService.CreateIndexAsync(productSuggestIndex).Result;

            if(isCreated)
            {
                autocompleteService.IndexAsync(productSuggestIndex, products).Wait();
            }
        }
    }
}

If we look at code blog above, firstly we created the products that will be used in autocomplete. Also while creating the products, we set the inputs, these are wanted to match with the related product, to “CompletionField” property of each product. Thus, if a user writes an input such as “Galaxy Note 8” or “Note 8“, we can provide matching these inputs with “Samsung Galaxy Note 8“.

After creation of the products, we have provided feed operation to elasticsearch using “AutocompleteService” class, which created before.

So we have a feeder project now, and let’s run it. If it runs successfully, we can see the “product_suggest” index on the elasticsearch with the following mapping.

“GET product_suggest/_mapping”

{
   "product_suggest": {
      "mappings": {
         "product": {
            "properties": {
               "id": {
                  "type": "integer"
               },
               "name": {
                  "type": "text",
                  "fields": {
                     "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                     }
                  }
               },
               "suggest": {
                  "type": "completion",
                  "analyzer": "simple",
                  "preserve_separators": true,
                  "preserve_position_increments": true,
                  "max_input_length": 50
               }
            }
         }
      }
   }
}

2) Designing of Autocomplete API

Now, all we need is to do design an API, to expose autocomplete feature. Let’s create a .NET Core Web API project called “Autocomplete.API“, then include the “Autocomplete.Business.Objects“, “Autocomplete.Business” and “NEST” libraries.

After that, create a controller called “ProductSuggests“.

using System.Threading.Tasks;
using Autocomplete.Business;
using Autocomplete.Business.Objects;
using Microsoft.AspNetCore.Mvc;

namespace Autocomplete.API.Controllers
{
    [Route("api/product-suggests")]
    public class ProductSuggestsController : Controller
    {
        readonly IAutocompleteService _autocompleteService;
        const string PRODUCT_SUGGEST_INDEX = "product_suggest";

        public ProductSuggestsController(IAutocompleteService autocompleteService)
        {
            _autocompleteService = autocompleteService;
        }

        [HttpGet]
        public async Task<ProductSuggestResponse> Get(string keyword)
        {
            return await _autocompleteService.SuggestAsync(PRODUCT_SUGGEST_INDEX, keyword);
        }
    }
}

In the “Get” method, we just return the “ProductSuggestResponse” using the “IAutocompleteService“.

Also in the “Startup” class, we need to inject services into service collection.

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    services.AddSingleton(x => new ConnectionSettings(new Uri("http://localhost:9200")));
    services.AddTransient<IAutocompleteService, AutocompleteService>();
}

That’s all.

Let’s run the API project for testing autocomplete feature, and then assume a user entered “iph“.

GET “http://localhost:5000/api/product-suggests?keyword=iph”

If we look at the response above, we can see that the related results for the “iph” such as “Iphone 8“, “Iphone X” and “iPad Pro“.

Now if we assume the user entered “app” instead of the “iph“:

and this time “Apple Iphone 8“, “Apple Iphone X” and “Apple iPad Pro” results will suggested to the user.

Conclusion

As I mentioned the beginning of this article, there are many ways implement to autocomplete feature in the elasticsearch with some tradeoffs. Completion suggester works faster than term-based queries. Also, it is a disadvantage of completion suggester to start matching operation at the beginning of the text. In addition, sort order options are limited.

https://github.com/GokGokalp/Elasticsearch-Autocomplete-API-Sample

References

https://www.elastic.co/blog/you-complete-me

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters-completion.html