Let’s Elastic Search 🔍 with NET 8🔥 and Kibana

Elastic Search offers several benefits, including high-performance search, real-time search, full-text search, faceting, geolocation search, analytics capabilities, ease of use, scalability, reliability, and open-source nature. These features make it a popular choice for search and analytics applications, as it can handle large datasets, provide fast and accurate results, and be easily integrated into different systems.

Setting up Elastic Search Locally using Docker Compose

Let's begin by creating a new .NET project and then add a new file named docker-compose.yml to the root of the project. This file will be used to define the services that will be used in the Docker Compose environment.

version: '3.8'
services:
  elasticsearch:
    container_name: els
    image: elasticsearch:8.15.0
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    networks:
      - elk
  kibana:
    container_name: kibana
    image: kibana:8.15.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    networks:
      - elk
networks:
  elk:
    driver: bridge
volumes:
  elasticsearch-data:

This docker-compose.yml file defines two services: Elasticsearch and Kibana.

The Elasticsearch service is configured to run a single node of Elastic search version 8.15.0. It also maps port 9200 of the container to port 9200 of the host machine. This is the port that Elastic search uses to communicate with clients. The volumes section defines a volume named Elasticsearch-data, that will be used to store the data of the Elasticsearch service. This volume will be mounted to the /usr/share/elasticsearch/data directory of the container. The environment section disables Elastic search’s security features. Finally, the networks section specifies that the Elasticsearch service should be attached to the elk network, which is defined later in the file.

The Kibana service is configured to run Kibana version 8.15.0. It maps port 5601 of the container to port 5601 of the host machine. Kibana is a web interface for Elastic search and allows you to visualize and interact with your data. The depends_on section specifies that the Kibana service depends on the Elasticsearch service, meaning that Kibana will not start until Elastic search is running. The environment section sets the ELASTICSEARCH_URL environment variable, which tells Kibana where to find the Elastic search instance. Finally, the networks section specifies that the Kibana service should be attached to the elk network.

The networks section defines a bridge network called elk. This network will be used to allow the Elasticsearch and Kibana services to communicate with each other.

The volumes section defines a named volume called Elasticsearch-data. This volume will be used to store the data of the Elasticsearch service.

To start the Elastic search and Kibana services, lets opens a terminal window and runs the following command:

docker-compose up

This command will start the Elastic search and Kibana services in the background. Once the services are running, you can access Kibana by navigating to http://localhost:5601 in your web browser.

Integrating Elastic Search with a .NET Web API

Then creates a new .NET Web API project and installs the following NuGet package:

dotnet add package Elastic.Clients.Elasticsearch

This package provides the necessary classes to interact with Elastic search from .NET.

Next, let’s create a new folder named Models and add a new class named User to this folder. This class will represent a user in the system.

public class User
{
  [JsonProperty("Id")]
  public int Id { get; set; }

[JsonProperty("FirstName")]
  public string FirstName { get; set; }
  [JsonProperty("LastName")]
  public string LastName { get; set; }
}

Then create a new folder named Services and adds a new interface named IElasticService to this folder. This interface will define the methods that will be used to interact with Elastic search.

public interface IElasticService
{
    /// create index
    Task CreateIndexIfNotExistsAsync(string indexName);
    /// add or update user
    Task<bool> AddOrUpdate(User user);
    /// add or update user bulk
    Task<bool> AddOrUpdateBulk(IEnumerable<User> users, string indexName);
    /// get user by id
    Task<User> Get(string key);
    /// get all users
    Task<List<User>> GetAll();
    /// remove 
    Task<bool> Remove(string key);
    /// remove all
    Task<long> RemoveAll();
}

The IElasticService interface defines seven methods:

• CreateIndexIfNotExistsAsync: This method creates an index in Elastic search if it does not already exist.
• AddOrUpdate: This method adds or updates a user in Elastic search.
• AddOrUpdateBulk: This method adds or updates multiple users in Elastic search.
• Get: This method retrieves a user from Elastic search by its key.
• GetAll: This method retrieves all users from Elastic search.
• Remove: This method removes a user from Elastic search by its key.
• RemoveAll: This method removes all users from Elastic search.

Then adds a new class named ElasticService to the Services folder. This class will implement the IElasticService interface.

public class ElasticService : IElasticService
{
    private readonly ElasticsearchClient _client;
    private readonly ElasticSettings _elasticSettings;

    public ElasticService(IOptions<ElasticSettings> options)
    {
        _elasticSettings = options.Value;
        var settings = new ElasticsearchClientSettings(new Uri(_elasticSettings.Url))
            // .Authentication(new BasicAuthentication(_elasticSettings.Username, _elasticSettings.Password))
            .DefaultIndex(_elasticSettings.DefaultIndex);
        _client = new ElasticsearchClient(settings);
    }
    public async Task CreateIndexIfNotExistsAsync(string indexName)
    {
        if (!_client.Indices.Exists(indexName).Exists)
        {
            await _client.Indices.CreateAsync(indexName);
        }
    }
    public async Task<bool> AddOrUpdate(User user)
    {
        var response = await _client.IndexAsync(user, idx => idx
            .Index(_elasticSettings.DefaultIndex)
            .Id(user.Id)
            .Refresh(Refresh.WaitFor)); // TaskIndexResponse
        return response.IsValidResponse;
    }
    public async Task<bool> AddOrUpdateBulk(IEnumerable<User> users, string indexName)
    {
        var response = await _client.BulkAsync(b => b
            .Index(_elasticSettings.DefaultIndex)
            .UpdateMany(users, (ud, u) => ud.Doc(u).DocAsUpsert(true))); // TaskBulkResponse
        return response.IsValidResponse;
    }
    public async Task<User> Get(string key)
    {
        var response = await _client.GetAsync<User>(key,
            g => g.Index(_elasticSettings.DefaultIndex)); // TaskGetResponse<User>
        return response.Source;
    }
    public async Task<List<User>> GetAll()
    {
        var response = await _client.SearchAsync<User>(s => s
            .Index(_elasticSettings.DefaultIndex)); // TaskSearchResponse<User>
        return response.IsValidResponse ? response.Documents.ToList() : default;
    }
    public async Task<bool> Remove(string key)
    {
        var response = await _client.DeleteAsync<User>(key,
            d => d.Index(_elasticSettings.DefaultIndex)); // TaskDeleteResponse
        return response.IsValidResponse;
    }
    public async Task<long> RemoveAll()
    {
        var response = await _client.DeleteByQueryAsync<User>(d => d
            .Index(_elasticSettings.DefaultIndex)
            .Query(q => q.MatchAll()));
        return response.IsValidResponse ? response.Deleted : default;
    }
}

The ElasticService class has two private fields: _client and _elasticSettings. The _client field is an instance of the ElasticsearchClient class, which is used to interact with Elastic search. The _elasticSettings field is an instance of the ElasticSettings class, which contains the configuration settings for Elastic search.

The ElasticService class has a constructor that takes an IOptions<ElasticSettings> object as a parameter. This object is used to retrieve the configuration settings for Elastic search from the application’s configuration file. The constructor uses these settings to create an instance of the ElasticsearchClient class.

The ElasticService class implements the seven methods defined in the IElasticService interface. Each of these methods uses the _client field to interact with Elastic search.

Configuring the .NET Web API to Use Elastic Search

Then configures the .NET Web API to use Elastic search. To do this, he opens the appsettings.json file and adds the following section:

"ElasticSettings": {
  "Url": "http://localhost:9200",
  "DefaultIndex": "users",
  "Username": "",
  "Password": ""
}

This section defines the configuration settings for Elastic search. The Url property specifies the URL of the Elastic search instance. The DefaultIndex property specifies the default index that will be used for all Elastic search operations. The Username and Password properties can be used to specify the credentials for authenticating with Elastic search, although in this case they are left blank because security is disabled.

Next, the let’s open the Program.cs file and adds the following code:

builder.Services.Configure<ElasticSettings>(builder.Configuration.GetSection("ElasticSettings"));
builder.Services.AddSingleton<IElasticService, ElasticService>();

The first line of code registers the ElasticSettings class with the .NET Core dependency injection container. This makes the configuration settings for Elastic search available to the ElasticService class. The second line of code registers the ElasticService class with the dependency injection container as a singleton. This means that only one instance of the ElasticService class will be created for the lifetime of the application.

Creating an Elastic Search Controller

Finally, create a new controller named UsersController. This controller will provide endpoints for performing CRUD operations on the users stored in Elastic search.

[ApiController]
[Route("[controller]")]
public class UsersController : ControllerBase
{
  private readonly ILogger<UsersController> _logger;
  private readonly IElasticService _elasticService;

  public UsersController(ILogger<UsersController> logger, IElasticService elasticService)
  {
    _logger = logger;
    _elasticService = elasticService;
  }
  [HttpPost("create-index")]
  public async Task<IActionResult> CreateIndex(string indexName)
  {
    await _elasticService.CreateIndexIfNotExistsAsync(indexName);
    return Ok($"Index {indexName} created or already exists.");
  }
  [HttpPost("add-user")]
  public async Task<IActionResult> AddUser([FromBody] User user)
  {
    var result = await _elasticService.AddOrUpdate(user);
    return result ? Ok("User added or updated successfully.") : StatusCode(500, "Error adding or updating user.");
  }
  [HttpPost("update-user")]
  public async Task<IActionResult> UpdateUser([FromBody] User user)
  {
    var result = await _elasticService.AddOrUpdate(user);
    return result ? Ok("User added or updated successfully.") : StatusCode(500, "Error adding or updating user.");
  }
  [HttpGet("get-user/{key}")]
  public async Task<IActionResult> GetUser(string key)
  {
    var user = await _elasticService.Get(key);
    return user != null ? Ok(user) : NotFound("User not found.");
  }
  [HttpGet("get-all-users")]
  public async Task<IActionResult> GetAllUsers()
  {
    var users = await _elasticService.GetAll();
    return users != null ? Ok(users) : StatusCode(500, "Error retrieving users.");
  }
  [HttpDelete("delete-user/{key}")]
  public async Task<IActionResult> DeleteUser(string key)
  {
    var result = await _elasticService.Remove(key);
    return result ? Ok("User deleted successfully.") : StatusCode(500, "Error deleting user.");
  }
}

The UsersController class defines five endpoints:

• CreateIndex: This endpoint creates an index in Elastic search. It takes the name of the index as a parameter.
• AddUser: This endpoint adds a user to Elastic search. It takes a User object as a parameter.
• UpdateUser: This endpoint updates a user in Elastic search. It takes a User object as a parameter.
• GetUser: This endpoint retrieves a user from Elastic search by its key. It takes the key as a parameter.
• GetAllUsers: This endpoint retrieves all users from Elastic search.
• DeleteUser: This endpoint removes a user from Elastic search by its key. It takes the key as a parameter.

Each of these endpoints uses the _elasticService field to interact with Elastic search.

We explored the process of integrating ElasticSearch with a web API. We learned how to set up ElasticSearch and Kibana locally using Docker Compose, connect to ElasticSearch from the web API, and create a simple CRUD operation. ElasticSearch offers several benefits, such as high-performance search, real-time search, full-text search, faceting, geolocation search, analytics capabilities, ease of use, scalability, reliability, and open-source nature. By leveraging these benefits, developers can build powerful and efficient search and analytics applications.