ASP.NET Core Series 02: Log Centralization with using NLog and the GrayLog

Hi, ASP.NET Core lovers!

In this second part of my ASP.NET Core series, we will handle the log operations that we want to keep in the application, and then push the log messages to the GrayLog using NLog library.

It’s possible, that in a lot of applications that we have developed on the traditional .NET Framework, we will have used the “NLog” and “NLog.Targets.Gelf” libraries to push log messages to the GrayLog. I’m working hard these days on the ASP.NET Core as I mentioned in my previous post titled “ASP.NET Core Series 01: Building RESTful API with Dapper and Deploy to Azure App Services“. We are currently in a transformation stage at the company I’m currently working at, and we are developing new APIs on the ASP.NET Core. We have also performed centralized logging infrastructure in our existing .NET Framework applications using the GrayLog.

So, why?

Centralized logging is especially important for the end users to be happy. Also, our other goal is to improve system performance continuously and become aware of errors as quickly as possible.

Anyway, when I was developing a project on the ASP.NET Core, I noticed that NLog‘s “NLog.Targets.Gelf” package had not been transformed to ASP.NET Core yet (or, I could not find it). Then, I wanted to make a contribution for those who want to use GrayLog on the ASP.NET Core. Therefore, I got the “NLog.Targets.Gelf” package’s source code from GitHub and I transformed that to ASP.NET Core. Then, I published the new package on the NuGet server called “NLog.Web.AspNetCore.Targets.Gelf“.

Well, Why Do We Have a Centralized Log Infrastructure?

In today’s technological age, we have dynamic systems that are rapidly changing and evolving rather than fixed systems. However, we also have microservices that are distributed and autonomous, in addition to dynamically scalable systems. We have to know what’s going on in the infrastructure to be able to provide continuity to these evolving systems. In this way, it’s possible to follow changes in the application, improve system performance continuously and we can take quick actions by keeping track of faults. Especially, if we have a distributed system, and we build the log infrastructure on a central platform, it will be much easier to monitor from a single point when any action occurs within a whole system.

There are a lot of applications to apply centralized logging operations such as Splunk, GrayLog, Logstash and Logwatch. In this article, we will handle the log operations using NLog library, and then push the log messages to the GrayLog.

Why GrayLog?

• First of all, it’s an open source log management solution.
• It supports many inputs such as “Syslog, GELF, TCP, UDP, AMQP“.
• It stores the messages on ElasticSearch, which makes the quick search possible on archives.
• It uses MongoDB for statistics processing.
• It’s possible to classify the logs and perform graph operations.
• It’s also easy to configure proactive alerts when something needs our attention.

and more…

GrayLog on Docker

We will run the GrayLog on Docker using following “docker-compose.yml” file.

version: '2'
services:
  some-mongo:
    image: "mongo:3"
  some-elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
  graylog:
    image: graylog2/server:2.1.1-1
    environment:
      GRAYLOG_PASSWORD_SECRET: somepasswordpepper
      GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      GRAYLOG_WEB_ENDPOINT_URI: http://192.168.99.100:9000/api
    links:
      - some-mongo:mongo
      - some-elasticsearch:elasticsearch
    ports:
      - "9000:9000"
      - "12201/udp:12201/udp"
      - "1514/udp:1514/udp"

Let’s create a new folder called “aspnetcore-graylog-sample-with-nlog“, and then copy “docker-compose.yml” file to this folder. After this, we have to run following command using any command tool.

docker-compose up

Now, we can reach GrayLog UI from this URL: “http://192.168.99.100:9000“.

NOTE: Username and Password are: “admin“.

The GrayLog is almost ready. Only one thing left to do so that we can keep the logs on the GrayLog using NLog library. As I said before, the GrayLog supports many inputs such as “GELF“. Now, let’s open “Inputs” page with follow “System / Inputs > Inputs” path on the GrayLog UI.

In here, we need to select “GELF UDP” option from “Select input” drop-down menu, and then click the “Launch new input” button. After this, the “GELF UDP” creation page will be opened, and we need to just fill the “Title” field. Now, the “GELF UDP” input is ready to listening on “12201” port.

ASP.NET Core NLog Implementation

We created the “aspnetcore-graylog-sample-with-nlog” folder before running the GrayLog on the Docker. Now, let’s create an “ASP.NET Core MVC” project in the “aspnetcore-graylog-sample-with-nlog” folder as the following command.

dotnet new mvc

The Visual Studio Code will be opened, after using “code .” command.

First, we need to include a few libraries in the project from the NuGet server. These “NLog.Web.AspNetCore” and “NLog.Web.AspNetCore.Targets.Gelf” libraries are required to push the log messages on the GrayLog2.

After including the packages to the project, copy the following “nlog.config” file in the root folder of the project.

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      throwExceptions="false"
      internalLogLevel="Off"
      internalLogFile="c:\temp\internal-nlog.txt">
  <extensions>
    <add assembly="NLog.Web.AspNetCore"/>
    <add assembly="NLog.Web.AspNetCore.Targets.Gelf"/>
  </extensions>
  <targets>
    <target xsi:type="File" name="errorFile" filename="C:\@Logs\${shortdate}-${level}-AspNetCoreTest.txt" layout="${longdate}|${level:upperCase=true}|${logger}|${aspnet-Request-Method}|url: ${aspnet-Request-Url}${aspnet-Request-QueryString}|${message}" concurrentWrites="false" />
    <target xsi:type="Gelf" name="graylog" endpoint="udp://192.168.99.100:12201" facility="console-runner" SendLastFormatParameter="true" />
  </targets>
  <rules>
    <logger name="*" minlevel="Error" writeTo="errorFile, graylog" />
  </rules>
</nlog>

We have two targets in the “nlog.config” file. The first target is to log the message to the specified file, the second target is to push the log messages to the GrayLog2.

Now, let’s add a few of lines of code in the “Startup.cs” for NLog as shown below.

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    loggerFactory.AddNLog();
    app.AddNLogWeb();

    env.ConfigureNLog("nlog.config");

    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseBrowserLink();
    }
    else
    {
        app.UseExceptionHandler("/Home/Error");
    }

    app.UseStaticFiles();

    app.UseMvc(routes =>
    {
        routes.MapRoute(
            name: "default",
            template: "{controller=Home}/{action=Index}/{id?}");
    });
}

We configured in the above code block to use NLog as a logger. Then we specified the relative path of “nlog.config“. That’s all.

Let’s create a log in “HomeController” as shown below:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace aspnetcore_graylog_sample_with_nlog.Controllers
{
    public class HomeController : Controller
    {
        private ILogger _logger;

        public HomeController(ILogger logger)
        {
            _logger = logger;
        }
        
        public IActionResult Index()
        {
            _logger.LogError("Something happened.");

            return View();
        }
    }
}

“ILogger” will be injected from the constructor. If we look at the index action method, we kept an error log message for testing.

Now, let’s run the project with the “dotnet run” command on terminal. After running the project, we need to reach the project via the “http://localhost:5000” URL. In this way, index action method will be executing, and then the logger will be keeping an error message in the specified “C:\@Logs” path as shown below.

Also, we can see the same error log message in the GrayLog. For this, we have to follow the “System / Inputs > Inputs > Show received messages” path on the GrayLog UI.

Conclusion

We pushed the log messages to the GrayLog2 using NLog and NLog’s GELF (that I transformed for .NET Core) target. This push operation was our main focus point in this article. Also, I wanted to remind you of the importance of a centralized log management system for an enterprise-level project.

See you in the next ASP.NET Core article series.

https://github.com/GokGokalp/aspnetcore-graylog-sample-with-nlog/tree/master
https://github.com/GokGokalp/NLog.Web.AspNetCore.Targets.Gelf

Some References

https://github.com/Graylog2/graylog-docker
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging