Audit Log Service-Evaluating Different Approaches

Background

Audit Log should be capable of storing complete track records of your system’s operations. These audit logs can give an administrator invaluable insight into what behavior is normal and what behavior isn’t. Also, this can be useful in identifying whether a system component is misconfigured or likely to fail.

There are multiple reasons why we need an audit log are as follows -

• For better security.
• For faster troubleshooting.
• For Compliance related things.
• For IPO readiness.
• For better business insights.

Approaches

Let’s deep dive into various approaches for audit log service. There are a couple of approaches, and each has its own pros & cons some of which are listed below -

1.Custom Audit Log Service

1. Creating a separate service that will create audit logs whenever any updates happen.
2. This service will be plugged into all core services wherever updates are happening.
3. It must be in the same transaction boundary — which means any failure to user update (say) should roll back audit logs.
4. Database schema looks something as- KEY_TYPE: users, SUB_KEY_TYPE: name (column or a logical collection of columns), KEY_ID: users.id, FROM_DATA: original data in serialized form, TO_DATA: changed data in serialized form.

pros

• Custom metadata can be added.
• The developer has full freedom to design in its own way — in such a way has been shown above.

cons

• Tight coupling. Lots of overhead involves in maintaining the code.
• Transactional issues might be frequent if not handled carefully.
• Writing audit logs will increase the number of executed statements for each update, hence increasing the response time of the application.
• Changes done directly from DB cannot be tracked.

2. Spring Boot — Javers

• Annotation-based over repository class.
• It doesn’t create new connections to the database, reuses the same connections.
• Audit data is committed or rolled back along with application data in the same transaction.

1. Adding Javers dependency into your SpringBoot Application will automatically table schema defined as below

• jv_snapshot: contains logs data, jv_global: contains entity information for which data is changed, jv_commit: who performed the updates.
• jv_global_id — ( type_name: stores the table name / object)
• jv_snapshot — (changed_properties: contains column_name that is changed, state: stores the current state of a table object.

pros

• Best performant when the system is read-heavy.
• It can be easily integrated with the SpringBoot Application.
• It requires almost zero configuration changes.
• Custom Metadata Can be added.
• Easy query to get snapshots change of selected entities.

cons

• Writing audit logs increases the number of executed statements for each update, hence increasing the response time of the application.
• Changes done directly from DB cannot be tracked.

3. CDC — Change Data Capture

Design Flow diagram would look something as -

1. There are quite a few variants of CDC available in the market such as Debezium from Redhat, Netflix’s DB Log, and LinkedIn’s Brooklyn.
2. Debezium can write data only in Kafka — at least that is the primary producer it supports. On the other hand, MD supports a variety of producers including Kafka.
3. Any update that happens on the database, creates a transaction log irrespective of whether it is created from a DB query or from application. Thus, data change events can be produced.

pros

• Best standard practices to capture audit logs when the system is write-intensive.

cons

• Requires time to explore.
• Maintenance of separate stack that is Debezium, Kafka, etc.
• Joining logic — combining logs is a challenge.
• Kafka Stream would be required. Another overhead.

4. EFK– Elastic Search, Fluent.d, & Kibana

1. EFK is a suite of tools combining Elasticsearch, Fluentd, and Kibana to manage logs.
2. Fluentd will collect the logs and send them to Elasticsearch. This latter will receive the logs and save them on its database. Kibana will fetch the logs from Elasticsearch and display them on a nice web app.
3. This could be easily installed on Kubernetes.
4. Design flow would look something as -

PROS

• This is the best practice as far as application logs are considered.

CONS

• Capturing DB changelogs is a challenge here.

Inference

While designing the Audit Log Service, we evaluated the above approaches and choose the “Spring Boot — Javers” approach. The selection of the right approach will largely depend upon the nature of the application. Our application is a Spring-based Microservice, traffic lies under the Medium bucket plus we needed faster development.

Note: Before designing the Audit Log, it is very important to get the requirements from the ground team and know the existing problem that they are facing in absence of an audit feature. This will help to identify extra metadata information like Ip Address, browser detail, user_meta_info, etc. Such metadata needs to be captured and stored along with regular audit logs.