Exploring Datomic for Audit Trails

Ackerley Tng

Background

Day of Datomic Cloud Workshop at Strange Loop 2019
- That was my introduction to Datomic
Many of the apps we build at work have a requirement for audit trails
- Who changed this entity in the database?
- When was it changed in the database?
- Approvals - when was it approved?
- What changed between these two dates?
This talk explains what I've explored using a sample problem

Setting up Datomic Starter for exploration

Hop over to https://www.datomic.com/get-datomic.html (requires free sign up)
Click the downloads tab and download the latest zip
Start datomic with no persistent storage

bin/run -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic:mem://hello

Setting up Datomic client

Add com.datomic/client-pro {:mvn/version "0.9.41"} to your deps.edn

Motivating Problem

Want to build a system to manage firewall rules
Firewall rejects all connections by default
Users request to allow traffic through, for selected IPv4 ranges and ports
Need to track who requested any changes to firewall rules

Data Model (conventional)

Firewall Rule Entry
- Name
- Description
- Source IP Range
- Destination IP Range
- Destination Port
Users
- UUID (from SSO service)

Modelling Requester Info…?

`name`	…	`src_ip_range`	`dst_ip_range`	`port`	`requester`
`magical-unicorn`	…	`192.168.1.0/24`	`192.168.50.0/24`	`443`	`<alice>`
`mutant-reindeer`	…	`192.168.2.0/24`	`192.168.51.0/24`	`8443`	`<bob>`
`magical-unicorn`	…	`192.168.1.0/24`	`192.168.50.0/24`	`80`	`<carol>`
		…	…

In a traditional database, I would probably have modelled this concept of requester as a field in the firewall_rule_entry table, as a foreign key into the user table. I would perhaps have 1 row per state of each firewall rule entry, and every change users make would be added as a new row in the table, and the requester field would be updated with the person who requested this change.

As you can probably tell, this is starting to be a mess because

To figure out the latest state of the firewall, I would have to pull the latest version of each rule with a unique name.
To determine what changed with each entry, I would have to do some kind of diff with the previous entry

Another option would be to do event sourcing and store all the changes requested and cache the eventual outcome, and then we would have to code up all of that.

This is the part that I really loved about exploring Datomic, because exploring Datomic allowed me to rethink the meaning of attaching a requester to a firewall rule entry.

Reified Transactions

Transactions are themselves entities in Datomic
- Can attach attribute to every transaction
The requester is a property of the change to the firewall rule entry and not the entry itself
Datomic automatically stores the transaction time of every transaction

Datomic Facts

[entity attribute value transaction added?]

`entity`	firewall-rule-entry
`attribute`	:firewall/name
`value`	"magical-unicorn"
`transaction`	internal reference
`added?`	asserted or retracted

Summary

Reified transactions
Built-in tracking of transaction time
Convenient d/history, d/since, d/as-of functions

Thanks for listening!

Slides and code available at

https://github.com/ackerleytng/datomic-for-audit-trails-talk