Centralized Logging: Loki + Grafana + Promtail Part 1

Having Splunk experience, centralized logging is a must. As my lab grows, I wan't better visibility into the services I’m running, and I was itching to bring a centralized logging solution into the mix. You can lay the person off from the enterprise, but you can't take the enterprise out of the person 😀

After some research, I chose Loki as the log collector, Grafana as the search and visualization layer, and Promtail agents for log shipping. I have worked with Splunk in previous cycles and deployed it across four data centers for internal log collection and alerting, so I understand the value of getting this in place early in any implementation. You also cannot beat the price.

This will be a multi part blog series, and I am starting with the Grafana installation, followed by Loki on the same VM. While you could install Loki first, I prefer to begin with web UI based services. I like to see the application come up cleanly, confirm that I can log in as an administrator, and verify that everything works before moving on to the next component. We will then install Loki on the same VM, although in production you would probably separate the two.

Part 1 covers installing Grafana on AlmaLinux 9.7, then Loki. I will save the Promtail agent log shipping for another post. I chose AlmaLinux because it allows me to deploy quickly without dependency or versioning issues, and in practice it resulted in a smooth, predictable install.

cat <<EOF > /etc/yum.repos.d/grafana.repo
[grafana]
name=Grafana OSS
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
EOF

dnf install -y grafana
systemctl enable --now grafana-server

Add firewalld rules:

firewall-cmd --permanent --add-port=3000/tcp
firewall-cmd --permanent --add-port=3100/tcp
firewall-cmd --permanent --add-port=1514/udp
firewall-cmd --permanent --add-port=1514/tcp
firewall-cmd --reload

Grafana should be available at:

http://<host>:3000

admin / admin

Loki install, we will be pretty straightforward

Logs are pushed to Loki by agents running on servers or clusters (Promtail in my case) Loki provides lightweight, label-based log storage that lets Grafana show what happened and why,

Install Loki:

dnf install -y loki

Enable and start Loki (no config edits)

systemctl enable --now loki

Immediately inspect logs:

journalctl -u loki -n 50 --no-pager

If Loki fails here, stop and debug Loki alone.

Verify Loki is alive (before Grafana)

systemctl status loki
ss -tlnp | grep 3100
curl http://localhost:3100/ready

Expected output:

ready

SELinux: allow Grafana → Loki

RHEL-based systems block this by default.

Enable the required boolean:

setsebool -P httpd_can_network_connect 1

• SELinux remains Enforcing

Add Loki to Grafana:

In the Grafana UI:

1. Settings → Data Sources → Add data source

2. Select Loki

3. URL:

http://localhost:3100

That’s it. You should now have Grafana talking to Loki at this point.

I’ll add a follow up post later covering installing agents on your servers, basic and advanced visualizations, and how to shape and refine the results.

This also puts Prometheus on the white board.

Thanks for reading! -Christian