OKD-OpenShift-Safe-Harbor

Harbor Install on AlmaLinux 9 HTTP Only

I was playing with my new OKD cluster and getting some workloads running. It was all simple enough, but I really don’t like spending time doing things that are not prod-like.

I know that most shops are probably not trusting upstream registries , additionally OpenShift does not allow containers to run as root by default, and many upstream images assume they are root. I wanted a way to rebuild upstream container images so they satisfy OKD/OpenShift security requirements, store them locally in Harbor, and deploy them reliably without pulling from upstream registries at runtime.

So I went on the hunt for a enterprise container registry and landed on Harbor.

This document details my installation of Harbor. Because I have not yet implemented a Certificate Authority in my environment and I’m being deliberate about where I spend my time this installation is configured to use HTTP rather than HTTPS.

I initially attempted to use self-signed certificates, but consistently ran into certificate-related errors across multiple components. While switching Harbor to HTTP introduced a different set of issues, those problems were at least predictable and debuggable, allowing me to make forward progress.

For now, the goal of this deployment is functionality and understanding, not production-grade TLS. HTTPS will be introduced later once a proper CA strategy is in place.

The Goal:

  • Install harbor http

  • Pull images from dockerhub

  • Push them into the "golden" Project

  • Configure OKD to pull from harbor

  • Test a non-root image (grafana)

  • Test an image that has to be rebuilt to run within OKD's constraints.

Environment Assumptions
  • OS: AlmaLinux 9
  • Hostname: vv-harbor-01.vv-int.io
  • IP: 172.26.2.10
  • Install path: /opt/harbor
  • Docker Engine (not Podman)
  • Lab / internal environment
Base System Prep
dnf update -y
dnf install -y curl wget tar vim firewalld dnf-plugins-core
systemctl enable --now firewalld

Open required ports (HTTP only):

firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --reload
Install Docker Engine

Add Docker repo:

dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install Docker:

dnf install -y docker-ce docker-ce-cli

Enable and start Docker:

systemctl enable --now docker

Verify:

docker version

Install Docker Compose v2

curl -L https://github.com/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 \
  -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

docker compose version

Download Harbor Offline Installer

cd /opt
wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz
tar xvf harbor-offline-installer-v2.10.0.tgz
cd harbor

CRITICAL: Fix Name Resolution Before Installing Harbor

Hard-pin the Harbor hostname in /etc/hosts (REQUIRED)

This is the authoritative fix we actually used.

Edit /etc/hosts:

vi /etc/hosts

Add this exact line at the top (above other entries), whatever hostname you choose:s

172.26.2.10 vv-harbor-01.vv-int.io

⚠️ Do NOT point this hostname to 127.0.0.1 or ::1.

Verify:

getent hosts vv-harbor-01.vv-int.io

getent hosts vv-harbor-01.vv-int.io

Expected output (ONE line only):

172.26.2.10 vv-harbor-01.vv-int.io

172.26.2.10 vv-harbor-01.vv-int.io

If you see ::1 or fe80::, stop and fix this before continuing.

Disable mDNS / Avahi resolution precedence (Strongly Recommended)

Even with /etc/hosts pinned, mDNS can reintroduce IPv6 resolution in other contexts.

Edit /etc/nsswitch.conf:

vi /etc/nsswitch.conf

Change:

hosts: files mdns4_minimal [NOTFOUND=return] dns

To:

hosts: files dns

This makes DNS and /etc/hosts authoritative.

Re-verify:

getent hosts vv-harbor-01.vv-int.io

It must still return IPv4 only.

Configure Harbor (HTTP ONLY)

cp harbor.yml.tmpl harbor.yml

Edit harbor.yml:

hostname: vv-harbor-01.vv-int.io http: port: 80

HTTPS MUST NOT EXIST AT ALL
(commenting incorrectly can still break nginx)
https:
port: 443
hostname: vv-harbor-01.vv-int.io

http:
  port: 80

# HTTPS MUST NOT EXIST AT ALL
# (commenting incorrectly can still break nginx)
# https:
#   port: 443

harbor_admin_password: Harbor12345

database:
  password: HarborDB12345

⚠️ Do not leave an empty https: block.

Generate Harbor Configuration (MANDATORY)

./prepare

Expected output includes:

Generated configuration file: ./docker-compose.yml
Generated configuration file: ./common/config/nginx/nginx.conf

If prepare fails → do not proceed.

Start Harbor

docker compose up -d

Verify all services:

docker compose ps

All must be Up (healthy).

Verify Harbor is Listening (HTTP)

ss -lntp | grep :80

Expected:

LISTEN 0.0.0.0:80 docker-proxy

CRITICAL: Docker Behavior with HTTP Registries

Docker ALWAYS assumes HTTPS on port 443 unless told otherwise.

Therefore, HTTP registries REQUIRE insecure-registries.

Configure Docker

vi /etc/docker/daemon.json
{
  "insecure-registries": [
    "vv-harbor-01.vv-int.io"
  ]
}

Restart Docker:

systemctl restart docker

Verify:

docker info | grep -A5 -i insecure

You MUST see the registry listed.

⚠️ Restarting Docker stops Harbor.

Bring it back:

cd /opt/harbor
docker compose up -d

Verify Harbor UI

Use the hostname, not IP:

curl -H "Host: vv-harbor-01.vv-int.io" http://127.0.0.1

Browser:

http://vv-harbor-01.vv-int.io

Harbor does not serve a default vhost.

Add Docker Hub Registry

UI → Administration → Registries → New Endpoint

  • Provider: Docker Hub

  • Name: dockerhub

  • Auth: none (public images)

  • Test Connection: OK

You can Create two types of Projects in Harbor

  • Proxy Cache* = Harbor acting as a mirror in front of someone else’s registry*

  • Project = Harbor being the source of truth for images you own

We will be creating a standard Project, I like truth.

Projects → New Project

  • Name: golden

  • Type: standard

  • Provider: dockerhub

  • Access: Private

AUTHENTICATION (EXPECTED FAILURE)

First pull will FAIL with:

no basic auth credentials

This is correct behavior.

Login:

docker login vv-harbor-01.vv-int.io

Use Harbor admin credentials.

Test Image Pull

docker pull vv-harbor-01.vv-int.io/dockerhub-apps/library/nginx:1.25

Expected:

  • Layers download

  • Image cached locally

  • Image visible in Harbor UI

Restart Rules

If Docker is restarted:

systemctl restart docker
cd /opt/harbor
docker compose up -d

Harbor does not auto-start.

Known Failure Modes & Fixes (Summary)

Symptom Root Cause Fix
::1 / fe80:: mDNS + IPv6 hosts: files dns
TLS unknown authority Docker token HTTPS Use HTTP first
Harbor UI blank Host header mismatch Use hostname
Pull access denied Private project docker login
Harbor down after reboot Docker restart docker compose up -d

  • Proxy Cache = Harbor acting as a mirror in front of someone else’s registry

  • Project = Harbor being the source of truth for images you own---

Moving on to Harbor Config.

This is basically what we will be doing below

Upstream Image
    ↓
Harbor Host (Docker/Podman)
    ↓  (optional modification)
Golden Harbor Project
    ↓
Robot Account (pull)
    ↓
OKD Worker Nodes
    ↓
Running Pod + Route

I also want to set the project to private because I already punted on the TLS and CA. I really wanted to force "auth correctess"

High-Level Workflow

  • Pull the Grafana image from Docker Hub into Harbor and place it in the golden image project

  • Configure OKD to pull images from Harbor’s private golden repository and verify that Grafana runs successfully

  • Repeat the process for phpMyAdmin, demonstrating that it fails to run due to default security constraints (e.g., root user)

  • Modify and rebuild the phpMyAdmin image to comply with OKD requirements

  • Publish the updated phpMyAdmin image to the golden repository

  • Pull the corrected image into OKD and verify that it runs successfully

Phase 1 — Create the Golden Project in Harbor

UI steps

  • Log into Harbor

  • Create project:

Name: golden
Visibility: Private

Why private?

  • Forces auth correctness

  • Prevents accidental anonymous pulls

  • Matches real org security posture, I think anyway 😀

Phase 2 — Create a Robot Account

Inside golden project → Robot Accounts

Create robot:

Name: robot$golden-pull

Name: robot$golden-pull

Permissions:

  • Pull repository

  • Push repository

Save the generated token — this is the password.

Note: I gave both pull and push to this one robot. I assume in prod that these accounts would be sperate pull and push. OKD would use the pull for deploying images, this reduces the attack surfact if your note was comprimised

Log into Harbor from the Harbor Host

On the Harbor VM:

docker login vv-harbor-01.vv-int.io

Use:

  • Username: admin

  • Password: Harbor admin password

Why admin here?

  • You are building and pushing images

  • Robot accounts are for consumers (OKD), not builders

Pull the Grafana image from Docker Hub

Pull Upstream phpMyAdmin Image

From the Harbor host:

docker pull grafana/grafana:12.4.0-21230963995

This confirms:

  • Internet access

  • Docker is functional

  • You have a known-good base image

Next we tag the image:

docker tag grafana/grafana:12.4.0-21230963995 \
  vv-harbor-01.vv-int.io/golden/grafana:12.4.0

Then we push it to the golden repo:

docker push vv-harbor-01.vv-int.io/golden/grafana:12.4.0

Phase 4 — Pull Upstream phpMyAdmin Image

From the Harbor host:

docker pull phpmyadmin/phpmyadmin:5.2.1

This confirms:

  • Internet access

  • Docker is functional

  • You have a known-good base image

Phase 5 — (Optional but Real) Harden phpMyAdmin for OKD Compatibility

Why phpMyAdmin initially failed on OKD

Out of the box, the upstream phpmyadmin/phpmyadmin image assumes:

It can:

  • write to /etc/phpmyadmin

  • bind to privileged port 80

  • run as a predictable user/group

Kubernetes/OpenShift reality:

  • Random UID

  • Read-only root filesystem in many cases

  • No privileged ports unless explicitly allowed

  • SCC does not allow root assumptions

That’s why you saw errors like:

Permission denied: /etc/phpmyadmin/config.secret.inc.php
make_sock: could not bind to address 0.0.0.0:80

These are expected failures, not misconfiguration.

Phase 6 — Create a Golden OKD-Safe Image (phpMyAdmin)

Why we do this

This is the entire point of a golden registry:

  • Fix the image once

  • Never debug it again

  • Every cluster deploy behaves the same

    • *

Step 6.1 — Create a Working Directory

On the Harbor host:

mkdir -p ~/phpmyadmin-golden
cd ~/phpmyadmin-golden

Why:

  • Keeps Dockerfiles auditable

  • Makes rebuilds repeatable

  • Matches enterprise image pipelines

    • *

Step 6.2 — Create the Dockerfile

Create Dockerfile:

FROM phpmyadmin/phpmyadmin:5.2.1

# Switch Apache to non-privileged port
RUN sed -i 's/Listen 80/Listen 8080/' /etc/apache2/ports.conf && \
    sed -i 's/:80/:8080/g' /etc/apache2/sites-enabled/000-default.conf

# Ensure writable dirs for random UID
RUN mkdir -p /etc/phpmyadmin && \
    chgrp -R 0 /etc/phpmyadmin && \
    chmod -R g=u /etc/phpmyadmin

# Expose non-privileged port
EXPOSE 8080

# Do NOT force USER — let OpenShift inject one

Key decisions explained

  • 8080 instead of 80

    Required for non-root SCC

  • chgrp 0 + g=u

    OpenShift random UID is always in group 0

  • No USER directive

    OpenShift sets it dynamically

This is canonical OKD hardening.

Step 6.3 — Build the Golden Image

docker build -t vv-harbor-01.vv-int.io/golden/phpmyadmin:5.2.1-okd .

Verify:

docker images | grep phpmyadmin

Phase 7 — Push Golden Image to Harbor

docker push vv-harbor-01.vv-int.io/golden/phpmyadmin:5.2.1-okd

Expected output:

  • Layers push successfully

  • Digest printed

  • Image visible in Harbor UI under golden/phpmyadmin

At this point:

  • Harbor owns the image

  • Upstream is no longer needed

  • Version is pinned

    • *

Phase 8 — Create Pull Secret in OKD (Robot Account)

On an OKD node or admin workstation:

oc create secret docker-registry harbor-pull \
  --docker-server=vv-harbor-01.vv-int.io \
  --docker-username=robot$golden-pull \
  --docker-password='<ROBOT_TOKEN>' \
  [email protected] \
  -n harbor-test

Attach it to the default service account:

oc secrets link default harbor-pull --for=pull -n harbor-test

Verify:

oc describe sa default -n harbor-test

You must see:

Image pull secrets: harbor-pull

Phase 9 — Deploy phpMyAdmin in OKD

Step 9.1 — Create Deployment

oc create deployment phpmyadmin \
  --image=vv-harbor-01.vv-int.io/golden/phpmyadmin:5.2.1-okd \
  -n harbor-test

Step 9.2 — Expose Service

oc expose deployment/phpmyadmin \
  --port=8080 \
  -n harbor-test

Step 9.3 — Create Route

oc expose svc/phpmyadmin -n harbor-test

Check:

oc get route phpmyadmin -n harbor-test

Open the URL:

http://phpmyadmin-harbor-test.apps.okd.vv-int.io

✅ Web UI loads
✅ No permission errors
✅ No crashes
✅ Fully OKD-compatible

Phase 10 — Verification & Debug Commands (Keep These)

Confirm image source

oc get pod -n harbor-test -l deployment=phpmyadmin \
  -o jsonpath='{.items[0].spec.containers[0].image}'

Should point to:

vv-harbor-01.vv-int.io/golden/phpmyadmin@sha256:...

Check logs

oc logs deployment/phpmyadmin -n harbor-test

Healthy logs look like:

Apache configured -- resuming normal operations

Node-level pull test (ultimate truth)

oc debug node/<worker-node>
chroot /host
podman pull vv-harbor-01.vv-int.io/golden/phpmyadmin:5.2.1-okd

If this works, everything works.

Why This Is the “Enterprise” Way

This approach gives you:

✔ Version pinning
✔ No upstream dependency
✔ OKD-safe images
✔ Repeatable builds
✔ Clean security boundaries
✔ No proxy-cache weirdness
✔ No “why did it work yesterday” surprises

This is exactly how mature orgs run Harbor + OpenShift.

Mental Model to Keep

  • Harbor proxy cache → convenience, not control

  • Golden images → control, auditability, reliability

  • If OKD breaks it once, fix the image forever

    • *

If you want next steps, we can:

  • Turn this into a Makefile-based image pipeline

  • Add Trivy scanning + admission control

  • Build a Grafana / Adminer / Redis / Postgres golden suite

  • Convert this into a runbook.md

Previous Post Next Post