Open in app

Sign in

Write

Sign in

Install Nomad Cluster

Kidong Lee
9 min readApr 8, 2021
Photo by Alex Ovs on Unsplash

Kubernetes is a popular container orchestrator nowadays. I have also proposed the concept of private cloud platform based on Kubernetes, and I have implemented a multi-tenant data platform for this concept. From my experiences of building my platform , it is great to run stateless applications on Kubernetes, but I have noticed there are many things to take care of for stateful applications running on Kuberntes. Most of my data platform components running on Kubernetes are statefulset, things to be considered are, for instance, Pod QoS Class rules against statefulset pod being killed, careful Rollout, backup of PVs for DR periodically. I think, Kubernetes is too complicated and in particular, not a good choice as orchestrator for stateful applications at the moment. I need more simpler orchestrator which supports stateful applications which can be run on it in a very stable way. Nomad is a workload orchestrator from Hashicorp. I am considering it as an alternative to Kubernetes, in particular for the stateful components of my platform.

In this post, I am going to talk about installing Nomad Cluster with Consul first. You will see how to deploy Nomad along with Consul typically and make applications running on Nomad accessible each other with service name provided by Consul Service Discovery through BIND.

Typical Nomad Deployment with Consul

Nomad Cluster can be deployed along with Consul Cluster which runs as a service discovery. There is two agent type of Nomad, server and client. Nomad Server has the role of cluster management, and Nomad Client has the role of executing tasks. Nomad Cluster consists of 3–5 Nomad Servers typically, and Nomad Clients can be run on many nodes. As seen in the above picture, Consul can be deployed on the same node where Nomad is installed.

Prerequisites

It is assumed that Consul Cluster is installed on the same node where Nomad Cluster will be installed.

Nomad Nodes

Let’s assume that the host names of nomad nodes are as follows.

# nomad server nodes.
10.0.0.3    nomad-server-0.nomad.cluster    nomad-server-0
10.0.0.4    nomad-server-1.nomad.cluster    nomad-server-1
10.0.0.5    nomad-server-2.nomad.cluster    nomad-server-2# nomad client nodes.
10.0.0.6    nomad-client-0.nomad.cluster    nomad-client-0
10.0.0.7    nomad-client-1.nomad.cluster    nomad-client-1

Install Nomad Binary

Let’s install nomad binary on all the nodes.

# Install yum-config-manager to manage your repositories.
sudo yum install -y yum-utils;# Use yum-config-manager to add the official HashiCorp Linux repository.
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo;# list nomad with version.
yum --showduplicate list nomad;# install nomad binary.
sudo yum install nomad-1.0.4-1.x86_64 -y;# verify the installation.
nomad;

Allow ports on firewall

Open ports.

sudo firewall-cmd  --add-port={4646,4647,4648}/tcp --permanent
sudo firewall-cmd  --add-port=4648/udp --permanent
sudo firewall-cmd --reload

Enable TLS

To enable TLS in Nomad Cluster, you have to generate certificates.

Install cfssl to generate certificate.

# install cfssl.
sudo yum install go -y;git clone https://github.com/cloudflare/cfssl.git;
cd cfssl;
git checkout tags/v1.5.0;
make;sudo su - ;
cat > /etc/profile.d/cfssl.sh <<EOF
export CFSSL_HOME=/home/opc/cfssl;
export PATH=\${PATH}:\${CFSSL_HOME}/bin;
EOFsudo su - opc;
source /etc/profile;# generate cert and private key.
cfssl print-defaults csr | cfssl gencert -initca - | cfssljson -bare nomad-ca;# create cfssl configuration.
cat <<EOF > cfssl.json
{
  "signing": {
    "default": {
      "expiry": "87600h",
      "usages": ["signing", "key encipherment", "server auth", "client auth"]
    }
  }
}
EOF

Generate certificate for nomad server, client and CLI.

# Generate a certificate for the Nomad server.
echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \
    -hostname="server.global.nomad,localhost,127.0.0.1" - | cfssljson -bare server# Generate a certificate for the Nomad client.
echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -config=cfssl.json \
    -hostname="client.global.nomad,localhost,127.0.0.1" - | cfssljson -bare client# Generate a certificate for the CLI.
echo '{}' | cfssl gencert -ca=nomad-ca.pem -ca-key=nomad-ca-key.pem -profile=client \
    - | cfssljson -bare cli

Create nomad directories on all the nodes.

sudo mkdir -p /etc/nomad.d /export/nomad-data;
sudo chown -R nomad:nomad /etc/nomad.d /export/nomad-data;
sudo chmod 777 /etc/nomad.d /export/nomad-data;

Now, copy the certificates to nomad server nodes.

scp nomad-ca.pem nomad-server-0:/etc/nomad.d;
scp nomad-ca.pem nomad-server-1:/etc/nomad.d;
scp nomad-ca.pem nomad-server-2:/etc/nomad.d;scp server.pem nomad-server-0:/etc/nomad.d;
scp server.pem nomad-server-1:/etc/nomad.d;
scp server.pem nomad-server-2:/etc/nomad.d;scp server-key.pem nomad-server-0:/etc/nomad.d;
scp server-key.pem nomad-server-1:/etc/nomad.d;
scp server-key.pem nomad-server-2:/etc/nomad.d;

Copy the certificates to the nomad client nodes.

scp nomad-ca.pem nomad-client-0:/etc/nomad.d;
scp nomad-ca.pem nomad-client-1:/etc/nomad.d;scp client.pem nomad-client-0:/etc/nomad.d;
scp client.pem nomad-client-1:/etc/nomad.d;scp client-key.pem nomad-client-0:/etc/nomad.d;
scp client-key.pem nomad-client-1:/etc/nomad.d;

Finally, copy the certificates to the all the nodes.

scp nomad-ca.pem nomad-server-0:/etc/nomad.d;
scp nomad-ca.pem nomad-server-1:/etc/nomad.d;
scp nomad-ca.pem nomad-server-2:/etc/nomad.d;
scp nomad-ca.pem nomad-client-0:/etc/nomad.d;
scp nomad-ca.pem nomad-client-1:/etc/nomad.d;scp cli.pem nomad-server-0:/etc/nomad.d;
scp cli.pem nomad-server-1:/etc/nomad.d;
scp cli.pem nomad-server-2:/etc/nomad.d;
scp cli.pem nomad-client-0:/etc/nomad.d;
scp cli.pem nomad-client-1:/etc/nomad.d;scp cli-key.pem nomad-server-0:/etc/nomad.d;
scp cli-key.pem nomad-server-1:/etc/nomad.d;
scp cli-key.pem nomad-server-2:/etc/nomad.d;
scp cli-key.pem nomad-client-0:/etc/nomad.d;
scp cli-key.pem nomad-client-1:/etc/nomad.d;

Create Nomad Server Configuration

Create server configuration on Nomad Servers.

sudo su -;
echo "" > /etc/nomad.d/nomad.hcl;
cat <<EOF > /etc/nomad.d/nomad.hcl
log_level = "INFO"
data_dir = "/export/nomad-data"
server {
  enabled = true
  bootstrap_expect = 3
}
tls {
  http = true
  rpc  = true  ca_file   = "/etc/nomad.d/nomad-ca.pem"
  cert_file = "/etc/nomad.d/server.pem"
  key_file  = "/etc/nomad.d/server-key.pem"  verify_server_hostname = true
  verify_https_client    = true
}
EOF

Create Nomad Client Configuration

Create client configuration on client nodes.

sudo su -;
echo "" > /etc/nomad.d/nomad.hcl;
cat <<EOF > /etc/nomad.d/nomad.hcl
log_level = "INFO"
data_dir = "/export/nomad-data"
client {
  enabled = true
  servers = ["10.0.0.3:4647","10.0.0.4:4647","10.0.0.5:4647"]
}
ports {
  http = 4646
}
tls {
  http = true
  rpc  = true  ca_file   = "/etc/nomad.d/nomad-ca.pem"
  cert_file = "/etc/nomad.d/client.pem"
  key_file  = "/etc/nomad.d/client-key.pem"  verify_server_hostname = true
  verify_https_client    = true
}
EOF

Run Nomad

To run nomad agents, create a service.

sudo su -;
cat <<EOF > /etc/systemd/system/nomad.service
[Unit]
Description=Nomad
Documentation=https://www.nomadproject.io/docs
Wants=network-online.target
After=network-online.target[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=0
StartLimitIntervalSec=10
TasksMax=infinity[Install]
WantedBy=multi-user.target
EOF

And, run agents.

sudo systemctl enable nomad;
sudo systemctl start nomad;
sudo systemctl status nomad;

Check Node Status

To check node status, add the following env.

sudo su -;
cat <<EOF > /etc/profile.d/nomad-cli.sh
export NOMAD_ADDR=https://localhost:4646;
export NOMAD_CACERT=/etc/nomad.d/nomad-ca.pem;
export NOMAD_CLIENT_CERT=/etc/nomad.d/cli.pem;
export NOMAD_CLIENT_KEY=/etc/nomad.d/cli-key.pem;
EOF

And run Nomad CLI to show nodes status.

nomad node status;
ID        DC   Name            Class   Drain  Eligibility  Status
dd258138  dc1  nomad-client-1  <none>  false  eligible     ready
25b3c3dd  dc1  nomad-client-0  <none>  false  eligible     ready

Check DNS with Consul

To get endpoints of nomad servers.

### with dig
dig @127.0.0.1 -p 8600 nomad.service.consul; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.4 <<>> @127.0.0.1 -p 8600 nomad.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8066
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nomad.service.consul.          IN      A;; ANSWER SECTION:
nomad.service.consul.   0       IN      A       10.0.0.3
nomad.service.consul.   0       IN      A       10.0.0.4
nomad.service.consul.   0       IN      A       10.0.0.5;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Apr 01 06:28:13 GMT 2021
;; MSG SIZE  rcvd: 97### with nslookup.
nslookup nomad.service.consul 127.0.0.1 -port=8600
Server:         127.0.0.1
Address:        127.0.0.1#8600Name:   nomad.service.consul
Address: 10.0.0.4
Name:   nomad.service.consul
Address: 10.0.0.3
Name:   nomad.service.consul
Address: 10.0.0.5

Run Example Job

Before running example job, docker has to be installed on nomad client nodes.

# install docker on nomad client nodes.
## remove old version.
sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine -y;
      
## setup repo.
sudo yum install -y yum-utilssudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
 
## install docker engine.
sudo yum install docker-ce docker-ce-cli containerd.io -y;## start docker.
sudo systemctl start docker;

Run example job.

nomad job init;
nomad job run example.nomad;
==> Monitoring evaluation "37ae33bd"
    Evaluation triggered by job "example"
==> Monitoring evaluation "37ae33bd"
    Evaluation within deployment: "708789f6"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "37ae33bd" finished with status "complete"

Forward DNS for Consul Service Discovery

The question is how to access the applications running on Nomad each other with the service name which is registered in Consul service discovery.

To forward DNS for Consul Service Discovery, you can use BIND Name Server. Let’s install BIND.

sudo yum install bind bind-utils -y;

Allow ports.

sudo firewall-cmd  --add-port=53/tcp --permanent
sudo firewall-cmd  --add-port=53/udp --permanent
sudo firewall-cmd --reload

Configure Named on Master.

sudo su -;
echo "" > /etc/named.conf;
cat <<EOF > /etc/named.conf
options {
        listen-on port 53 { 127.0.0.1; 10.0.0.3; };
        # listen-on-v6 port 53 { ::1; };
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query { any; };
        allow-transfer { localhost; 172.31.27.18; };
        recursion yes;
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;
        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";
};
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
zone "nomad.cluster" IN {
        type master;
        file "nomad.cluster.zone";
        allow-update { none; };
};
zone "0.10.in-addr.arpa" IN {
        type master;
        file "0.10.zone";
        allow-update { none; };
};
zone "consul" IN {
        type forward;
        forward only;
        forwarders { 127.0.0.1 port 8600; };
};
controls {
        inet 127.0.0.1 allow { localhost; }
        keys { rndc-key; };
};
key "rndc-key" {
        algorithm hmac-md5;
        secret "DWsZoZ8xRALstyi7w5WWzw==";
};
EOF

Create RNDC Key.

rndc-confgen -a -r /dev/urandom

And, update RNDC Key with generated key, for instance.

key "rndc-key" {
        algorithm hmac-md5;
        secret "DWsZoZ8xRALstyi7w5WWzw==";
};

Take a look at consul zone part in the configuration.

zone "consul" IN {
        type forward;
        forward only;
        forwarders { 127.0.0.1 port 8600; };
};

All the DNS of xxx.xxx.consul will be forwarded to the local Consul which is listening on the port of 8600 .

Let’s create zone files.

# create zone file.
cat <<EOF > /var/named/nomad.cluster.zone
\$TTL 86400
@ IN SOA nomad-server-0.nomad.cluster. root.nomad.cluster. (
        2 ;Serial
        3600 ;Refresh
        1800 ;Retry
        604800 ;Expire
        86400 ;Minimum TTL
)
; Specify our two nameservers
        IN NS nomad-server-0.nomad.cluster.
        IN NS nomad-server-1.nomad.cluster.
; Resolve nameserver hostnames to IP, replace with your two droplet IP addresses.
@ IN A 127.0.0.1
 
nomad-server-0   IN A 10.0.0.3
nomad-server-1   IN A 10.0.0.4
nomad-server-2   IN A 10.0.0.5
nomad-client-0   IN A 10.0.0.6
nomad-client-1   IN A 10.0.0.7
EOF
cat <<EOF > /var/named/0.10.zone
@ IN SOA nomad-server-0.nomad.cluster. root.nomad.cluster. (
        1 ;Serial
        3600 ;Refresh
        1800 ;Retry
        604800 ;Expire
        86400 ;Minimum TTL
)
; Specify our two nameservers
        IN NS nomad-server-0.
        IN NS nomad-server-1.
 
; Define ip -> host.
0.3  IN PTR nomad-server-0.
0.4  IN PTR nomad-server-1.
0.5  IN PTR nomad-server-2.
0.6  IN PTR nomad-client-0.
0.7  IN PTR nomad-client-1.
EOF

Create Named configuration on Client.

sudo su -;
echo "" > /etc/named.conf;
cat <<EOF > /etc/named.conf
options {
        listen-on port 53 { 127.0.0.1; 10.0.0.4; };
        #listen-on-v6 port 53 { ::1; };
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query { any; };
        recursion yes;
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;
        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
        managed-keys-directory "/var/named/dynamic";
};
 
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
zone "nomad.cluster" IN {
        type slave;
        masters { 10.0.0.3; };
        file "nomad.cluster.zone";
};
zone "0.10.in-addr.arpa" IN {
        type slave;
        masters { 10.0.0.3; };
        file "0.10.zone";
};
zone "consul" IN {
        type forward;
        forward only;
        forwarders { 127.0.0.1 port 8600; };
};
EOF

Start Name Server.

# Named check:
named-checkzone nomad.cluster /var/named/nomad.cluster.zone
named-checkzone 0.10.in-addr.arpa /var/named/0.10.zone# restart named.
systemctl enable named
systemctl restart named
rndc reload

Add DNS Server in network script, for instance, /etc/sysconfig/network-scripts/ifcfg-ens3.

...
DNS1=10.0.0.3
DNS2=10.0.0.4
...

And restart network.

sudo systemctl restart network;

Finally, check Consul DNS Interface through BIND.

nslookup nomad.service.consul;
Server:         10.0.0.3
Address:        10.0.0.3#53Non-authoritative answer:
Name:   nomad.service.consul
Address: 10.0.0.3
Name:   nomad.service.consul
Address: 10.0.0.5
Name:   nomad.service.consul
Address: 10.0.0.4

As expected, BIND forwards DNS to Consul Service Discovery. Note that with BIND, All the applications running on Nomad can be accessed each other with the service name registered as a DNS entry in Consul Service Discovery.

I have a plan to write Nomad jobs to run the stateful applications like Trino(formerly PrestoSQL), Hive, Spark, Kafka, MinIO etc. on Nomad. I hope that I have a chance to talk about it in the next posts.

Nomad
Consul
Kubernetes
Bind

--

--

Kidong Lee

Written by Kidong Lee

279 Followers

Founder of Cloud Chef Labs

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams