Open in app

Sign in

Write

Sign in

Secure MinIO Not on Kubernetes

Kidong Lee
6 min readFeb 21, 2021

--

Photo by Kieran Sheehan on Unsplash

MinIO is a popular S3 compatible Object Storage which can be run on Kubernetes. MinIO running on Kubernetes can be managed in the kubernetes standard way with ease, but you have to also worry about that MinIO as Statefulsets would be restarted by unexpected reasons(for instance, Statefulset Pods can be killed according to the Pod QoS Class rules), which could result in buckets loss. Against that, you should use such tools for the backup and restore of PV in Statefulsets. The concept of Storage on Kubernetes is great, but it is not free to make it stable on Kubernetes.

Here, I am going to talk about installing MinIO with TLS in distributed mode Not on Kubernetes. You will also see how to access the external secured MinIO from the pods in Kubernetes with a selectorless service.

Install MinIO Server

Let’s say, we have three nodes for MinIO servers. Add the following hosts to /etc/hosts on every node:

10.0.0.3        minio-0
10.0.0.4        minio-1 
10.0.0.5        minio-2

Add the MinIO environment variables like minio admin user and password on every node:

cat <<EOF > /etc/profile.d/minio.sh
export MINIO_ROOT_USER=cclminio
export MINIO_ROOT_PASSWORD=rhksflja!@#
EOF

And, you have to mount disks for MinIO data, for instance, LVM mount can be done for MinIO data. For this example, it is assumed that the lvm mounted path is /export . Let’s create 6 minio data directories in the mounted path of /export :

sudo mkdir -p /export/{data1,data2,data3,data4,data5,data6}
sudo chown opc: -R /export;

Download MinIO binary:

wget https://dl.minio.io/server/minio/release/linux-amd64/minio
chmod +x minio;
sudo cp minio /usr/local/bin;
sudo chmod +x /usr/local/bin/minio;

And make sure that the port of 9000 is open to access MinIO server:

sudo firewall-cmd --get-active-zones;
sudo firewall-cmd --zone=public --add-port=9000/tcp --permanent;
sudo firewall-cmd --reload;

Now, run MinIO server on every node:

nohup minio server http://minio-{0...2}/export/data{1...6} >minio.log &

Test MinIO Object Storage with MinIO Client

There is MinIO Client mc to administer MinIO Server. With mc , you can test S3 functions to MinIO server.

Let’s install mc .

wget https://dl.min.io/client/mc/release/linux-amd64/mc;
sudo cp mc /usr/local/bin;
sudo chmod +x /usr/local/bin/mc;

Set MinIO alias minio with the endpoint of http://minio-0:9000 , user and password:

mc alias set minio http://minio-0:9000 cclminio rhksflja\!\@\# --api S3v4

You can create a bucket mykidong and copy a file to that bucket.

# create a bucket.
mc mb minio/mykidong;# copy a file to the bucket.
mc cp .bash_history minio/mykidong;

Let’s list object in the bucket:

mc ls minio/mykidong;
...
[2021-02-18 05:52:30 GMT]  12KiB .bash_history

Now, you have a distributed MinIO cluster which can be just accessed by HTTP not in secure way.

Make MinIO secure with TLS

To access MinIO server by HTTPS, we need a certificate of CA like Let’s Encrypt. Let’s add certificate generated from Let’s Encrypt to MinIO server home directory.

Before generating certificate, add DNS record for each minio node to your public DNS server. I have added the following host names for MinIO nodes to my DNS server.

minio-test0.cloudchef-labs.com
minio-test2.cloudchef-labs.com
minio-test3.cloudchef-labs.com

Generate Certificates from Let’s Encrypt

Certbot is a tool to generate certificate from Let’s Encrypt with ease. Let’s install it first.

## install snapd.
sudo yum install snapd -y;
sudo systemctl enable --now snapd.socket;
sudo ln -s /var/lib/snapd/snap /snap;## Ensure that your version of snapd is up to date: run the following command twice if failed.
sudo snap install core; sudo snap refresh core;## remove previous certbot.
sudo yum remove certbot;## install certbot.
sudo snap install --classic certbot;
sudo ln -s /snap/bin/certbot /usr/bin/certbot;

And don’t forget to allow port access of 80, 443 on each node:

# allow 80, 443.
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent;
sudo firewall-cmd --zone=public --add-port=443/tcp --permanent;
sudo firewall-cmd --reload;

Let’s generate certificate on the node of minio-0 .

# generte cert of let's encrypt.
sudo certbot certonly --standalone -d minio-test0.cloudchef-labs.com --staple-ocsp -m mykidong@cloudchef-labs.com --agree-tos;# copy generated key to minio home dir.
sudo cp /etc/letsencrypt/live/minio-test0.cloudchef-labs.com/fullchain.pem ~/.minio/certs/public.crt;
sudo cp /etc/letsencrypt/live/minio-test0.cloudchef-labs.com/privkey.pem ~/.minio/certs/private.key;# chown.
sudo chown opc:opc ~/.minio/certs/public.crt
sudo chown opc:opc ~/.minio/certs/private.key

The generated certificates will be copied to the minio home directory. You have to also generate certificates on the other nodes like minio-1 , minio-2 .

You need to let minio server listen on less than 1024:

sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/minio;

And, update your hosts file /etc/hosts :

10.0.0.3        minio-0         minio-test0.cloudchef-labs.com
10.0.0.4        minio-1         minio-test1.cloudchef-labs.com
10.0.0.5        minio-2         minio-test2.cloudchef-labs.com

Let’s run MinIO server with TLS on every node.

minio server --address ":443" https://minio-test{0...2}.cloudchef-labs.com/export/data{1...6}

Test secured MinIO with mc

Let’s test minio secured with TLS. Set alias minio-https with the endpoint of https://minio-test0.cloudchef-labs.com :

mc alias set minio-https https://minio-test0.cloudchef-labs.com cclminio rhksflja\!\@\# --api S3v4

You can list objects in the bucket. It looks like this:

mc ls minio-https/mykidong;
...
[2021-02-18 05:52:30 GMT]  12KiB .bash_history

Ok, it is great that we have a secured MinIO cluster.

But MinIO cluster is located in a private secured network in most cases. You can also consider setting up a proxy like secured NGINX in front of MinIO servers.

Setting up Secure NGINX

Let’s install NGINX first.

sudo yum install nginx -y;
sudo systemctl start nginx;

And, add a dns entry for nginx node to your public dns server, let’s say, the hostname is:

nginx-test.cloudchef-labs.com

Create certificates for NGINX like this:

sudo certbot certonly --nginx -d nginx-test.cloudchef-labs.com --staple-ocsp -m mykidong@cloudchef-labs.com --agree-tos;

And now, let’s create nginx configuration to proxy MinIO servers:

sudo su -;
cat > /etc/nginx/conf.d/nginx-test.cloudchef-labs.com.conf <<EOF
server {
    #listen 80 default_server;
    #listen [::]:80 default_server;
    root /var/www/html;
    server_name  nginx-test.cloudchef-labs.com;    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;    listen 443 ssl; # managed by Certbot    # RSA certificate
    ssl_certificate /etc/letsencrypt/live/nginx-test.cloudchef-labs.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/nginx-test.cloudchef-labs.com/privkey.pem; # managed by Certbot    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot    # Redirect non-https traffic to https
    if (\$scheme != "https") {
        return 301 https://;
    } # managed by Certbot    location / {
        proxy_set_header X-Real-IP \$remote_addr;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto \$scheme;
        proxy_set_header Host \$http_host;        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;        proxy_pass http://minio; # If you are using docker-compose this would be the hostname i.e. minio
        # Health Check endpoint might go here. See https://www.nginx.com/resources/wiki/modules/healthcheck/
        # /minio/health/live;
   } 
}
upstream minio {    
    server 10.0.0.3:9000;  
    server 10.0.0.4:9000; 
    server 10.0.0.5:9000; 
}
EOFnginx -t && nginx -s reload

Take a look at the configuration about the generated certificates:

...
ssl_certificate /etc/letsencrypt/live/nginx-test.cloudchef-labs.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/nginx-test.cloudchef-labs.com/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
...

The backing MinIO servers are listed in the configuration:

upstream minio {    
    server 10.0.0.3:9000;  
    server 10.0.0.4:9000; 
    server 10.0.0.5:9000; 
}

Test MinIO with the proxy of NGINX

Set the endpoint of https://nginx-test.cloudchef-labs.com to the alias minio-https .

mc alias set minio-https https://nginx-test.cloudchef-labs.com cclminio rhksflja\!\@\# --api S3v4;

You can list up the objects of the bucket.

mc ls minio-https/mykidong;
...
[2021-02-18 05:52:30 GMT]  12KiB .bash_history

Now, you have seen that the S3 requests have been sent to MinIO server with NGINX proxying in secure way.

Selectorless Service to access the external secure NGINX backed by MinIO

If the pods in kubernetes want to access external MinIO Object Storage through secured NGINX proxy, selectorless service can be used to access it in Kubernetes.

Let’s create a selectorless service to access the external secured NGINX.

cat <<EOF > nginx-tls-selectorless-service.yaml
---
kind: Service
apiVersion: v1
metadata:
 name: nginx-tls
spec:
  ports:
  - name: minio-https
    port: 9000
    protocol: TCP
    targetPort: 443
  type: ExternalName
  externalName: nginx-test.cloudchef-labs.com
EOFkubectl apply -f nginx-tls-selectorless-service.yaml;

Take a look at the externalName above, nginx-test.cloudchef-labs.com is the host name of external NGINX.

Let’s run the pod of mc to access the external secured NGINX backed by MinIO.

### run minio client as pod to access external nginx with tls.
kubectl run -it --rm --restart=Never --image=minio/mc --command=true mc -- /bin/bash -c \
'mc alias set minio-kube http://nginx-tls.default:9000 cclminio rhksflja\!\@\# --api S3v4 && mc ls minio-kube/mykidong'...
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
Added `minio-kube` successfully.
[2021-02-18 05:52:30 UTC]  12KiB .bash_history

Pods in Kubernetes can use the host name of nginx-tls.default with the port 9000 to access the external secured NGINX in flexible way.

I have talked about securing MinIO In-Transit for now. To make MinIO secure At-Rest, you can see here: https://mykidong.medium.com/secure-minio-at-rest-with-vault-ce319f8b6274

Minio
Kubernetes
Lets Encrypt
Nginx

--

--

Kidong Lee
Kidong Lee

Written by Kidong Lee

291 Followers
134 Following

Founder of Cloud Chef Labs | Chango | Unified Data Lakehouse Platform | Iceberg centric Data Lakehouses https://www.cloudchef-labs.com/

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams