Installing Community Kubernetes & DKube

This section describes how to install DKube on:

  • GCP

  • AWS

  • On-prem

The installation procedure is executed from the $HOME/.dkube folder on the installation node. A fully automated process will install:

  • Community k8s

  • DKube

Editing the Configuration Files

There are 2 configuration files that need to be edited for installation of k8s and DKube.

File

Description

k8s.ini

Configuration for k8s installation and cluster node setup

dkube.ini

Configuration for DKube installation

Important

Both ini files should be configured before executing any commands

Editing the k8s ini File

The k8s.ini file provides the information needed to access the cluster and install Kubernetes.

The k8s.ini file has the following format:

_images/k8s-ini-File.png

Only the following fields should be filled in:

Field

Value

provider

Type of platform

distro

OS type

master

Set to true to allow jobs to be run on the master nodes

master_HA

Set to true to enable a resilient cluster

nodes

IP addresses of the nodes in the cluster

user

The user name for the account on the DKube cluster

Provider & OS

This identifies the type of platform that DKube will run on, and the operating system.

Schedule Jobs on Master Node

On a cluster that includes only a Master node, jobs will automatically be used on that node. If the cluster includes workers nodes, DKube can allow the jobs to be used on the Master node, or - if desired - only on the Worker nodes. If the system is being set up as a resilient platform, this option will allow jobs to be scheduled on all of the master nodes or none.

Note

Since the Master node does the overall coordination of the cluster, it is recommended that jobs not be schedulable on the Master node

Resilient Operation

If the cluster has at least 3 nodes, it can be configured to be resilient, guaranteeing that it will continue when one of the nodes becomes inoperable. This is explained in the section Cluster and DKube Resiliency. Note that this configuration only controls cluster resiliency, and not DKube resiliency (which is configured during the DKube installation procress). To enable this option, set the master_HA option to true.

Nodes

The node IP addresses should reflect how the installation node accesses the cluster nodes. Typically, the public IP address is all that is required. But if both are required in some cases, the ini file shows how they need to be provided.

Note

The first IP address in the list will be the master node, and the rest will be worker nodes. When master_HA is set to true, the first 3 nodes are the master nodes.

User

This is the DKube cluster user account name. It can be a root or non-root account, but the same account must be available on all cluster nodes, and must have passwordless access through an ssh key and sudoers permissions.

In general, the default User will work for installation on the cloud, and this field does not need to be changed. For an on-prem installation, or a non-standard cloud installation, the user name will be provided by the IT group.

Important

The user must be the same on each node of the cluster

Editing the DKube ini File

The dkube.ini file controls the DKube installation options.

_images/dkube-ini-File.png

Field

Value

KUBE_PROVIDER

dkube

HA

Set true or false to enable/disable DKube resiliency

USERNAME

User-chosen initial login username

PASSWORD

User-chosen initial login password

Resilient Operation

DKube can run as a resilient system that guarantees the databases remain usable when one of the nodes become inoperable. This requires at least 3 schedulable nodes in the cluster. This is explained in the section Cluster and DKube Resiliency. Note that this only applies to DKube resiliency. The k8s cluster can be resilient or not and still run DKube in HA mode, as long as the DKube resiliency requirements are met. If you have provided that minimum configuration, you can set the HA field to be true for resilient DKube operation.

Username and Password

This provides the credentials for initial DKube local login. The initial login user has both Operator and Data Scientist access. Only a single user can log in with this method. More users can be added through a backend access configuration using the OAuth screen.

Do not use the following names:

  • dkube

  • monitoring

  • kubeflow

Storage Options

The storage options are configured in the [STORAGE] section of the dkube.ini file. The settings depend upon the type of storage configured, and whether the DKube installation will be HA or non-HA.

Storage Type

Instructions

Local

DKube Installation with Local Storage

NFS

DKube Installation with NFS

Ceph

DKube Installation with Ceph

DKube Installation with Local Storage

DKube can be configured to use local storage on the nodes. The storage configuration will depend upon whether DKube is in HA or non-HA mode. To select local storage, the following field value should be selected:

Field

Value

STORAGE_TYPE

disk

The node field will depend upon the resiliency configuration (HA or non-HA).

Field

Resiliency

Value

STORAGE_DISK_NODE

non-HA

auto

STORAGE_DISK_NODE

HA

Value ignored - DKube will create an internal Ceph cluster using the disks from all of the nodes

_images/dkube-ini-File-k8s-storage-local.png

Proceed to Cluster Access Options.

DKube Installation with NFS

NFS is configured the same for HA and non-HA. In order to configure an external NFS for DKube use, the following fields should be filled in:

Field

Value

STORAGE_TYPE

nfs

STORAGE_NFS_SERVER

Internal IP address of nfs server

STORAGE_NFS_PATH

Absolute path of the exported share

Note

The path must exist on the share, but should not be mounted. DKube will perform its own mount

_images/nfs_dkube_ini.png

Proceed to Cluster Access Options.

DKube Installation with Ceph

Ceph is configured the same for HA and non-HA. For an external Ceph configuration, the following fields should be filled in:

Field

Value

STORAGE_TYPE

ceph

STORAGE_CEPH_MONITORS

IP addresses of the Ceph monitors

STORAGE_CEPH_SECRET

Ceph token

Important

Ceph must be installed with 3 monitors

_images/dkube-ini-File-Ceph.png

Cluster Access Options

Cluster access is configured in the [EXTERNAL] section of the dkube.ini file. The fields should be configured as follows, depending upon the load balancer installed.

IP Access or External Load Balancer

Use the following configuration if the cluster is accessed by:

  • The IPs of the cluster nodes, or

  • By a VIP on a load balancer that is external to the k8s cluster

Field

Value

ACCESS

nodeport

INSTALL_LOADBALANCER

false

_images/dkube-ini-External-Default.png

Proceed to Cluster Access from Installation Node

DKube-Installed Load Balancer

If the cluster is accessed by the MetalLB load balancer provided by DKube, use the following configuration:

Field

Value

ACCESS

loadbalancer

INSTALL_LOADBALANCER

true

LB_VIP_POOL

Pool of IP addresses used to provision the VIPs for the load balancer

Proceed to Cluster Access from Installation Node

User-Deployed Load Balancer

If the cluster is accessed by a user-deployed load balancer that is aware of the k8s cluster, use the following configuration:

Field

Value

ACCESS

loadbalancer

INSTALL_LOADBALANCER

false

Cluster Access from Installation Node

In order to run the scripts, the installation node needs to be able to access each node in the cluster without requiring a password. In each case, sudoers account access must be provided.

The sudoers file on each node must include the DKube cluster account name with the necessary access. This can be accomplished using the visudo command, and adding the following line:

<username> ALL=(ALL) NOPASSWD:ALL

Cluster Access for an AWS Cluster

If DKube is going to be installed on an AWS cluster, the pem file from that cluster is used to provide access.

  • Copy the .pem key to the $HOME/.dkube folder

  • Use the following commands in the $HOME/.dkube folder to set up cluster access

sudo chmod 400 <pem file> sudo ./setup-ssh.sh --key=<pem file>

Continue the k8s installation at Final Access Verification

Cluster Access for a GCP or On-Prem Cluster

If DKube is going to be installed on a GCP or On-prem cluster, an ssh key pair is used to provide cluster access.

Important

Even in the case where the master node is used as the installation node, the ssh key pair must still be added to the master node authorized_keys file in the $HOME/.ssh directory

Using Your Own Key Pair

If you have your own ssh key pair, it is assumed that the private key works with all of the DKube cluster nodes, including the master node. In this case, the following steps are required:

  • Copy the private key to the $HOME/.dkube directory. It needs to be copied with the name ssh-rsa

  • Delete the file ssh-rsa.pub from the $HOME/.dkube directory, since it will not match your new private ssh-key file

Docker-Supplied ssh Key Pair

The initial Docker init creates an ssh key pair to allow passwordless access to the DKube cluster nodes.

If the ssh key pair created by the Docker init will be used for cluster access, then it’s public key file contents need to be added to the $HOME/.ssh/authorized_keys file on each node of the DKube cluster, including the master node. This can generally be accomplished by simply adding it with:

sudo ssh-copy-id -i $HOME/.dkube/ssh-rsa.pub <username>@<Master Node IP Address>

If that does not work, you can append the contents of ssh-rsa.pub to $HOME/.ssh/authorized_keys manually using the command:

sudo cat ssh-rsa.pub >> $HOME/.ssh/authorized_keys

Note

For gcp, it it sometimes necessary to also copy the contents of ssh-rsa.pub to the VM instance manually from the VM Instances dashboard

_images/gcp-ssh-edit.png

Final Access Verification

After the security access steps have been taken, the user should ensure that each node in the cluster can be properly accessed by the installation node without a password.

The installation process should not move ahead if the verification step is not successful.

sudo ssh -i ssh-rsa <username>@<Master Node IP Address>

If both need to be installed, start with the following section. If the cluster already has k8s installed, go directly to the section Installing the Necessary Packages on the Cluster.

Installing Kubernetes

The installation script will check that the cluster is accessible, and that the prerequisites are all in place. It will then initiate the installation. If an error is detected the script will stop with an error message that identifies the error.

sudo ./dkubeadm cluster install

Error During Installation

If the cluster install procedure detects that there is an existing kubernetes cluster already in place, a node on the cluster cannot be accessed passwordlessly, or that some of the prerequisites are not correct, the first troubleshooting step is to uninstall the cluster, which will also do a cleanup. The command to uninstall and cleanup is:

sudo ./dkubeadm cluster uninstall

After the cleanup is complete, run the install command again. If it still fails, contact your IT manager.

Once the Kubernetes cluster has been successfully installed, you can use the Kubernetes dashboard to see the status of the system. Follow the instructions that are provided as part of the installation log, right after the message that the installation has been successful.

Selecting the link in the log output will take you to a browser window that asks for credentials. Copying the contents of the file k8s-token will access the dashboard.

Installing DKube

This section describes how to install DKube after Kubernetes has been installed on the cluster.

Installing the Necessary Packages on the Cluster

Before the DKube installation is started, it is important that all of the required software packages are installed on the DKube cluster. The script will automatically audit what software packages are installed on the cluster nodes and install anything that is required. If an existing package is already installed, it will not be re-installed.

Note

This command requires that the k8s.ini be filled in as described in Installing Kubernetes, and that the installation node have passwordless access to the cluster as described in Cluster Access from Installation Node. This is required even if the user installed k8s on their own.

Note

If the user wants to install some of their own software packages on the DKube cluster, they should be installed prior to running the node setup command

sudo ./dkubeadm node setup

Installing DKube on the Cluster

After completing the configuration files and installing the prerequisites, the DKube installation can begin. The initial setup will start, and the script log will provide a url link to a browser-based dashboard that can be used to track the progress of the installation. The url is highlighted in the figure. The public-IP referenced in the url is the first IP address used in the k8s.ini file, which is the Master node.

sudo ./dkubeadm dkube install
_images/dkube-install-log.png
_images/DKube-Install-Dashboard.jpg

If the installation fails, go to DKube Installation Failure

If the installation in successful, the dashboard will show the status of COMPLETED.

Accessing DKube

After the DKube installation dashboard shows that the installation has completed, the DKube UI is shown as part of the dashboard.

The form of the url is:

https://xxx.xxx.xxx.xxx:32222/

The IP address in the url is the first IP address used in the k8s.ini file, which is the Master node.

Initial Login

The initial login after installation is accomplished with the username and password entered in the dkube.ini file. Authorization is based on a backend mechanism is explained in the User Guide in the section “Getting Started”.


DKube Installation Failure

If the DKube install procedure detects that some of the prerequisites are not correct, the first troubleshooting step is to uninstall and cleanup the system. The command to uninstall and cleanup is:

sudo ./dkubeadm dkube uninstall sudo ./dkubeadm node cleanup sudo ./dkubeadm node setup

After this successfully completes, run the DKube install command again at Installing DKube on the Cluster . If it still fails, contact your IT manager.


Uninstalling DKube & Kubernetes

DKube and k8s can be uninstalled by following the steps in this section from the $HOME/.dkube directory.

Uninstalling DKube

DKube should always be uninstalled from the cluster first. The following command is used to initiate the DKube removal:

sudo ./dkubeadm dkube uninstall sudo ./dkubeadm node cleanup

Uninstalling Kubernetes

If k8s was installed through the script described in this guide, then it can also be uninstalled through a One Convergence script. If k8s was installed by the customer independently of the script, then it must be uninstalled manually through the appropriate method. The following command is used to uninstall k8s:

sudo ./dkubeadm cluster uninstall