Installation DKube on a Rancher Cluster

This section describes how to install DKube on an existing Rancher cluster.

To install DKube on a system, it is assumed that:

  • A Rancher Server has been created

  • The DKube cluster is running CentOS 7 on all of the nodes

  • All the nodes have a static IP address

  • Docker CE has been installed on the installation node

For instructions on how to set up the cluster first, jump to Creating a Rancher Cluster. Then return here for the DKube instructions.

Setting up the Cluster

Execute the Rancher Server Run Command

In order to create k8s on the DKube cluster, the Run command must be executed on each node in the DKube cluster. The Run command is generated during the Add Cluster procedure on the Rancher Server, and can be obtained later from the Edit screen as shown below.

_images/Rancher-Cluster-Edit.png _images/Rancher-Run-Command.png

The execution of the Run command on the DKube node will initiate activity on the Rancher Server. When the activities have been complete the Rancher Server will show an “Active” status.

_images/Rancher-Active.png

Copying the Kubeconfig file to the DKube Cluster Master Node

The Kubeconfig file from the Rancher Server must be copied to the installation node (either a remote installation node or the master node in the cluster). The Kubeconfig file can be found by selecting the cluster name.

_images/Rancher-Kubeconfig.png

The contents of the Kubeconfig file should be put into the file $HOME/.dkube/kubeconfig on the installation node.

Note

Note that the name kubeconfig must be all lower case

Identifying the Node Names & IP Addresses from the Rancher Server

The DKube installation configuration will need the node names and IP addresses from the Rancher Server. This can be identified from the Nodes screen.

_images/Rancher-Nodename.png

Installing DKube on the Cluster

This section describes how to install DKube on an existing Rancher cluster. The rest of the installation is executed from the $HOME/.dkube folder.

There are 2 configuration files that need to be edited for installation of k8s and DKube.

File

Description

k8s.ini

Configuration for cluster node setup

dkube.ini

Configuration for DKube installation

Important

Both ini files should be configured before executing any commands

Editing the k8s ini File

The k8s.ini file has the following format:

_images/k8s-ini-file-Rancher.png

Only the following fields should be filled in:

Field

Value

provider

onprem

distro

centos

nodes

IP addresses of the nodes in the DKube cluster

user

The user name for the DKube cluster account

Nodes

The node IP addresses should reflect how the installation node accesses the cluster nodes. These are provided as described in the section Identifying the Node Names & IP Addresses from the Rancher Server.

User

This is the DKube cluster user account name. It can be a root or non-root account, but the same account must be available on all cluster nodes, and must have passwordless access through an ssh key and sudoers permissions.

Cluster Access from the Installation Node

In order to run the scripts, the installation node needs to be able to access each node in the cluster without requiring a password. In each case, sudoers account access must be provided.

The sudoers file on each node must include the DKube cluster account name with the necessary access. This can be accomplished using the visudo command, and adding the following line:

<username> ALL=(ALL) NOPASSWD:ALL

Cluster Access for an AWS Cluster

If DKube is going to be installed on an AWS cluster, the pem file from that cluster is used to provide access

  • Copy the .pem key to the $HOME/.dkube folder

  • Use the following commands in the $HOME/.dkube folder to set up cluster access

sudo chmod 400 <pem file> sudo ./setup-ssh.sh --key=<pem file>

Continue the instructions at Final Access Verification

Cluster Access for a GCP or On-Prem Cluster

If DKube is going to be installed on a GCP or On-prem cluster, an ssh key pair is used to provide cluster access.

Important

Even in the case where the master node is used as the installation node, the ssh key pair must still be added to the master node authorized_keys file in the $HOME/.ssh directory

Using Your Own Key Pair

If you have your own ssh key pair, it is assumed that the private key works with all of the DKube cluster nodes, including the master node. In this case, the following steps are required:

  • Copy the private key to the $HOME/.dkube directory. It needs to be copied with the name ssh-rsa

  • Delete the file ssh-rsa.pub from the $HOME/.dkube directory, since it will not match your new private ssh-key file

Docker-Supplied ssh Key Pair

The initial Docker init creates an ssh key pair to allow passwordless access to the DKube cluster nodes.

If the ssh key pair created by the Docker init will be used for cluster access, then it’s public key file contents need to be added to the $HOME/.ssh/authorized_keys file on each node of the DKube cluster, including the master node. This can generally be accomplished by simply adding it with:

sudo ssh-copy-id -i $HOME/.dkube/ssh-rsa.pub <username>@<Master Node IP Address>

If that does not work, you can append the contents of ssh-rsa.pub to $HOME/.ssh/authorized_keys manually using the command:

sudo cat ssh-rsa.pub >> $HOME/.ssh/authorized_keys

Note

For gcp, it it sometimes necessary to also copy the contents of ssh-rsa.pub to the VM instance manually from the VM Instances dashboard

_images/gcp-ssh-edit.png

Final Access Verification

After the security access steps have been taken, the user should ensure that each node in the cluster can be properly accessed by the installation node without a password.

The installation process should not move ahead if this verification step is not successful.

sudo ssh -i ssh-rsa <username>@<Master Node IP Address>

Editing the DKube ini File

The dkube.ini file controls the DKube installation options.

_images/dkube-ini-File-Rancher.png

Field

Value

KUBE_PROVIDER

dkube

HA

Set true or false to enable/disable DKube resiliency

USERNAME

User-chosen initial login username

PASSWORD

User-chosen initial login password

Resilient Operation

DKube can run as a resilient system that guarantees the databases remain usable when one of the nodes become inoperable. This requires at least 3 schedulable nodes in the cluster. This is explained in the section Cluster and DKube Resiliency. Note that this only applies to DKube resiliency. The k8s cluster can be resilient or not and still run DKube in HA mode, as long as the DKube resiliency requirements are met. If you have provided that minimum configuration, you can set the HA field to be true for resilient DKube operation.

Username and Password

This provides the credentials for initial DKube local login. The initial login user has both Operator and Data Scientist access. Only a single user can log in with this method. More users can be added through a backend access configuration using the OAuth screen.

The Username has the following restrictions:

Do not use the following names:

  • dkube

  • monitoring

  • kubeflow

Storage Options

The storage options are configured in the [STORAGE] section of the dkube.ini file. The settings depend upon the type of storage configured, and whether the DKube installation will be HA or non-HA.

Storage Type

Instructions

Local

DKube Installation with Local Storage

NFS

DKube Installation with NFS

Ceph

DKube Installation with Ceph

DKube Installation with Local Storage

DKube can be configured to use the local storage on the nodes. The storage configuration will depend upon whether DKube is in HA or non-HA mode. To select local storage, the following field value should be selected:

Field

Value

STORAGE_TYPE

disk

The node field will depend upon the resiliency configuration (HA or non-HA).

Field

Resiliency

Value

STORAGE_DISK_NODE

non-HA

Node name as identified in the Rancher Server in section Identifying the Node Names & IP Addresses from the Rancher Server

STORAGE_DISK_NODE

HA

Value ignored - DKube will create an internal Ceph cluster using the disks from all of the nodes

_images/dkube-ini-rancher.png

Proceed to Cluster Access Options.

DKube Installation with NFS

NFS is configured the same for HA and non-HA. In order to configure an external NFS for DKube use, the following fields should be filled in:

Field

Value

STORAGE_TYPE

nfs

STORAGE_NFS_SERVER

Internal IP address of nfs server

STORAGE_NFS_PATH

Absolute path of the exported share

Note

The path must exist on the share, but should not be mounted. DKube will perform its own mount

_images/nfs_dkube_ini.png

Proceed to Cluster Access Options.

DKube Installation with Ceph

Ceph is configured the same for HA and non-HA. For external Ceph configuration, the following fields should be filled in:

Field

Value

STORAGE_TYPE

ceph

STORAGE_CEPH_MONITORS

IP addresses of the Ceph monitors

STORAGE_CEPH_SECRET

Ceph token

Important

Ceph must be installed with 3 monitors

_images/dkube-ini-File-Ceph.png

Cluster Access Options

Cluster access is configured in the [EXTERNAL] section of the dkube.ini file. The fields should be configured as follows, depending upon the load balancer installed.

IP Access or External Load Balancer

Use the following configuration if the cluster is accessed by:

  • The IPs of the cluster nodes, or

  • By a VIP on a load balancer that is external to the k8s cluster

Field

Value

ACCESS

nodeport

INSTALL_LOADBALANCER

false

_images/dkube-ini-External-Default.png

Proceed to Running the DKube Installation Script

DKube-Installed Load Balancer

If the cluster is accessed by the MetalLB load balancer provided by DKube, use the following configuration:

Field

Value

ACCESS

loadbalancer

INSTALL_LOADBALANCER

true

LB_VIP_POOL

Pool of IP addresses used to provision the VIPs for the load balancer

Proceed to Running the DKube Installation Script

User-Deployed Load Balancer

If the cluster is accessed by a user-deployed load balancer that is aware of the k8s cluster, use the following configuration:

Field

Value

ACCESS

loadbalancer

INSTALL_LOADBALANCER

false

Running the Node Setup Command

The node setup command will install the necessary software packages on each node of the DKube cluster.

sudo ./dkubeadm node setup

Running the DKube Installation Script

After completing the configuration of the dkube.ini file and installing the prerequisites, the DKube installation can begin. The initial setup will start, and the script log will provide a url link to a browser-based dashboard that can be used to track the progress of the installation. The url is highlighted in the figure. The public-IP referenced in the url is the first IP address used in the k8s.ini file, which is the Master node.

sudo ./dkubeadm dkube install
_images/dkube-install-log.png
_images/DKube-Install-Dashboard.jpg

The dashboard will show the status of COMPLETED when DKube has been successfully installed.

If the installation fails, go to DKube Installation Failure

If the installation in successful, the dashboard will show the status of COMPLETED.

Accessing DKube

After the DKube installation dashboard shows that the installation has completed, the DKube UI is shown as part of the dashboard.

The form of the url is:

https://xxx.xxx.xxx.xxx:32222/

The IP address in the url is the first IP address used in the k8s.ini file, which is the Master node.

Initial Login

The initial login after installation is accomplished with the username and password entered in the dkube.ini file. Authorization based on a backend mechanism is explained in the User Guide in the section “Getting Started”.


DKube Installation Failure

If the DKube install procedure detects that some of the prerequisites are not correct, the first troubleshooting step is to uninstall and cleanup the system. The command to uninstall and cleanup is:

sudo ./dkubeadm dkube uninstall sudo ./dkubeadm node cleanup sudo ./dkubeadm node setup

After this successfully completes, run the DKube install command again at Running the DKube Installation Script . If it still fails, contact your IT manager.


Uninstalling DKube

DKube can be uninstalled by following the steps in this section from the $HOME/.dkube directory:

sudo ./dkubeadm dkube uninstall sudo ./dkubeadm node cleanup