AntDB Installation and Deployment Documentation - P1

[antdb]

Overview

The node types of AntDB cluster are gtmcoord (master and slave), coordinator, datanode (master and slave). ADBMGR is the cluster management node of AntDB, users can use this node to finish the cluster installation and deployment after installing and deploying ADBMGR. There are mainly five major steps.

•add host is to add all the host information in the cluster to the host table, which exists in ADBMGR to facilitate the management of cluster hosts.
•deploy deploys the binaries to all hosts in the cluster.
•start agent starts the agent process on the hosts, one agent per host.
•add node adds all nodes in the cluster (including gtmcoord, coordinator, datanode) to the node table, which exists on ADBMGR to facilitate the management of all types of nodes in the cluster.
•init all initializes the cluster, and once the initialization is complete, users can log in to the cluster to perform operations.

The following will introduce the installation and deployment of AntDB in detail, starting from the environment preparation.

Pre-installation Preparation

This section introduces the hardware devices and operating system that need to be prepared before installing AntDB.

Operating System

This section introduces the common operating system versions that AntDB is compatible with.

Hardware Environment

This section describes the hardware configuration required to install AntDB.

Installation files

Please obtain the installation package from the delivery person.

Linux Environment Configuration

This section describes the configuration of the Linux system before installing AntDB, including user creation, dependency installation, and system parameter adjustment.

User Configuration

New User Creation

Create a new normal user, such as uatadb, on all hosts where AntDB is to be installed (or use an existing normal user). For example, new user reference:

useradd -d /home/uatadb uatadb
passwd uatadb

Configure the user limit parameters

Edit the /etc/security/limits.conf file and configure the antdb user parameters.

antdb soft nproc 65536
antdb hard nproc 65536
antdb soft nofile 278528
antdb hard nofile 278528
antdb soft stack unlimited
antdb soft core unlimited
antdb hard core unlimited
antdb soft memlock 250000000
antdb hard memlock 250000000

After saving the file, execute su - antdb to switch to antdb user and execute ulimit -a to check if it takes effect.

Configure user sudo privileges

If security allows, it is recommended to add sudo privileges to the antdb user. Execute visudo for the root user to edit the interface, find the line where Allow root to run any commands anywhere, and add below the line.

antdb ALL=(ALL) ALL

Save the file and exit. su - antdb Switch to the antdb user and execute sudo id. Expect to be prompted for the user password and the output will be

uid=0(root) gid=0(root) groups=0(root)

indicates that sudo privileges were added successfully.

System parameter adjustment

Turn off the firewall

Using the centos 7 operating system as an example:
1.Disable the firewall service.

systemctl stop firewalld.service

2.Disable the firewall self-start service.

systemctl disable firewalld.service

3.Check the firewall status.

systemctl status firewalld.service

Shut down numa and tuned

Take redhat/centos 7 system as an example:
1.Shut down numa.

grubby --update-kernel=ALL --args="numa=off " # This command modifies this file:
/etc/grub2.cfg
grub2-mkconfig

2.Shut down the tuned service.

systemctl stop tuned
systemctl disable tuned

3.Reboot the host after the changes in this way take effect.

reboot

4.Verify the cmdline of grub after rebooting.

cat /proc/cmdline

5.Check numa

numactl --hardware

The expected result is Available: 1 nodes (0)

Turn off Transparent Huge Pages

The use of Transparent Huge Pages (THP for short) can cause performance problems, so it is recommended to turn them off.
1.Check the on status of Transparent Huge Pages.

cat /sys/kernel/mm/transparent_hugepage/enabled

If the result is [always] madvise never, then Transparent Huge Pages is on and needs to be turned off; if the result is always madvise [never], then Transparent Huge Pages is off and this step is skipped;

2.Turn off Transparent Huge Pages.

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag Check Transparent Large Page Open Status

3.Check again that the Transparent Big Page is on.

cat /sys/kernel/mm/transparent_hugepage/enabled

Configure sysctl.conf

1.Modify the sysctl.conf file.

cat >> /etc/sysctl.conf << EOF
# add for antdb
kernel.shmmax=137438953472 137438953472
kernel.shmall=53689091
kernel.shmmni=4096
kernel.msgmnb=4203520
kernel.msgmax=65536
kernel.msgmni=32768
kernel.sem=501000 641280000 501000 12800
 
fs.aio-max-nr=6553600
fs.file-max=26289810
net.core.rmem_default=8388608
net.core.rmem_max=16777216
net.core.wmem_default=8388608
net.core.wmem_max=16777216
net.core.netdev_max_backlog=262144
net.core.somaxconn= 65535
net.ipv4.tcp_rmem=8192 87380 16777216
net.ipv4.tcp_wmem=8192 65536 16777216
net.ipv4.tcp_max_syn_backlog=262144
net.ipv4.tcp_keepalive_time=180
net.ipv4.tcp_keepalive_intvl=10
net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_fin_timeout=1
net.ipv4.tcp_synack_retries=1
net.ipv4.tcp_syn_retries=1
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_max_tw_buckets=256000
net.ipv4.tcp_retries1=2
net.ipv4.tcp_retries2=3
vm.dirty_background_ratio=5
vm.dirty_expire_centisecs=6000
vm.dirty_writeback_centisecs=500
vm.dirty_ratio=20
vm.overcommit_memory=0
vm.overcommit_ratio= 120
vm.vfs_cache_pressure = 100
vm.swappiness=10
vm.drop_caches = 2
vm.min_free_kbytes = 2048000
vm.zone_reclaim_mode=0
kernel.core_uses_pid=1
kernel.core_pattern= /data/antdb/core/core-%e-%p-%s-%t
fs.suid_dumpable=1
kernel.sysrq=0
EOF

The path of kernel.core_pattern needs to be modified according to the actual environment information.

2.execute the following command to make the above parameters effective.

sysctl -p

Software Installation and Deployment

This section describes how to install the AntDB Distributed Edition.

Distributed Edition (Single Center) Installation and Deployment

Installation using RPM packages (not recommended)

Dependency package installation

For offline environment, it is recommended to build a local yum source on the host computer through the image file of the operating system to facilitate the installation of dependencies. Under cento or redhat operating system, execute the following command to install the dependencies.

sudo yum install -y perl-ExtUtils-Embed
sudo yum install -y flex
sudo yum install -y bison
sudo yum install -y readline-devel
sudo yum install -y zlib-devel
sudo yum install -y openssl-devel
sudo yum install -y pam-devel
sudo yum install -y libxml2-devel
sudo yum install -y libxslt-devel
sudo yum install -y openldap-devel
sudo yum install -y python-devel
sudo yum install -y gcc-c++ 

For other operating systems, please just change the installation command, and the dependency package names will remain the same. Some dependencies do not provide libssh2 package, you can install it by compiling the source code.
•Download at: https://www.libssh2.org/download/
•Source code compilation and installation (executed by root).

wget https://www.libssh2.org/download/libssh2-1.9.0.tar.gz
tar xzvf libssh2-1.9.0.tar.gz
cd libssh2-1.9.0
./configure
make
make install

Install RPM packages

Upload the RPM package provided by the delivery person to the server on the host where ADBMGR is located and install it under root or a user with sudo privileges. Install to the default path /opt/app/antdb.

sudo rpm -ivh antdb-xxx.rpm

If you want to install to other paths, you can do so as follows:

sudo rpm -ivh antdb-xxx.rpm --relocate=/opt/app/antdb=$ADBHOME

$ADBHOME as a custom directory, the name is best to be known by its name, e.g.:

/home/antdb/app/antdb

After the RPM package is installed, the value of the variable ADB_HOME will either be /opt/app/antdb, or a directory of your choice. Next, you need to modify the directory permissions:

sudo chown -R antdb:antdb $ADBHOME
sudo chmod -R 755 $ADBHOME

Configure Environment Variables

After installing the RPM package, the initmgr and mgr_ctl executables will be generated in the bin directory of the specified directory (i.e. $ADBHOME directory). To initialize adbmgr you also need to configure the PATH variable to do so, by appending the following to the hidden file .bashrc under the current user (execute vim ~/.bashrc to open the file):

export ADBHOME=/opt/app/antdb
export PATH=$ADBHOME/bin:$PATH
export LD_LIBRARY_PATH=$ADBHOME/lib:$LD_LIBRARY_PATH
export PGDATABASE=postgres
 
export mgrdata=/data/antdb/mgr1
alias adbmgr='psql -p 6432 -d postgres '
alias mgr_stop='mgr_ctl stop -D $mgrdata -m fast'
alias mgr_start='mgr_ctl start -D $mgrdata' 

Note: ADBHOME needs to be set according to the path of the binary executable generated by the compilation of AntDB. mgrdata also needs to be modified according to the actual path of the respective environment. Then execute the following command to make it effective:

source ~/.bashrc

Install with tarball

Copy the installation package (ADB-Community.tar.gz) to the directory of AntDB user (e.g. uatadb) and change the permissions of the package.

chown -R uatadb:uatadb ADB-Community.tar.gz #Change the user permission of the installer to uatadb

Login to the new user to install AntDB

su - uatadb                           #Login to the newly created AntDB user
tar -xzvf ADB-Community.tar.gz    #Unzip the installation package
cd ADB-Community                  #Enter the extracted directory ADB-Community with the cd command 

Interactive installation

Execute sh AntDB_install.sh under ADB-Community to install AntDB interactively.

sh AntDB_install.sh

The interactive installation can be customized in three ways, as shown below. You can choose the default value or customize it. However, it is important to note that you must select "d" for the second place, which means that the installation is a distributed database.

# Step one:
# Select whether you agree to the user agreement, only if you agree (Y) can you continue to install further, if you do not agree (N), the installation will stop.
Do you agree?(Y/N)[Default:Y]
You input:
OK, you agree to the agreement. continue...
check_agreement...success.
 
# Step two:
# Select the mode which you want to install, centralized (c), distributed (d). The default selection is c, for a centralized installation. Enter Q to abort this installation.
# Here, we choose d for a distributed database installation
Please enter the deployment mode "(c)entralized" or "(d)istributed" or "Q" to exit the installation.[Default:centralized]d
You input : d
 
# Step three:
# Fill in the installation path. Note: You need to fill in the absolute path.
# If the user does not enter the absolute path, just enter, it will be installed to the default path. Entering Q aborts this installation.
Please enter a valid empty directory to install antdb. Or enter "Q/q" to exit the installation.[Default:/data/adb01/app/antdb]
You input :
Use default install path:[/data/adb01/app/antdb]. continue...
check install path...
check install path ...success.
decompress_package...
decompress_package...success.
copy files...
copy files...success.
set bash environment...
please excute cmd 'source /data/adb01/.bashrc' to make the environment variables take effect.
set bash environment...success.
set kernel config...
Please refer to the following files in the installation directory to manually set the recommended kernel config.
/data/adb01/app/antdb/setup/common/kernel_limits.conf
/data/adb01/app/antdb/setup/common/kernel_sysctl.conf
antdb install ...success.

After the above execution, according to the prompt, you need to source the environment variables, and after the source is done, the installation of the binaries is finished.

source /data/adb01/.bashrc

Deploy adbmgr

1.Initialize adbmgr.

initmgr -D $mgrdata

Where $mgrdata is the user-specified installation directory for adbmgr (this file directory needs to be created in advance). After initialization, the following files are generated in the specified directory.

[gd@INTEL175 ~]$ cd /data/antdb/mgr1 
[gd@INTEL175 mgr1]$ ll
total 112
drwx------ 5 gd gd  4096 Oct 17 17:28 base
drwx------ 2 gd gd  4096 Oct 17 17:28 global
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_clog
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_commit_ts
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_dynshmem
-rw------- 1 gd gd  4450 Oct 17 17:28 pg_hba.conf
-rw------- 1 gd gd  1636 Oct 17 17:28 pg_ident.conf
drwx------ 4 gd gd  4096 Oct 17 17:28 pg_logical
drwx------ 4 gd gd  4096 Oct 17 17:28 pg_multixact
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_notify
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_replslot
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_serial
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_snapshots
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_stat
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_stat_tmp
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_subtrans
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_tblspc
drwx------ 2 gd gd  4096 Oct 17 17:28 pg_twophase
-rw------- 1 gd gd     4 Oct 17 17:28 PG_VERSION
drwx------ 3 gd gd  4096 Oct 17 17:28 pg_xlog
-rw------- 1 gd gd    88 Oct 17 17:28 postgresql.auto.conf
-rw------- 1 gd gd 22234 Oct 17 17:28 postgresql.conf

2.Modify the adbmgr configuration parameters.
a. Modify postgresql.conf.

cat >> ${mgrdata}/postgresql.conf << EOF
port = 6432
listen_addresses = '*'
log_directory = 'pg_log'
log_destination ='csvlog'
logging_collector = on
log_min_messages = error
max_wal_senders = 3
hot_standby = on
wal_level = replica
EOF

The default port for adbmgr is 6432, and the user can set an available port according to the planning.

b. Modify pg_hba.conf.

cat >> ${mgrdata}/pg_hba.conf << EOF
host replication all 10.0.0.0/8 trust
host all all 10.0.0.0/8 trust
EOF

The IP in hba needs to be modified according to the actual situation.

3.Start adbmgr; After adbmgr is initialized successfully, you can start it.

mgr_ctl start -D /data/antdb/mgr1
/data/antdb/mgr 

To be consistent with the initialization path, both are the values of the variable mgrdata.

[gd@INTEL175 ~]$ mgr_ctl start -D /data/antdb/mgr
server starting
[gd@INTEL175 ~]$ LOG: database system was shut down at 2017-10-17 17:38:15 CST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: adb monitor launcher started

If adbmgr reports that the port is occupied during startup, the reason is that the current port number is already used by another process (adbmgr uses port 6432 by default). You need to change the value of the variable port in /data/antdb/mgr/postgresql.conf file so that it does not conflict with other processes. After starting, ps can view the processes of mgr as follows.

67456 ?       S      0:00 /data/antdb/app/bin/adbmgrd -D mgr
67458 ?       Ss     0:00 \_ adbmgr: checkpointer process
67459 ?       Ss     0:00 \_ adbmgr: writer process    
67460 ?       Ss     0:00 \_ adbmgr: wal writer process    
67461 ?       Ss     0:00 \_ adbmgr: autovacuum launcher process   
67462 ?       Ss     0:00 \_ adbmgr: stats collector process   
67463 ?       Ss     0:00 \_ adbmgr: adb monitor launcher process

4.Configure standby adbmgr (optional); In a real production environment, adbmgr also needs to be highly available, so it is also necessary to configure a standby for adbmgr. The standby for adbmgr may not be added during the feature verification phase, and will be added later when high availability is verified. On the selected host where the adbmgr standby is deployed, use the antdb user to execute:

pg_basebackup -h 10.20.16.175 -p 6432 -U antdb -D /data/antdb/mgr --
nodename mgr -Xs -Fp -R

Where:
•-h adb05 in adb05 is the hostname of the adbmgr master, where the IP can also be used.
•-p 6432 for the port of the adbmgr master.
•-U antdb means use antdb user to connect to adbmgr master.
•-D specifies the data directory of the adbmgr slave on the execution host, ensuring that the permissions are correct.
•Other parameters remain the same; after execution is complete, start adbmgr slave.

 mgr_ctl start -D /data/antdb/mgr_standby

The value of adbhome here should match the value of ADBHOME in .bashrc.

Add hosts and deploy binaries

1.Add host; add several hosts according to the planning, the cluster is deployed on several hosts.

add host hostname (port=22,protocol='ssh',adbhome='AntDB installation path coresponding to the host]',address='server IP',agentport=agent port number,user=host user);

•port: ssh port number, default 22.
•protocol: login protocol, default ssh, users can leave it out.
•adbhome: installation directory of antdb, the next deploy binary will copy the binary on mgr to this directory.
•address: host IP address (IPV4, IPV6 and domain names are supported).
•agentport: since an agent process will be started on each host, here is the corresponding port number.
•user: ssh login username; For example:

--IPv4
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='10.23.16.227',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='10.23.16.228',agentport=31001,user='uatadb');
 
--IPv6
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='fc00::227',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='fc00::228',agentport=31001,user='uatadb');
 
--doamin name
add host host227(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='adb227.host.com',agentport=31001,user='uatadb');
add host host228(port=22,protocol='ssh',adbhome='/home/uatadb/app',address='adb228.host.com',agentport=31001,user='uatadb');

2.deploy binary program; The deploy command will package the AntDB binary executable files and send it to all hosts in the host table. If mutual trust is not set among the hosts in the cluster, deploy all requires user password (current user's login password), if mutual trust is set among the hosts, the tedious setting of password can be eliminated.

•Deploy all hosts at once:

deploy all password '123456';

•Deploy the specified hosts, for example, deploy the specified hosts adb01 and adb02:

deploy host227,host228 password '123456';

3.Start agent. There are two ways: start all the agents at once and start the specified host agent. Note: password is the linux system password corresponding to the host user in the host table, which is used to communicate with the host, not the user password of the AntDB database. When the password starts with a number, you need to add single quotes or double quotes, for example, password '12345z' is correct, password 12345z will report an error; if the password does not start with a number, you can add quotes or not.

•Start all agents at once:

start agent all password '123456';

•Start the specified agent, multiple hosts need to be executed multiple times, e.g.:

start agent host227,host228 password '123456';
start agent host227 password '123456';

When the password is incorrect and the start of the agent fails, the following error is reported:

postgres=# start agent host227 password '123456abc'; 
hostname | status | description                
------------+--------+------------------------------------------- 
adb01 | f | Authentication failed (username/password)
(1 row) 

Add Nodes

Add node information such as gtmcoord, coordinator, datanode master, datanode slave in Node table. Note: host name must be from host table, port number should not conflict, and the folder specified by path must be empty, otherwise the initialization will fail and error will be reported. This setting is to prevent users from operating and forgetting that there is useful data information under the current node. Add command:

1.add gtmcoord information. The slave node must have a different name from the master node, so the specified master must exist, and the same asynchronous relationship is set by the SYNC_STATE parameter.

add gtmcoord master name(host='localhost3',port=xxx, path='xxx');add gtmcoord slave name for maste_name(host='localhost2',port=xxx, path='xxx'

2.add coordinator node.

add coordinator master name(path = 'xxx', host='localhost1', port=xxx);

3.add datanode master node.

add datanode master name(path = 'xxx', host='localhost1', port=xxx);

4.add datanode slave node add datanode slave information, the slave node must have a different name from the master, so the specified master must exist, and the same asynchronous relationship is set by the SYNC_STATE parameter.

add datanode slave name for master_name (host='localhost2', port=xxx, path='xxx', SYNC_STATE='sync');

After adding, use the command list node to view the information of the node just added. A complete example: (gtmcoord/datanode are one master and one slave)

add host adb01(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.201",agentport=8432,user='antdb');
add host adb02(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.202",agentport=8432,user='antdb'); 
add host adb03(port=22,protocol='ssh',adbhome='/opt/app/antdb',address="10.1.226.203",agentport=8432,user='antdb'); 
deploy all password '123456';
start agent all password '123456';
add gtmcoord master gc_1(host='adb03',port=6655, path='/home/antdb/data/gc_1'); 
add gtmcoord slave gc_2 for gc_1(host='adb02',port=6655,path='/home/antdb/data/gc_2'); 
add coordinator master cn1(host='adb01', port=5432,path = '/home/antdb/data/cn1'); 
add coordinator master cn2(host='adb02', port=5432,path = '/home/antdb/data/cn2'); 
add datanode master dn1_1(host='adb01', port=14332,path = '/home/antdb/data/dn1_1'); 
add datanode slave dn1_2 for dn1_1 (host='db02',port=14332,path='/home/antdb/data/dn1_2'); 
add datanode master dn2_1(host='adb02', port=24332,path = '/home/antdb/data/dn2_1'); 
add datanode slave dn2_2 for dn2_1(host='adb01',port=24332,path='/home/antdb/data/dn2_2'); 

Recommendations on node deployment for production environments:
•At most one coordinator on each host, with the same port for each coordinator.
•Because the application can also connect to gtmcoord, the host with gtmcoord can no longer deploy the coordinator during deployment.
•the ports of the datanode can use the same port on different hosts according to a group of nodes (one master and multiple slaves as a group, e.g. dn1_x).