Deepgreen数据库安装配置详解

    xiaoxiao2023-12-22  24

    可能很多朋友并不是很了解Deepgreen,Deepgreen是大规模并行数据库,他做为Greenplum的升级版,在原Greenplum基础上优化改进并添加了很多的功能,他也是MPP(大规模并行数据库),shared-nothing架构,在数据仓库和数据分析计算领域的前景不可小觑。如果有朋友想更详尽的了解这款数据库,请继续关注我的博客,同时可以通过其官网了解更多内容:http://vitessedata.com/deepgreen-db 今天我们先来看看这款数据库的安装,由于架构的一致性,Deepgreen与Greenplum的安装步骤几乎一致,稍有不同的地方本文用红色/斜体下划线标出。官网上并未放出安装手册,具体可以参见Greenplum的安装手册: 1、测试环境规划: 192.168.100.106 规划为1 Master 3 Segment 内存:126G 硬盘:1T 操作系统:CentOS release 6.5 (Final) 单网卡 192.168.100.107 规划为3 Segment 内存:126G 硬盘:2T 操作系统:CentOS release 6.5 (Final) 单网卡 2、安装操作系统及软件包: CentOS 6.x bash shell GNU tars GNU zip sed 3、关闭selinux # 查看状态: # sestatus SELinuxstatus: disabled # 如果与上面不一致,请编辑文件,并将状态改为disabled # vim /etc/selinux/config SELINUX=disabled 4、关闭iptables service iptables stop 5、配置操作系统参数 vim /etc/sysctl.conf kernel.shmmax = 500000000 kernel.shmmni = 4096 kernel.shmall = 4000000000 kernel.sem = 250 512000 100 2048 kernel.sysrq = 1 kernel.core_uses_pid = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.msgmni = 2048 net.ipv4.tcp_syncookies = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.conf.all.arp_filter = 1 net.ipv4.ip_local_port_range = 10000 65535 net.core.netdev_max_backlog = 10000 net.core.rmem_max = 2097152 net.core.wmem_max = 2097152 vm.overcommit_memory = 2 /etc/security/limits.conf * soft nofile 65536 * hard nofile 65536 * soft nproc 131072 * hard nproc 131072 6、修改磁盘I/O调度器 查看命令:cat /sys/block/sdx/queue/scheduler 临时修改,重启失效 echo deadline > /sys/block/sbx/queue/scheduler 永久修改 vim /boot/grub/grub.conf 增加内核参数elevator = deadline,例如: kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/ elevator=deadline crashkernel=128M@16M quiet console=tty1console=ttyS1,115200 panic=30 transparent_hugepage=neverinitrd /initrd-2.6.18-274.3.1.el5.img 7、设置磁盘的预读扇区(blockdev)值为16384  lsblk -a 查看设备块信息 查看命令: /sbin/blockdev --getra /dev/sdb 设置命令: /sbin/blockdev --setra 16384 /dev/sdb 为防止重启失效,将配置写入/etc/rc.local: echo '/sbin/blockdev --setra 16384 /dev/sda' >> /etc/rc.local 8、关闭THP(Transparent Huge Pages) vim /boot/grub/grub.conf 添加transparent_hugepage参数,例如: kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/elevator=deadline crashkernel=128M@16M quiet console=tty1console=ttyS1,115200 panic=30 transparent_hugepage=never initrd /initrd-2.6.18-274.3.1.el5.img 9、新建dgadmin用户组和用户,这里和Greenplum稍有区别,需要先建立用户,然后将数据库安装在心间用户家目录下面: # useradd dgadmin -r -m # passwd dgadmin 10、上传软件并切换到用户: # su - dgadmin 备注:因为有可能由于权限不足导致出错,不要把deepgreen 安装在/usr/local下。安装到新用户home下,即/home/dgadmin。 11、执行bin文件安装,自动生成vitesse.deepgreendb.16.17文件夹及软连接文件deepgreendb 12、添加初始化变量到dgadmin用户.bashrc文件。 # vim .bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi # User specific aliases and functions source /home/dgadmin/deepgreendb/greenplum_path.sh 使立即生效:source .bashrc 13、使用root用户修改两台主机的host文件,配置两台主机功能如下: 192.168.100.106 ha2 192.168.100.107 linux1 14、root用户新建hostfile_exkeys vim hostfile_exkeys: ha2 linux1 15、root用户临时生效dg命令: source /home/dgadmin/deepgreendb/greenplum_path.sh 16、root用户Exchange keys: # gpssh-exkeys -f /home/dgadmin/dgconfig/hostfile_exkeys [STEP 1 of 5] create local ID and authorize on local host ... /root/.ssh/id_rsa file exists ... key generation skipped [STEP 2 of 5] keyscan all hosts and update known_hosts file [STEP 3 of 5] authorize current user on remote hosts ... send to linux1 [STEP 4 of 5] determine common authentication file content [STEP 5 of 5] copy authentication files to all remote hosts ... finished key exchange with linux1 [INFO] completed successfully 17、root用户安装软件到各个节点: # gpseginstall -f /home/dgadmin/dgconfig/hostfile_exkeys -u dgadmin -p dgadmin 18、切换回dgadmin用户,检查安装是否正确: # gpssh -f dgconfig/hostfile_exkeys -e ls -l $GPHOME 如有问题使用下面命令修复: # gpssh-exkeys -f dgconfig/hostfile_exkeys 19、用root用户执行gpcheckos检查主机参数设置,有问题的地方参考文章开头调整。 20、新建各个节点目录: [dgadmin@linux1 ~]$ exit logout [root@ha2 ~]# mkdir /dgdata [root@ha2 ~]# mkdir /dgdata/master [root@ha2 ~]# mkdir /dgdata/primary [root@ha2 ~]# chown dgadmin /dgdata [root@ha2 ~]# chown dgadmin /dgdata/master [root@ha2 ~]# chown dgadmin /dgdata/primary [root@ha2 ~]# gpssh -h linux1 -e 'mkdir /dgdata' [root@ha2 ~]# gpssh -h linux1 -e 'mkdir /dgdata/primary' [root@ha2 ~]# gpssh -h linux1 -e 'chown dgadmin /dgdata' [root@ha2 ~]# gpssh -h linux1 -e 'chown dgadmin /dgdata/primary' 21、切换到dgadmin用户,创建及修改集群初始化参数文件: [dgadmin@ha2 ~]$ vim cluster.conf [dgadmin@ha2 ~]$ cat dgconfig/cluster.conf # Set this to anything you like ARRAY_NAME="highgo data warehouse" CLUSTER_NAME="highgo data warehouse" # This file must exist in the same directory that you execute gpinitsystem in MACHINE_LIST_FILE=hostfile # This names the data directories for the Segment Instances and the Entry Postmaster SEG_PREFIX=hgdwseg # Prefix for script created database DATABASE_PREFIX=hgdw # This is the port at which to contact the resulting Greenplum database, e.g. # psql -p $PORT_BASE -d template1 PORT_BASE=25432 # Array of data locations for each hosts Segment Instances, the number of directories in this array will # set the number of segment instances per host declare -a DATA_DIRECTORY=(/dgdata/primary /dgdata/primary /dgdata/primary) # Name of host on which to setup the QD MASTER_HOSTNAME=ha2 # Name of directory on that host in which to setup the QD MASTER_DIRECTORY=/dgdata/master MASTER_PORT=15432 # Hosts to allow to connect to the QD (and Segment Instances) # By default, allow everyone to connect (0.0.0.0/0) IP_ALLOW=0.0.0.0/0 # Shell to use to execute commands on all hosts TRUSTED_SHELL=/usr/bin/ssh CHECK_POINT_SEGMENTS=8 ENCODING=UNICODE export MASTER_DATA_DIRECTORY export TRUSTED_SHELL # Keep max_connection settings to reasonable values for # installcheck good execution. DEFAULT_QD_MAX_CONNECT=25 QE_CONNECT_FACTOR=5 # mirror segment conf #MIRROR_PORT_BASE=50000 #REPLICATION_PORT_BASE=41000 #MIRROR_REPLICATION_PORT_BASE=51000 #declare -a MIRROR_DATA_DIRECTORY=(/dgdata/mirror) 22、初始化系统: [dgadmin@ha2 ~]$ gpinitsystem -c cluster.conf 23、添加环境变量到: vim greenplum_path.sh export MASTER_DATA_DIRECTORY=/dgdata/master/dg-1 export PGPORT=15432 24、检查系统状态: gpstate 25、至此,一个Master,六个Segment的集群搭建完毕。
    最新回复(0)