ceph集群搭建Jewel版本

阅读量：5789 次

发布时间：2019-06-18

本文共 8672 字，大约阅读时间需要 28 分钟。

记录一下Ceph集群搭建的过程。

Ceph简介

Ceph 是统一分布式存储系统，具有优异的性能、可靠性、可扩展性。Ceph 的底层是 RADOS(可靠、自动、分布式对象存储)，可以通过 LIBRADOS 直接访问到 RADOS 的对象存储系统。RBD(块设备接口)、RADOS Gateway(对象存储接口)、Ceph File System(POSIX 接口)都是基于 RADOS 的。

CEPH快速部署(Centos7+Jewel)

环境

1. 三台装有 CentOS 7 的主机，每台主机有三个磁盘(虚拟机磁盘要大于100G),详细信息如下：

[root@ceph-1 ~]# cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core)[root@ceph-1 ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0 128G 0 disk├─sda1 8:1 0 500M 0 part /boot└─sda2 8:2 0 127.5G 0 part├─centos-root 253:0 0 50G 0 lvm /├─centos-swap 253:1 0 2G 0 lvm [SWAP]└─centos-home 253:2 0 75.5G 0 lvm /homesdb 8:16 0 2T 0 disksdc 8:32 0 2T 0 disksdd 8:48 0 2T 0 disksr0 11:0 1 1024M 0 rom[root@ceph-1 ~]# cat /etc/hosts..192.168.57.222 ceph-1192.168.57.223 ceph-2192.168.57.224 ceph-3

2. 集群配置如下：


主机	ip	功能
ceph-1	192.168.57.222	deploy \| mon1 \| osd3
ceph-2	192.168.57.223	mon1 \| osd3
ceph-3	192.168.57.224	mon1 \| osd3

环境清理！

如果之前部署失败了，不必删除ceph客户端，或者重新搭建虚拟机，只需要在每个节点上执行如下指令即可将环境清理至刚安装完ceph客户端时的状态！强烈建议在旧集群上搭建之前清理干净环境，否则会发生各种异常情况。

ps aux|grep ceph |awk '{print $2}'|xargs kill -9ps aux|grep ceph |awk '{print $2}'|xargs kill -9ps -ef|grep ceph#确保此时所有ceph进程都已经关闭！！！如果没有关闭，多执行几次。umount /var/lib/ceph/ osd /*rm -rf /var/lib/ceph/ osd /*rm -rf /var/lib/ceph/ mon /*rm -rf /var/lib/ceph/ mds /*rm -rf /var/lib/ceph/bootstrap- mds /*rm -rf /var/lib/ceph/bootstrap- osd /*rm -rf /var/lib/ceph/bootstrap- mon /*rm -rf /var/lib/ceph/ tmp /*rm -rf /etc/ceph/*rm -rf /var/run/ceph/*

yum源及ceph的安装

需要在每个主机上执行以下指令

yum clean allrm -rf /etc/yum.repos.d/*.repowget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repowget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.reposed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.reposed -i 's/$releasever/7.2.1511/g' /etc/yum.repos.d/CentOS-Base.repo

增加ceph的源

vim /etc/yum.repos.d/ceph.repo

添加以下内容：

[ceph]name=cephbaseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/gpgcheck=0[ceph-noarch]name=cephnoarchbaseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/gpgcheck=0

安装ceph客户端：

yum makecacheyum install ceph ceph-radosgw rdate -y

关闭 selinux & firewalld

sed -i 's#SELINUX=.*#SELINUX=disabled#' /etc/selinux/config        #.*是代表所有的意思setenforce 0systemctl stop firewalldsystemctl disable firewalld

同步各个节点时间：

yum -y install rdaterdate -s time-a.nist.govecho rdate -s time-a.nist.gov >> /etc/rc.d/rc.localchmod +x /etc/rc.d/rc.local

开始部署

在部署节点( ceph-1 )安装ceph-deploy，下面的部署节点统一指ceph-1:

[root@ceph-1 ~]# yum -y install ceph-deploy[root@ceph-1 ~]# ceph-deploy --version1.5.34[root@ceph-1 ~]# ceph -vceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)

在部署节点创建部署目录并开始部署：

[root@ceph-1 ~]# cd[root@ceph-1 ~]# mkdir cluster[root@ceph-1 ~]# cd cluster/[root@ceph-1 cluster]# ceph-deploy new ceph-1 ceph-2 ceph-3

如果之前没有 ssh-copy-id 到各个节点，则需要输入一下密码，过程log如下：

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf[ceph_deploy.cli][INFO ] Invoked (1.5.34): /usr/bin/ceph-deploy new ceph-1 ceph-2 ceph-3[ceph_deploy.cli][INFO ] ceph-deploy options:[ceph_deploy.cli][INFO ] username : None[ceph_deploy.cli][INFO ] func : 
     
      [ceph_deploy.cli][INFO ] verbose : False[ceph_deploy.cli][INFO ] overwrite_conf : False[ceph_deploy.cli][INFO ] quiet : False[ceph_deploy.cli][INFO ] cd _conf : 
      
       [ceph_deploy.cli][INFO ] cluster : ceph[ceph_deploy.cli][INFO ] ssh_copykey : True[ceph_deploy.cli][INFO ] mon : ['ceph-1', 'ceph-2', 'ceph-3']....ceph_deploy.new][WARNIN] could not connect via SSH[ceph_deploy.new][INFO ] will connect again with password promptThe authenticity of host 'ceph-2 (192.168.57.223)' can't be established.ECDSA key fingerprint is ef:e2:3e:38:fa:47:f4:61:b7:4d:d3:24:de:d4:7a:54.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'ceph-2,192.168.57.223' (ECDSA) to the list of known hosts.rootroot@ceph-2's password:[ceph-2][DEBUG ] connected to host: ceph-2....[ceph_deploy.new][DEBUG ] Resolving host ceph-3[ceph_deploy.new][DEBUG ] Monitor ceph-3 at 192.168.57.224[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-1', 'ceph-2', 'ceph-3'][ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.57.222', '192.168.57.223', '192.168.57.224'][ceph_deploy.new][DEBUG ] Creating a random mon key...[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

此时，目录内容如下：

[root@ceph-1 cluster]# lsceph.conf ceph-deploy-ceph.log ceph.mon.keyring

根据自己的IP配置向ceph.conf中添加 public_network ，并稍微增大mon之间时差允许范围(默认为0.05s，现改为2s)：

[root@ceph-1 cluster]# echo public_network=192.168.57.0/24 >> ceph.conf[root@ceph-1 cluster]# echo mon_clock_drift_allowed = 2 >> ceph.conf[root@ceph-1 cluster]# cat ceph.conf[global]fsid = 0248817a-b758-4d6b-a217-11248b098e10mon_initial_members = ceph-1, ceph-2, ceph-3mon_host = 192.168.57.222,192.168.57.223,192.168.57.224auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephxpublic_network=192.168.57.0/24mon_clock_drift_allowed = 2

开始部署monitor:

[root@ceph-1 cluster]# ceph-deploy mon create-initial....若干log[root@ceph-1 cluster]# lsceph.bootstrap-mds.keyring ceph.bootstrap-rgw.keyring ceph.conf ceph.mon.keyringceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log

查看集群状态：

[root@ceph-1 cluster]# ceph -scluster 0248817a-b758-4d6b-a217-11248b098e10health HEALTH_ERRno osdsMonitor clock skew detectedmonmap e1: 3 mons at {ceph-1=192.168.57.222:6789/0,ceph-2=192.168.57.223:6789/0,ceph-3=192.168.57.224:6789/0}election epoch 6, quorum 0,1,2 ceph-1,ceph-2,ceph-3osdmap e1: 0 osds: 0 up, 0 inflags sortbitwisepgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects0 kB used, 0 kB / 0 kB avail64 creating

开始部署OSD:

ceph-deploy --overwrite-conf osd prepare ceph-1:/dev/sdb ceph-1:/dev/sdc ceph-1:/dev/sdd ceph-2:/dev/sdb ceph-2:/dev/sdcceph-2:/dev/sdd ceph-3:/dev/sdb ceph-3:/dev/sdc ceph-3:/dev/sdd --zap-disk

我在部署的时候出了个小问题，有一个OSD没成功(待所有OSD部署完毕后，再重新部署问题OSD即可解决)，如果不出意外的话，集群状态应该如下：

[root@ceph-1 cluster]# ceph -scluster 0248817a-b758-4d6b-a217-11248b098e10health HEALTH_WARNtoo few PGs per OSD (21 < min 30)monmap e1: 3 mons at {ceph-1=192.168.57.222:6789/0,ceph-2=192.168.57.223:6789/0,ceph-3=192.168.57.224:6789/0}election epoch 22, quorum 0,1,2 ceph-1,ceph-2,ceph-3osdmap e45: 9 osds: 9 up, 9 inflags sortbitwisepgmap v82: 64 pgs, 1 pools, 0 bytes data, 0 objects273 MB used, 16335 GB / 16336 GB avail64 active+clean

去除这个WARN，只需要增加rbd池的PG就好：

[root@ceph-1 cluster]# ceph osd pool set rbd pg_num 128set pool 0 pg_num to 128[root@ceph-1 cluster]# ceph osd pool set rbd pgp_num 128set pool 0 pgp_num to 128[root@ceph-1 cluster]# ceph -scluster 0248817a-b758-4d6b-a217-11248b098e10health HEALTH_ERR19 pgs are stuck inactive for more than 300 seconds12 pgs peering19 pgs stuck inactivemonmap e1: 3 mons at {ceph-1=192.168.57.222:6789/0,ceph-2=192.168.57.223:6789/0,ceph-3=192.168.57.224:6789/0}election epoch 22, quorum 0,1,2 ceph-1,ceph-2,ceph-3osdmap e49: 9 osds: 9 up, 9 inflags sortbitwisepgmap v96: 128 pgs, 1 pools, 0 bytes data, 0 objects308 MB used, 18377 GB / 18378 GB avail103 active+clean12 peering9 creating4 activating[root@ceph-1 cluster]# ceph -scluster 0248817a-b758-4d6b-a217-11248b098e10health HEALTH_OKmonmap e1: 3 mons at {ceph-1=192.168.57.222:6789/0,ceph-2=192.168.57.223:6789/0,ceph-3=192.168.57.224:6789/0}election epoch 22, quorum 0,1,2 ceph-1,ceph-2,ceph-3osdmap e49: 9 osds: 9 up, 9 inflags sortbitwisepgmap v99: 128 pgs, 1 pools, 0 bytes data, 0 objects310 MB used, 18377 GB / 18378 GB avail128 active+clean

至此，集群部署完毕。

config推送

不要使用直接修改某个节点的 /etc/ceph/ceph.conf 文件的方式，而是去部署节点(此处为 ceph-1:/root/cluster/ceph.conf )目录下修改。因为节点

到几十个的时候，不可能一个个去修改的，采用推送的方式快捷安全！

修改完毕后，执行如下指令，将conf文件推送至各个节点：

[root@ceph-1 cluster]# ceph-deploy --overwrite-conf config push ceph-1 ceph-2 ceph-3

此时，需要重启各个节点的 monitor 服务

mon&osd启动方式

#monitor start/stop/restart#ceph-1为各个monitor所在节点的主机名。systemctl start ceph-mon@ceph-1.servicesystemctl restart ceph-mon@ceph-1.servicesystemctl stop ceph-mon@ceph-1.service#OSD start/stop/restart#0为该节点的OSD的id，可以通过`ceph osd tree`查看systemctl start/stop/restart ceph-osd@0.service[root@ceph-1 cluster]# ceph osd treeID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY-1 17.94685 root default-2 5.98228 host ceph-10 1.99409 osd.0 up 1.00000 1.000001 1.99409 osd.1 up 1.00000 1.000008 1.99409 osd.2 up 1.00000 1.00000-3 5.98228 host ceph-22 1.99409 osd.3 up 1.00000 1.000003 1.99409 osd.4 up 1.00000 1.000004 1.99409 osd.5 up 1.00000 1.00000-4 5.98228 host ceph-35 1.99409 osd.6 up 1.00000 1.000006 1.99409 osd.7 up 1.00000 1.000007 1.99409 osd.8 up 1.00000 1.00000

转载于:https://www.cnblogs.com/ningchenghui/p/9045425.html

你可能感兴趣的文章

实现hive proxy1-hive认证实现

查看>>

LinuxShell脚本之利用rsync+ssh实现Linux文件系统远程备份

查看>>

设计和使用维护计划

查看>>

Hyper-V 2016 系列教程3 Hyper-V 组件的添加

查看>>

func install in ubuntu-server

查看>>