arrow-left arrow-right brightness-2 chevron-left chevron-right circle-half-full facebook-box facebook loader magnify menu-down rss-box star twitter-box twitter white-balance-sunny window-close
How to integrate Ceph with OpenStack Object Storage on a Kolla deployment
11 min read

How to integrate Ceph with OpenStack Object Storage on a Kolla deployment

How to integrate Ceph with OpenStack Object Storage on a Kolla deployment

Integration of Ceph RadosGW with Keystone in an OpenStack deployment.

On my OpenStack deployment I like to use Ceph as backend storage. The integration is smooth, straight forward and work with lot of different OpenStack components.

On this post I'll focus on the integration of Ceph as Object storage backend, deployed with kolla-ansible. Because I love kolla-ansible šŸ’•šŸ’•!

Before you start

ā—ā—ā— Note that the deployment of Ceph by kolla-ansible is a deprecated feature since Train and totally removed in the Ussuri release.ā—ā—ā—

This documentation will not work with versions after Train, more informations here.

The deployment of OpenStack with kolla-ansible and an external Ceph cluster will be the subject of a next post.

Since only few people are already using Ussuri or even the latest stable version I think this documentation still could be relevant for a moment šŸ™‚

Requirements

I'll not explain how to deploy OpenStack with kolla/kolla-ansible.
You need to have a good knowledge about this type of deployment (virtualenv, etc) and an infrastructure ready to be deployed.

Notes

On my deployment for this demo, I'll use 6 nodes (3 controllers, 3 computes) and a node for Kolla. It's a very light lab for sandbox and demo purpose, but neat and deployed like a production environment.

(openstack) [[email protected] ~] $ openstack hypervisor list --long
+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP      | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |
+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+
|  6 | openstack-compute-1 | QEMU            | 172.16.11.21 | up    |          0 |     2 |            512 |      4095 |
|  9 | openstack-compute-2 | QEMU            | 172.16.11.22 | up    |          0 |     2 |            512 |      2047 |
| 12 | openstack-compute-3 | QEMU            | 172.16.11.23 | up    |          1 |     2 |           1024 |      2047 |
+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+
(openstack) [[email protected] ~] $ openstack volume service list
+------------------+---------------------------+------+---------+-------+----------------------------+
| Binary           | Host                      | Zone | Status  | State | Updated At                 |
+------------------+---------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | openstack-controller-1    | nova | enabled | up    | 2020-04-11T02:19:51.000000 |
| cinder-scheduler | openstack-controller-3    | nova | enabled | up    | 2020-04-11T02:19:47.000000 |
| cinder-volume    | [email protected] | nova | enabled | up    | 2020-04-11T02:19:47.000000 |
| cinder-backup    | openstack-compute-1       | nova | enabled | up    | 2020-04-11T02:19:46.000000 |
| cinder-volume    | [email protected] | nova | enabled | up    | 2020-04-11T02:19:52.000000 |
| cinder-backup    | openstack-compute-3       | nova | enabled | up    | 2020-04-11T02:19:52.000000 |
| cinder-backup    | openstack-compute-2       | nova | enabled | up    | 2020-04-11T02:19:48.000000 |
| cinder-scheduler | openstack-controller-2    | nova | enabled | up    | 2020-04-11T02:19:55.000000 |
| cinder-volume    | [email protected] | nova | enabled | up    | 2020-04-11T02:19:48.000000 |
+------------------+---------------------------+------+---------+-------+----------------------------+
  • I use a VMWare hypervisor
  • I have a pfsense that does the firewalling, the dhcp+dns+ntp
  • Each compute node have 3 additionnal dedicated disks for the OSD, sdc, sdd and sde, each OSDs has a size of 20 GB with full flash storage
  • The external floating API IP is 172.16.10.10 and internal floating IP API is 172.16.11.10
  • On my nodes the storage_interface is on the subnet 172.16.12.0/24 and the dedicated nic is ens38, the cluster_interface is on the subnet 172.16.13.0/24 and the dedicated nic is ens39
  • The api_interface (internal) is on the subnet 172.16.11.0/24

I take advantage of the virtualization to have separate network interfaces with dedicated virtual network card for each network on each node:

 WAN		            1000baseT <full-duplex>	192.168.0.135
 MANAGEMENT		        1000baseT <full-duplex>	172.16.10.254
 KOLLA_API_INT		    1000baseT <full-duplex>	172.16.11.254
 KOLLA_STORAGE_INT	    1000baseT <full-duplex>	172.16.12.254
 KOLLA_CLUSTER_INT      1000baseT <full-duplex>	172.16.13.254
 KOLLA_TUNNEL_INT	    1000baseT <full-duplex>	172.16.14.254
 KOLLA_PUBLIC		    1000baseT <full-duplex>	172.16.15.254

The classic Swift terminology for a bucket is a container. And I'm calling that a container for years, I'm sorry, it has nothing to do with a Docker container.

Will we use Ceph as Swift backend? Nope!

I've heard a lot that Ceph is used as a backend storage for Swift. In fact the integration is different than other components. For example if you integrate Ceph with Nova (compute), you'll indeed use Ceph as backend storage for your ephemeral storage with RBD: Nova will talk with Ceph, and Ceph will present a block that Nova will use as disk for his ephemeral storage. But Nova is still the compute component.

With Ceph object storage integration you will use... Ceph. Swift will not be deployed at all. You'll take advantage of the Swift API compatibility on the Rados Gateway. And also take advantage of the S3 API compatibility if you want. You'll use Ceph and the Swift API, but not Swift itself.

An end user will not see any difference from a vanilla Swift deployment, he'll still use the Swift API, still authenticate with Keystone.

Here is a global example of Ceph integration with OpenStack:

Basic design and architecture

In high workload environment you will need high speed network interfaces to avoid any bottleneck and data consistency issue. It's always a good idea to have a logical (vlan) and sometimes physical separation (dedicated networks cards, dedicated switches) of your control plane and data plane. In some case you will also want to separate completely the trafic related to storage operations.

Usually it's a good idea to have at least 2 bonds, one for control plane and one for data plane.

I've already deployed some OpenStack with separation of the control plane, data plane and storage trafic, even some with data replication and data presentation logically and physically separated.

But on some deployment like this one I have only one physical bond or one physical nic for everything and it's enough, no stress.

Every deployment is different from one to another, depending on criticity, security need, workload and budget. Anyway this topic about design and architecture will need a complete separate post.

Prepare the Ceph deployment configuration

Edit your inventory to include the storage nodes. Put all the nodes that have dedicated devices for Ceph. If you want to deploy the Ceph cluster in a HCI way just put your compute nodes, in this example I use the 3 nodes, from compute01 to compute03:

[storage]
compute[01:03]

The control plane part will be automatically deployed on your controllers.
As I said the deployment is really simple. Just edit your globals.yml file:

enable_ceph: "yes"
enable_ceph_rgw: "yes"
enable_ceph_rgw_keystone: "yes"

You can take advantage of kolla-ansible deployment if you have dedicated network cards for the storage data plane. Kolla has 2 distinct networks for the storage: cluster_interface that is used for data replication, and storage_interface that is use by the VM to communicate with Ceph. For me:

storage_interface: "ens38"
cluster_interface: "ens39"

If you use only one interface you can just leave the default {{ network_interface }}.

Prepare the disks for OSD bootstraping

On deployment kolla will scan all disks and will bootstrap every disks with this specific label: KOLLA_CEPH_OSD_BOOTSTRAP_BS.
Since Queens release, bluestore is the default storage type (_BS).

In this example, I have 3 disks (sdc, sdd and sde) on each storage hosts (it will be an hyperconverged deployment with the osd on compute node):

parted /dev/sdc -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1
parted /dev/sdd -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1
parted /dev/sde -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1

Since I use full flash storage I do not particularly need dedicated device for each bluestore partition but you can separate the bluestore OSD block partition, the bluestore OSD block.wal and the bluestore OSD block.db partition. This is useful if you have spinning hard drive for data and SSD for WAL and DB.
For a WAL device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_W 1 -1

For a DB device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_D 1 -1

For a block device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_B 1 -1

If you do this you cannot use the KOLLA_CEPH_OSD_BOOTSTRAP_BS label and you need to use theses 3 separates label instead.

Deploy

Use the classical commands to deploy:

(openstack) [[email protected] ~] $ kolla-ansible -i ./inventory_file deploy

And wait until the end and this beautiful result that we love...

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
compute01                  : ok=137  changed=18   unreachable=0    failed=0    skipped=77   rescued=0    ignored=0
compute02                  : ok=131  changed=18   unreachable=0    failed=0    skipped=64   rescued=0    ignored=0
compute03                  : ok=131  changed=18   unreachable=0    failed=0    skipped=64   rescued=0    ignored=0
control01                  : ok=399  changed=66   unreachable=0    failed=0    skipped=309  rescued=0    ignored=0
control02                  : ok=300  changed=47   unreachable=0    failed=0    skipped=219  rescued=0    ignored=0
control03                  : ok=302  changed=49   unreachable=0    failed=0    skipped=218  rescued=0    ignored=0
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check your deployment

On my lab deployment I said that I've 3 controllers and 3 computes node. 3 OSDs are running on each compute nodes.
At first, check the health status of your Ceph cluster:

[[email protected] ~] $ docker exec -it ceph_mon ceph -s
  cluster:
    id:     66d428fe-f859-4769-b66e-c40e8ba3ada2
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum 172.16.12.11,172.16.12.12,172.16.12.13 (age 33m)
    mgr: control01(active, since 6w), standbys: control03, control02
    mds: cephfs:1 {0=control03=up:active} 2 up:standby
    osd: 9 osds: 9 up (since 3m), 9 in (since 3m)
    rgw: 1 daemon active (radosgw.gateway)

  data:
    pools:   12 pools, 144 pgs
    objects: 1.14k objects, 5.1 GiB
    usage:   23 GiB used, 156 GiB / 179 GiB avail
    pgs:     144 active+clean

  io:
    client:   2.2 KiB/s rd, 2 op/s rd, 0 op/s wr

I see that my 9 OSDs are up, I have an active rgw daemon and everything is working, the important part is health: HEALTH OK.

On one of my compute node I see my 3 OSDs:

[[email protected] ~] $ docker ps | grep ceph
d2e45b33894e        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         up 4 minutes                        ceph_osd_0
df5517709135        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                        ceph_osd_5
dd6525c658c5        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                        ceph_osd_7

And on one of my controller:

[[email protected] ~] $ docker ps | grep ceph
7c2fd10681ee        kolla/centos-source-ceph-rgw:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_rgw
db90351e0d13        kolla/centos-source-ceph-mgr:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_mgr
4ff360078c26        kolla/centos-source-ceph-mon:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_mon

You see now the object storage endpoint (which is the Ceph endpoint):

(openstack) [[email protected] ~] $ openstack endpoint list | grep object
| 1be42ae9c8ee43edbb7e4bf73eb9f958 | RegionOne | swift        | object-store   | True    | public    | http://172.16.10.10:6780/swift/v1         |
| 3cfcfe9eed924d7a9acf4d86761be6cb | RegionOne | swift        | object-store   | True    | internal  | http://172.16.11.10:6780/swift/v1         |
| c279c8bda5694eb19d4e6c0924a0e6fd | RegionOne | swift        | object-store   | True    | admin     | http://172.16.11.10:6780/swift/v1         |

A look to the configuration

On one of the controller:

[[email protected] ~] $ cat /etc/kolla/ceph-rgw/ceph.conf
[global]
log file = /var/log/kolla/ceph/$cluster-$name.log
log to syslog = false
err to syslog = false
log to stderr = false
err to stderr = false
fsid = 66d428fe-f859-4769-b66e-c40e8ba3ada2
mon initial members = 172.16.12.11, 172.16.12.12, 172.16.12.13
mon host = 172.16.12.11, 172.16.12.12, 172.16.12.13
mon addr = 172.16.12.11:6789, 172.16.12.12:6789, 172.16.12.13:6789
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
setuser match path = /var/lib/ceph/$type/$cluster-$id
osd crush update on start = false

[mon]
mon compact on start = true
mon cluster log file = /var/log/kolla/ceph/$cluster.log

[client.radosgw.gateway]
host = 172.16.12.11
rgw frontends = civetweb port=172.16.11.11:6780
rgw_keystone_url = http://172.16.11.10:35357
rgw_keystone_admin_user = ceph_rgw
rgw_keystone_admin_password = SoJqMgzw9L7ulMO2OIqdIeF2mQPQvuLxUwSIRc11
rgw_keystone_admin_project = service
rgw_keystone_admin_domain = default
rgw_keystone_api_version = 3
rgw_keystone_accepted_roles = admin, _member_
rgw_keystone_accepted_admin_roles = ResellerAdmin
rgw_swift_versioning_enabled = true
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/kolla/ceph/client.radosgw.gateway.log

You see all the interesting configuration under [client.radosgw.gateway]. You have the Keystone endpoint admin authentication URL, the frontend endpoint, the different default roles that will be used, etc...

Enable Keystone S3 authentication

You want to use S3 and Keystone authentication? No problem! Thanks to the incredibly easy way to customize configuration with kolla-ansible just do on your deployment node:

(openstack) [[email protected] ~] $ mkdir -p /etc/kolla/config
(openstack) [[email protected] ~] $ crudini --set /etc/kolla/config/ceph.conf client.radosgw.gateway rgw_s3_auth_use_keystone true

then reconfigure:

(openstack) [[email protected] ~] $ kolla-ansible -i multinode reconfigure --tags ceph

If you look now the /etc/kolla/ceph-rgw/ceph.conf you will see this line at the end of the file:

rgw_s3_auth_use_keystone = true

Note that this deployment is very simple, I don't use SSL and wildcard DNS domain, so I cannot use S3-style subdomains.

Test it!

Swift integration

I'll test the creation and some operations with Swift: creation of a container demo, uploading a file (here a 512 MB Debian image named debian-10-openstack-amd64.qcow2), list the objects and display information about the swift container :

(openstack) [[email protected] ~] $ swift list demo --lh
Container u'demo' not found
(openstack) [[email protected] ~] $ swift post demo
(openstack) [[email protected] ~] $ swift upload demo debian-10-openstack-amd64.qcow2 
(openstack) [[email protected] ~] $ swift list demo --lh
512M 2020-04-11 00:54:21                     None debian-10-openstack-amd64.qcow2
(openstack) [[email protected] ~] $ swift stat -v demo --lh
                          URL: http://172.16.10.10:6780/swift/v1/demo
                   Auth Token: gAAAAABekRZFlKB4cPmDkJsqpTqFILPSgPtOIXEzRzd01iZwMLg5Vq_WDBMPM-fIfu9f78w5S3i5EVYCosf1hJQ-X4YLJtwwpq6S-ZgqwUkK66SYKg02dbQT52rcxA3CuOsz3tuCebHrEq_KtSp2E9oKwD3vtJBvH-Zxpgwp5inVb-ml3QQhAbU
                      Account: v1
                    Container: demo
                      Objects: 2
                        Bytes: 512M
                     Read ACL:
                    Write ACL:
                      Sync To:
                     Sync Key:
              X-Storage-Class: STANDARD
                Accept-Ranges: bytes
             X-Storage-Policy: default-placement
X-Container-Bytes-Used-Actual: 537448448
                Last-Modified: Sat, 11 Apr 2020 00:54:09 GMT
                  X-Timestamp: 1586566161.46193
                   X-Trans-Id: tx00000000000000000000d-005e911645-fa7a-default
                 Content-Type: text/plain; charset=utf-8
       X-Openstack-Request-Id: tx00000000000000000000d-005e911645-fa7a-default

We see that our object is here in the demo container, and the container details show now the total size, Bytes: 512M.

S3 integration

For this example I'll use s3cmd command line to test the integration.
At first generate your credentials to get your access and secret keys:

(openstack) [[email protected] ~] $ openstack ec2 credentials create
+------------+--------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                |
+------------+--------------------------------------------------------------------------------------------------------------------------------------+
| access     | df2d93542ca243098f10033b62817635                                                                                                     |
| links      | {u'self': u'http://172.16.10.10:5000/v3/users/cd21a07450d64f3a9b277670d2e6d3f7/credentials/OS-EC2/df2d93542ca243098f10033b62817635'} |
| project_id | 993ff62625a84916bddd820a97e31cef                                                                                                     |
| secret     | 1f7da7ae6d5b455880832c052d1f49ae                                                                                                     |
| trust_id   | None                                                                                                                                 |
| user_id    | cd21a07450d64f3a9b277670d2e6d3f7                                                                                                     |
+------------+--------------------------------------------------------------------------------------------------------------------------------------+

Then configure s3cmd with the access and secret keys and use the Ceph endpoint, here I'll use the external API endpoint http://172.16.10.10:6780.

This is how my .s3cfg looks like:

(openstack) [[email protected] ~] $ cat ~/.s3cfg
[default]$
access_key = df2d93542ca243098f10033b62817635
secret_key = 1f7da7ae6d5b455880832c052d1f49ae
host_base = http://172.16.10.10:6780
host_bucket = http://172.16.10.10:6780

Now you can test:

(openstack) [[email protected] ~] $ s3cmd ls S3://demo --no-ssl
2020-04-11 00:54    537442816  s3://demo/debian-10-openstack-amd64.qcow2

Horizon

Now if we look at Horizon we see that we have exactly the same thing as if we had deployed Swift :

How it's look on Ceph

List the pools:

[[email protected] ~] $ docker exec -it ceph_mon rados lspools
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
cephfs_data
cephfs_metadata
images
volumes
backups
vms
default.rgw.buckets.index
default.rgw.buckets.data

On my deployment, in addition to object storage integration I've already integrated Nova (compute), Glance (image) and Cinder (block storage), that's why I have a vms (Nova), images (Glance), volumes and backup (Cinder and Cinder Backup) pools.

The interesting pools here will be default.rgw.buckets.index and default.rgw.buckets.data.

Remember that we have a demo container with 512 MB debian-10-openstack-amd64.qcow2 file.

If we look at the default.rgw.buckets.index pools we will see our container:

[[email protected] ~] $ docker exec -ti ceph_mon rados ls -p default.rgw.buckets.index
.dir.ee9f3b14-4996-45f5-9fee-f8f191ee2d18.64239.1

The container id is ee9f3b14-4996-45f5-9fee-f8f191ee2d18.64239.1.
If we look into the default.rgw.buckets.data:

[[email protected] ~] $ docker exec -ti ceph_mon rados ls -p default.rgw.buckets.data | grep -v shadow
ee9f3b14-4996-45f5-9fee-f8f191ee2d18.64239.1_debian-10-openstack-amd64.qcow2

we see our object!

And it's done!

That's it, now you know how to integrate a Ceph cluster to use object storage on OpenStack: authenticate with Keystone and do operation with both Swift and S3!

Don't forget to delete your deployment, --yes-i-really-really-mean-it šŸ˜‰

Resources

https://docs.ceph.com/docs/dumpling/start/quick-rgw/
https://docs.openstack.org/kolla-ansible/rocky/reference/ceph-guide.html
https://docs.openstack.org/python-swiftclient/ocata/cli.html
https://docs.openstack.org/kolla-ansible/latest/
https://docs.openstack.org/train/
https://s3tools.org/s3cmd
And my experience šŸ˜‰

Feel free to correct me if you see any typo or if something seems wrong to you.
You can send me an email or comment below.

Picture : JOSHUA COLEMAN