11 min read

How to integrate Ceph with OpenStack Object Storage on a Kolla deployment

How to integrate Ceph with OpenStack Object Storage on a Kolla deployment

Integration of Ceph RadosGW with Keystone in an OpenStack deployment.

On my OpenStack deployment I like to use Ceph as backend storage. The integration is smooth, straight forward and work with lot of different OpenStack components.

On this post I'll focus on the integration of Ceph as Object storage backend, deployed with kolla-ansible. Because I love kolla-ansible šŸ’•šŸ’•!

Before you start

ā—ā—ā— Note that the deployment of Ceph by kolla-ansible is a deprecated feature since Train and totally removed in the Ussuri release.ā—ā—ā—

This documentation will not work with versions after Train, more informations here.

The deployment of OpenStack with kolla-ansible and an external Ceph cluster will be the subject of a next post.

Since only few people are already using Ussuri or even the latest stable version I think this documentation still could be relevant for a moment šŸ™‚


I'll not explain how to deploy OpenStack with kolla/kolla-ansible.
You need to have a good knowledge about this type of deployment (virtualenv, etc) and an infrastructure ready to be deployed.


On my deployment for this demo, I'll use 6 nodes (3 controllers, 3 computes) and a node for Kolla. It's a very light lab for sandbox and demo purpose, but neat and deployed like a production environment.

(openstack) [openstack@centos-kolla ~] $ openstack hypervisor list --long
| ID | Hypervisor Hostname | Hypervisor Type | Host IP      | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |
|  6 | openstack-compute-1 | QEMU            | | up    |          0 |     2 |            512 |      4095 |
|  9 | openstack-compute-2 | QEMU            | | up    |          0 |     2 |            512 |      2047 |
| 12 | openstack-compute-3 | QEMU            | | up    |          1 |     2 |           1024 |      2047 |
(openstack) [openstack@centos-kolla ~] $ openstack volume service list
| Binary           | Host                      | Zone | Status  | State | Updated At                 |
| cinder-scheduler | openstack-controller-1    | nova | enabled | up    | 2020-04-11T02:19:51.000000 |
| cinder-scheduler | openstack-controller-3    | nova | enabled | up    | 2020-04-11T02:19:47.000000 |
| cinder-volume    | openstack-compute-1@rbd-1 | nova | enabled | up    | 2020-04-11T02:19:47.000000 |
| cinder-backup    | openstack-compute-1       | nova | enabled | up    | 2020-04-11T02:19:46.000000 |
| cinder-volume    | openstack-compute-3@rbd-1 | nova | enabled | up    | 2020-04-11T02:19:52.000000 |
| cinder-backup    | openstack-compute-3       | nova | enabled | up    | 2020-04-11T02:19:52.000000 |
| cinder-backup    | openstack-compute-2       | nova | enabled | up    | 2020-04-11T02:19:48.000000 |
| cinder-scheduler | openstack-controller-2    | nova | enabled | up    | 2020-04-11T02:19:55.000000 |
| cinder-volume    | openstack-compute-2@rbd-1 | nova | enabled | up    | 2020-04-11T02:19:48.000000 |
  • I use a VMWare hypervisor
  • I have a pfsense that does the firewalling, the dhcp+dns+ntp
  • Each compute node have 3 additionnal dedicated disks for the OSD, sdc, sdd and sde, each OSDs has a size of 20 GB with full flash storage
  • The external floating API IP is and internal floating IP API is
  • On my nodes the storage_interface is on the subnet and the dedicated nic is ens38, the cluster_interface is on the subnet and the dedicated nic is ens39
  • The api_interface (internal) is on the subnet

I take advantage of the virtualization to have separate network interfaces with dedicated virtual network card for each network on each node:

 WAN		            1000baseT <full-duplex>
 MANAGEMENT		        1000baseT <full-duplex>
 KOLLA_API_INT		    1000baseT <full-duplex>
 KOLLA_STORAGE_INT	    1000baseT <full-duplex>
 KOLLA_CLUSTER_INT      1000baseT <full-duplex>
 KOLLA_TUNNEL_INT	    1000baseT <full-duplex>
 KOLLA_PUBLIC		    1000baseT <full-duplex>

The classic Swift terminology for a bucket is a container. And I'm calling that a container for years, I'm sorry, it has nothing to do with a Docker container.

Will we use Ceph as Swift backend? Nope!

I've heard a lot that Ceph is used as a backend storage for Swift. In fact the integration is different than other components. For example if you integrate Ceph with Nova (compute), you'll indeed use Ceph as backend storage for your ephemeral storage with RBD: Nova will talk with Ceph, and Ceph will present a block that Nova will use as disk for his ephemeral storage. But Nova is still the compute component.

With Ceph object storage integration you will use... Ceph. Swift will not be deployed at all. You'll take advantage of the Swift API compatibility on the Rados Gateway. And also take advantage of the S3 API compatibility if you want. You'll use Ceph and the Swift API, but not Swift itself.

An end user will not see any difference from a vanilla Swift deployment, he'll still use the Swift API, still authenticate with Keystone.

Here is a global example of Ceph integration with OpenStack:

Basic design and architecture

In high workload environment you will need high speed network interfaces to avoid any bottleneck and data consistency issue. It's always a good idea to have a logical (vlan) and sometimes physical separation (dedicated networks cards, dedicated switches) of your control plane and data plane. In some case you will also want to separate completely the trafic related to storage operations.

Usually it's a good idea to have at least 2 bonds, one for control plane and one for data plane.

I've already deployed some OpenStack with separation of the control plane, data plane and storage trafic, even some with data replication and data presentation logically and physically separated.

But on some deployment like this one I have only one physical bond or one physical nic for everything and it's enough, no stress.

Every deployment is different from one to another, depending on criticity, security need, workload and budget. Anyway this topic about design and architecture will need a complete separate post.

Prepare the Ceph deployment configuration

Edit your inventory to include the storage nodes. Put all the nodes that have dedicated devices for Ceph. If you want to deploy the Ceph cluster in a HCI way just put your compute nodes, in this example I use the 3 nodes, from compute01 to compute03:


The control plane part will be automatically deployed on your controllers.
As I said the deployment is really simple. Just edit your globals.yml file:

enable_ceph: "yes"
enable_ceph_rgw: "yes"
enable_ceph_rgw_keystone: "yes"

You can take advantage of kolla-ansible deployment if you have dedicated network cards for the storage data plane. Kolla has 2 distinct networks for the storage: cluster_interface that is used for data replication, and storage_interface that is use by the VM to communicate with Ceph. For me:

storage_interface: "ens38"
cluster_interface: "ens39"

If you use only one interface you can just leave the default {{ network_interface }}.

Prepare the disks for OSD bootstraping

On deployment kolla will scan all disks and will bootstrap every disks with this specific label: KOLLA_CEPH_OSD_BOOTSTRAP_BS.
Since Queens release, bluestore is the default storage type (_BS).

In this example, I have 3 disks (sdc, sdd and sde) on each storage hosts (it will be an hyperconverged deployment with the osd on compute node):

parted /dev/sdc -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1
parted /dev/sdd -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1
parted /dev/sde -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS 1 -1

Since I use full flash storage I do not particularly need dedicated device for each bluestore partition but you can separate the bluestore OSD block partition, the bluestore OSD block.wal and the bluestore OSD block.db partition. This is useful if you have spinning hard drive for data and SSD for WAL and DB.
For a WAL device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_W 1 -1

For a DB device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_D 1 -1

For a block device:

parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP_BS_FOO_B 1 -1

If you do this you cannot use the KOLLA_CEPH_OSD_BOOTSTRAP_BS label and you need to use theses 3 separates label instead.


Use the classical commands to deploy:

(openstack) [openstack@centos-kolla ~] $ kolla-ansible -i ./inventory_file deploy

And wait until the end and this beautiful result that we love...

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
compute01                  : ok=137  changed=18   unreachable=0    failed=0    skipped=77   rescued=0    ignored=0
compute02                  : ok=131  changed=18   unreachable=0    failed=0    skipped=64   rescued=0    ignored=0
compute03                  : ok=131  changed=18   unreachable=0    failed=0    skipped=64   rescued=0    ignored=0
control01                  : ok=399  changed=66   unreachable=0    failed=0    skipped=309  rescued=0    ignored=0
control02                  : ok=300  changed=47   unreachable=0    failed=0    skipped=219  rescued=0    ignored=0
control03                  : ok=302  changed=49   unreachable=0    failed=0    skipped=218  rescued=0    ignored=0
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check your deployment

On my lab deployment I said that I've 3 controllers and 3 computes node. 3 OSDs are running on each compute nodes.
At first, check the health status of your Ceph cluster:

[openstack@openstack-controller-1 ~] $ docker exec -it ceph_mon ceph -s
    id:     66d428fe-f859-4769-b66e-c40e8ba3ada2
    health: HEALTH_OK

    mon: 3 daemons, quorum,, (age 33m)
    mgr: control01(active, since 6w), standbys: control03, control02
    mds: cephfs:1 {0=control03=up:active} 2 up:standby
    osd: 9 osds: 9 up (since 3m), 9 in (since 3m)
    rgw: 1 daemon active (radosgw.gateway)

    pools:   12 pools, 144 pgs
    objects: 1.14k objects, 5.1 GiB
    usage:   23 GiB used, 156 GiB / 179 GiB avail
    pgs:     144 active+clean

    client:   2.2 KiB/s rd, 2 op/s rd, 0 op/s wr

I see that my 9 OSDs are up, I have an active rgw daemon and everything is working, the important part is health: HEALTH OK.

On one of my compute node I see my 3 OSDs:

[openstack@openstack-compute-1 ~] $ docker ps | grep ceph
d2e45b33894e        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         up 4 minutes                        ceph_osd_0
df5517709135        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                        ceph_osd_5
dd6525c658c5        kolla/centos-source-ceph-osd:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                        ceph_osd_7

And on one of my controller:

[openstack@openstack-controller-1 ~] $ docker ps | grep ceph
7c2fd10681ee        kolla/centos-source-ceph-rgw:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_rgw
db90351e0d13        kolla/centos-source-ceph-mgr:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_mgr
4ff360078c26        kolla/centos-source-ceph-mon:train                    "dumb-init --single-ā€¦"   4 minutes ago         Up 4 minutes                              ceph_mon

You see now the object storage endpoint (which is the Ceph endpoint):

(openstack) [openstack@centos-kolla ~] $ openstack endpoint list | grep object
| 1be42ae9c8ee43edbb7e4bf73eb9f958 | RegionOne | swift        | object-store   | True    | public    |         |
| 3cfcfe9eed924d7a9acf4d86761be6cb | RegionOne | swift        | object-store   | True    | internal  |         |
| c279c8bda5694eb19d4e6c0924a0e6fd | RegionOne | swift        | object-store   | True    | admin     |         |

A look to the configuration

On one of the controller:

[openstack@openstack-controller-1 ~] $ cat /etc/kolla/ceph-rgw/ceph.conf
log file = /var/log/kolla/ceph/$cluster-$name.log
log to syslog = false
err to syslog = false
log to stderr = false
err to stderr = false
fsid = 66d428fe-f859-4769-b66e-c40e8ba3ada2
mon initial members =,,
mon host =,,
mon addr =,,
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
setuser match path = /var/lib/ceph/$type/$cluster-$id
osd crush update on start = false

mon compact on start = true
mon cluster log file = /var/log/kolla/ceph/$cluster.log

host =
rgw frontends = civetweb port=
rgw_keystone_url =
rgw_keystone_admin_user = ceph_rgw
rgw_keystone_admin_password = SoJqMgzw9L7ulMO2OIqdIeF2mQPQvuLxUwSIRc11
rgw_keystone_admin_project = service
rgw_keystone_admin_domain = default
rgw_keystone_api_version = 3
rgw_keystone_accepted_roles = admin, _member_
rgw_keystone_accepted_admin_roles = ResellerAdmin
rgw_swift_versioning_enabled = true
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/kolla/ceph/client.radosgw.gateway.log

You see all the interesting configuration under [client.radosgw.gateway]. You have the Keystone endpoint admin authentication URL, the frontend endpoint, the different default roles that will be used, etc...

Enable Keystone S3 authentication

You want to use S3 and Keystone authentication? No problem! Thanks to the incredibly easy way to customize configuration with kolla-ansible just do on your deployment node:

(openstack) [openstack@centos-kolla ~] $ mkdir -p /etc/kolla/config
(openstack) [openstack@centos-kolla ~] $ crudini --set /etc/kolla/config/ceph.conf client.radosgw.gateway rgw_s3_auth_use_keystone true

then reconfigure:

(openstack) [openstack@centos-kolla ~] $ kolla-ansible -i multinode reconfigure --tags ceph

If you look now the /etc/kolla/ceph-rgw/ceph.conf you will see this line at the end of the file:

rgw_s3_auth_use_keystone = true

Note that this deployment is very simple, I don't use SSL and wildcard DNS domain, so I cannot use S3-style subdomains.

Test it!

Swift integration

I'll test the creation and some operations with Swift: creation of a container demo, uploading a file (here a 512 MB Debian image named debian-10-openstack-amd64.qcow2), list the objects and display information about the swift container :

(openstack) [openstack@centos-kolla ~] $ swift list demo --lh
Container u'demo' not found
(openstack) [openstack@centos-kolla ~] $ swift post demo
(openstack) [openstack@centos-kolla ~] $ swift upload demo debian-10-openstack-amd64.qcow2 
(openstack) [openstack@centos-kolla ~] $ swift list demo --lh
512M 2020-04-11 00:54:21                     None debian-10-openstack-amd64.qcow2
(openstack) [openstack@centos-kolla ~] $ swift stat -v demo --lh
                   Auth Token: gAAAAABekRZFlKB4cPmDkJsqpTqFILPSgPtOIXEzRzd01iZwMLg5Vq_WDBMPM-fIfu9f78w5S3i5EVYCosf1hJQ-X4YLJtwwpq6S-ZgqwUkK66SYKg02dbQT52rcxA3CuOsz3tuCebHrEq_KtSp2E9oKwD3vtJBvH-Zxpgwp5inVb-ml3QQhAbU
                      Account: v1
                    Container: demo
                      Objects: 2
                        Bytes: 512M
                     Read ACL:
                    Write ACL:
                      Sync To:
                     Sync Key:
              X-Storage-Class: STANDARD
                Accept-Ranges: bytes
             X-Storage-Policy: default-placement
X-Container-Bytes-Used-Actual: 537448448
                Last-Modified: Sat, 11 Apr 2020 00:54:09 GMT
                  X-Timestamp: 1586566161.46193
                   X-Trans-Id: tx00000000000000000000d-005e911645-fa7a-default
                 Content-Type: text/plain; charset=utf-8
       X-Openstack-Request-Id: tx00000000000000000000d-005e911645-fa7a-default

We see that our object is here in the demo container, and the container details show now the total size, Bytes: 512M.

S3 integration

For this example I'll use s3cmd command line to test the integration.
At first generate your credentials to get your access and secret keys:

(openstack) [openstack@centos-kolla ~] $ openstack ec2 credentials create
| Field      | Value                                                                                                                                |
| access     | df2d93542ca243098f10033b62817635                                                                                                     |
| links      | {u'self': u''} |
| project_id | 993ff62625a84916bddd820a97e31cef                                                                                                     |
| secret     | 1f7da7ae6d5b455880832c052d1f49ae                                                                                                     |
| trust_id   | None                                                                                                                                 |
| user_id    | cd21a07450d64f3a9b277670d2e6d3f7                                                                                                     |

Then configure s3cmd with the access and secret keys and use the Ceph endpoint, here I'll use the external API endpoint

This is how my .s3cfg looks like:

(openstack) [openstack@centos-kolla ~] $ cat ~/.s3cfg
access_key = df2d93542ca243098f10033b62817635
secret_key = 1f7da7ae6d5b455880832c052d1f49ae
host_base =
host_bucket =

Now you can test:

(openstack) [openstack@centos-kolla ~] $ s3cmd ls S3://demo --no-ssl
2020-04-11 00:54    537442816  s3://demo/debian-10-openstack-amd64.qcow2


Now if we look at Horizon we see that we have exactly the same thing as if we had deployed Swift :

How it's look on Ceph

List the pools:

[openstack@openstack-controller-1 ~] $ docker exec -it ceph_mon rados lspools

On my deployment, in addition to object storage integration I've already integrated Nova (compute), Glance (image) and Cinder (block storage), that's why I have a vms (Nova), images (Glance), volumes and backup (Cinder and Cinder Backup) pools.

The interesting pools here will be default.rgw.buckets.index and default.rgw.buckets.data.

Remember that we have a demo container with 512 MB debian-10-openstack-amd64.qcow2 file.

If we look at the default.rgw.buckets.index pools we will see our container:

[openstack@openstack-controller-1 ~] $ docker exec -ti ceph_mon rados ls -p default.rgw.buckets.index

The container id is ee9f3b14-4996-45f5-9fee-f8f191ee2d18.64239.1.
If we look into the default.rgw.buckets.data:

[openstack@openstack-controller-1 ~] $ docker exec -ti ceph_mon rados ls -p default.rgw.buckets.data | grep -v shadow

we see our object!

And it's done!

That's it, now you know how to integrate a Ceph cluster to use object storage on OpenStack: authenticate with Keystone and do operation with both Swift and S3!

Don't forget to delete your deployment, --yes-i-really-really-mean-it šŸ˜‰


And my experience šŸ˜‰

Feel free to correct me if you see any typo or if something seems wrong to you.
You can send me an email or comment below.