How to enable Ceph multitenancy for object storage in OpenStack?

How to manage object storage multitenancy with an integrated Ceph cluster into an OpenStack deployment?

Note

If the manipulation is performed after a deployment, changes will only work for new users and new tenants.

Bonus

Integrate Ceph RGW multitenancy and S3 authentication with kolla-ansible

Standard behaviour

Regarding bucket operation, Ceph as some constraints (see here).

The part we're interested in is this: Bucket names must be unique.

Why? Because with OpenStack you have the concept of multitenancy. In a vanilla Swift deployment each tenants can overlap the bucket name between them.
The "classic" object-store service API endpoint are created like this:

openstack endpoint create --region RegionOne \
  object-store public http://endpoint:8080/v1/AUTH_%\(project_id\)s

This specific part AUTH_%\(project_id\)s is very important: each tenant have it's own namespace and can do anything without any impact on other tenant. If you create a container ($CONTAINER), the URL will be:

http://endpoint:8080/v1/AUTH_%\(project_id\)s/$CONTAINER

On the other side the default endpoint with an integrated Ceph is:

http://radosgw-endpoint-url:6780/v1/swift

It means that when you create a container $CONTAINER, the URL will be /v1/swift/$CONTAINER and that's a big problem if you want to have a multitenancy.

You'll have this regardless if you use Horizon/Swift API or S3 endpoint.

How it looks in Ceph

The key is to understand how Ceph works in this situation.

For example if I create (from Swift CLI) 2 containers named demo in 2 differents tenants from Swift endpoint, this is how it'll work:

On the first tenant I can create the container demo:

(openstack) [openstack@centos ~] $ swift post demo --debug
...
INFO:swiftclient:RESP STATUS: 404 Not Found
INFO:swiftclient:RESP BODY: NoSuchKey
DEBUG:urllib3.connectionpool:http://172.16.10.10:6780 "PUT /swift/v1/demo HTTP/1.1" 201 0
DEBUG:swiftclient:RESP STATUS: 201 Created
...

But on the second tenant:

(openstack) [openstack@centos ~] $ swift post demo
DEBUG:urllib3.connectionpool:http://172.16.10.10:6780 "POST /swift/v1/demo HTTP/1.1" 403 12
INFO:swiftclient:RESP STATUS: 403 Forbidden
INFO:swiftclient:RESP BODY: AccessDenied
ClientException: Container POST failed: http://172.16.10.10:6780/swift/v1/demo 403 Forbidden   AccessDenied
Container POST failed: http://172.16.10.10:6780/swift/v1/demo 403 Forbidden   AccessDenied
Failed Transaction ID: tx000000000000000000070-005e962c65-86cc-default

I'll not be allowed to create this container because the name already exists.

If we look on Ceph side we see clearly why we cannot overlap 2 containers:

[openstack@controller-1 ~] $ radosgw-admin bucket list
[
    "demo"
]

This behaviour might be fine for a variety of use cases, but if you want to take advantage of multitenancy, it's really problematic.

Enable multitenancy capabilities

We need to specify to Ceph that he needs to use specific bucket name and have an identifier of the tenant name on each bucket.

You have to add 2 options in your ceph.conf:

[client.radosgw.gateway]
...
rgw_keystone_implicit_tenants = true
rgw_swift_account_in_url = true

The first one will tell Ceph to put the bucket under the tenant id for each new user. The second one will allow you to use the endpoint with the tenant id identifier.

You also have to change your swift endpoint with adding %\(tenant_id\)s add the end of each endpoint. Get the IDs:

(openstack) [openstack@centos ~] $ openstack endpoint list | grep swift
| 1be42ae9c8ee43edbb7e4bf73eb9f958 | RegionOne | swift        | object-store   | True    | public    | http://172.16.10.10:6780/swift/v1         |
| 3cfcfe9eed924d7a9acf4d86761be6cb | RegionOne | swift        | object-store   | True    | internal  | http://172.16.11.10:6780/swift/v1         |
| c279c8bda5694eb19d4e6c0924a0e6fd | RegionOne | swift        | object-store   | True    | admin     | http://172.16.11.10:6780/swift/v1         |

Then edit them (remplace the id at the end by your endpoints id):

openstack endpoint set --region RegionOne --service swift --interface public --url http://172.16.10.10:6780/swift/v1/%\(tenant_id\)s 1be42ae9c8ee43edbb7e4bf73eb9f958
openstack endpoint set --region RegionOne --service swift --interface internal --url http://172.16.10.10:6780/swift/v1/%\(tenant_id\)s 3cfcfe9eed924d7a9acf4d86761be6cb
openstack endpoint set --region RegionOne --service swift --interface admin --url http://172.16.10.10:6780/swift/v1/%\(tenant_id\)s c279c8bda5694eb19d4e6c0924a0e6fd

It'll work both with and without AUTH_ prefix. More informations here.

How it looks now in Ceph

I've created 2 tenants:

(openstack) [openstack@centos ~] $ openstack project list | grep 'demo2\|demo3'
| 2ae80d2a85ce446bae46595611033a94 | demo2   |
| cdc03141fb1f4090a7de4cbaee55a667 | demo3   |

Then created a demo container with each of these tenant.

Now if we look at Ceph:

[openstack@controller-1 ~] $ radosgw-admin bucket list
[
    "cdc03141fb1f4090a7de4cbaee55a667/demo",
    "2ae80d2a85ce446bae46595611033a94/demo"
]

I see that each container is in the tenant namespace and I haven't any impact by overlapping the container names accross the tenants.

In a simpler way, here's what used to happen before:

Creation of a container named "demo"
Tenant A id: 2ae80d2a85ce446bae46595611033a94
Tenant B id: cdc03141fb1f4090a7de4cbaee55a667

    +-------------------+   OK
    | user A > tenant A +--------->  /demo 
    +-------------------+

    +-------------------+   KO
    | user B > tenant B +--------->  /demo already exist
    +-------------------+

and now:

Creation of a container named "demo"
Tenant A id: 2ae80d2a85ce446bae46595611033a94
Tenant B id: cdc03141fb1f4090a7de4cbaee55a667

    +-------------------+   OK
    | user A > tenant A +--------->  2ae80d2a85ce446bae46595611033a94/demo
    +-------------------+

    +-------------------+   OK
    | user B > tenant B +--------->  cdc03141fb1f4090a7de4cbaee55a667/demo
    +-------------------+

Swift public access

The endpoint now will be:

http://ceph_endpoint:6780/swift/v1/$tenant_id/$container

Example with the container demo of demo3 tenant:

(openstack) [openstack@centos ~] $ swift stat demo -v
                          URL: http://172.16.10.10:6780/swift/v1/cdc03141fb1f4090a7de4cbaee55a667/demo
                   Auth Token: gAAAAABeljlU0SW_cEqFomGOq0L33N45uDFAHsKBWzPg4w2WxguR4aD1WlexGD6mB3FQK4HDnpms8y-z3_O-KP5rc-DMyzkj6XoHqZagRyTiUFc9twyyg17CUyk5uTl8PENrtLUZN_K3vZRRqiubT__imIGMEqmrl0fivO6Q_CUwxi8obmmeTlQ
                      Account: cdc03141fb1f4090a7de4cbaee55a667
                    Container: demo
                      Objects: 0
                        Bytes: 0
                     Read ACL:
                    Write ACL:
                      Sync To:
                     Sync Key:
              X-Storage-Class: STANDARD
                Accept-Ranges: bytes
             X-Storage-Policy: default-placement
X-Container-Bytes-Used-Actual: 0
                Last-Modified: Tue, 14 Apr 2020 22:29:29 GMT
                  X-Timestamp: 1586903369.52510
                   X-Trans-Id: tx00000000000000000005f-005e963954-121c1-default
                 Content-Type: text/plain; charset=utf-8
       X-Openstack-Request-Id: tx00000000000000000005f-005e963954-121c1-default

S3 public access

The S3 endpoint will be now look like that:

http://ceph_endpoint:6780/$tenant_id:$container

Example with the container demo of demo3 tenant:

(openstack) [openstack@centos ~] $ curl -s http://172.16.10.10:6780/cdc03141fb1f4090a7de4cbaee55a667:demo | tidy -xml -i -
No warnings or errors were found.

<?xml version="1.0" encoding="utf-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Tenant>cdc03141fb1f4090a7de4cbaee55a667</Tenant>
  <Name>demo</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
</ListBucketResult>

That's all good 😀

Resources

https://docs.ceph.com/docs/master/radosgw/
https://docs.ceph.com/docs/luminous/radosgw/multitenancy/
https://docs.ceph.com/docs/master/radosgw/keystone/
https://docs.ceph.com/docs/master/radosgw/swift/tempurl/
https://docs.ceph.com/docs/master/radosgw/config-ref/
https://opendev.org/openstack/openstack-ansible/commit/9dbdf71de04425473143bdf36412c5278830e993
https://github.com/ceph/ceph/commit/e9259486decab52a362443d3fd3dec33b0ec654f

Picture : Fredy Jacob