Category Archives: Redhat Ceph Storage

How to configure ceph as cinder backend ?

In this article I am going to show how to configure ceph as cinder backend. In my last article, I have shown the integration of ceph as glance and nova backend. This integration is almost similar to nova with some changes in configuration file.

Step 1 :  Create one pool with name of volumes with 64 PGs.

[root@ceph1 ~]# ceph osd pool create volumes 64
pool ‘volumes’ created

Step 2 : Created user vole with full access to the pool ‘volumes’ and with read, execution access to the pool ‘vms’ which is used for glance integration.

[root@ceph1 ~]# ceph auth get-or-create client.cinder mon ‘allow r’ osd ‘allow object_prefix class-read rbd_children, allow rwx pool=volumes, allow rx pool=vms’ -o /etc/ceph/ceph.client.cinder.keyring

Step 3 : Copied the keyring and auth-key to the openstack node.

[root@ceph1 ~]# scp -p /etc/ceph/ceph.client.vole.keyring root@192.168.122.245:/etc/ceph/
root@192.168.122.245’s password:
ceph.client.vole.keyring

[root@ceph1 ~]# ceph auth get-key client.vole | ssh root@192.168.122.245 tee /root/test.key

AQAePjRWG5mIJRAAXVRPRSs13ViWWpxlZFluHQ==

Step 4 : Verify on the openstack node that you are able to list the content of the pool volumes.

[root@openstack1 ceph(keystone_admin)]# rbd –user vole -p volumes ls

Step 5 : Define the secret.

[root@openstack1 ceph(keystone_admin)]# virsh secret-define –file /root/cinder1.xml
Secret 14cf1630-801b-4dd6-aef5-2bed37fd086b created

Checking the content of the cinder1.xml file. You need to create this file simply using vi editor and the uuid part you can generate with simple uuidgen command.

[root@openstack1 ceph(keystone_admin)]# virsh secret-dumpxml 14cf1630-801b-4dd6-aef5-2bed37fd086b
<secret ephemeral=’no’ private=’no’>
<uuid>14cf1630-801b-4dd6-aef5-2bed37fd086b</uuid>
<usage type=’ceph’>
<name>client.vole secret</name>
</usage>
</secret>

Set the secret value. Note : test.key contains the key of user vole (recall step 3)

[root@openstack1 ceph(keystone_admin)]# virsh secret-set-value –secret 14cf1630-801b-4dd6-aef5-2bed37fd086b –base64 $(cat /root/test.key)
Secret value set

Step 6 : Take the backup of cinder.conf file before making any change and set the default store to rbd.

[root@openstack1 ceph(keystone_admin)]# cp -p  /etc/cinder/cinder.conf /etc/cinder/cinder.conf.orig
[root@openstack1 ceph(keystone_admin)]# crudini –set /etc/cinder/cinder.conf DEFAULT enabled_backends rbd

After making all changes to configuration file, below is the list of differences between the original and modified configuration file.

[root@openstack1 ceph(keystone_admin)]# diff /etc/cinder/cinder.conf /etc/cinder/cinder.conf.orig
662c662
< enabled_backends=rbd

> enabled_backends=lvm
813c813
< debug=True

> debug=False
2877,2883d2876
<
< [rbd]
< volume_driver=cinder.volume.drivers.rbd.RBDDriver
< rbd_pool=volumes
< rbd_user=vole
< rbd_ceph_conf=/etc/ceph/ceph.conf
< rbd_secret_uuid=14cf1630-801b-4dd6-aef5-2bed37fd086b

After restart of cinder service, make sure that cinder(rbd) service is up and running.

[root@openstack1 ceph(keystone_admin)]# openstack-service restart cinder-volume

[root@openstack1 ceph(keystone_admin)]# cinder service-list
+——————+—————-+——+———+——-+—————————-+—————–+
|      Binary      |      Host      | Zone |  Status | State |         Updated_at         | Disabled Reason |
+——————+—————-+——+———+——-+—————————-+—————–+
|  cinder-backup   |   openstack1   | nova | enabled |  down | 2015-10-31T04:15:46.000000 |       None      |
| cinder-scheduler |   openstack1   | nova | enabled |   up  | 2015-10-31T04:48:16.000000 |       None      |
|  cinder-volume   | openstack1@lvm | nova | enabled |  down | 2015-10-31T04:15:46.000000 |       None      |
|  cinder-volume   | openstack1@rbd | nova | enabled |   up  | 2015-10-31T04:48:08.000000 |       None      |
+——————+—————-+——+———+——-+—————————-+—————–+

Step 7 : Creating the test volume to verify the configuration.

[root@openstack1 ceph(keystone_admin)]# cinder create –display-name=”test1″ 1
+———————+————————————–+
|       Property      |                Value                 |
+———————+————————————–+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2015-10-31T04:34:41.306901      |
| display_description |                 None                 |
|     display_name    |                test1                 |
|      encrypted      |                False                 |
|          id         | 4fbc1eb7-b2ab-4247-9f09-792021fc505c |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+———————+————————————–+

[root@openstack1 ceph(keystone_admin)]# cinder list
+————————————–+———–+————–+——+————-+———-+————-+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+————————————–+———–+————–+——+————-+———-+————-+
| 4fbc1eb7-b2ab-4247-9f09-792021fc505c | available |    test1     |  1   |     None    |  false   |             |
+————————————–+———–+————–+——+————-+———-+————-+

Okay, our volume is successfully created.

Step 8 : Checking the content of the pool volumes which is configured to store the volume.

[root@openstack1 ceph(keystone_admin)]# rbd –user vole -p volumes ls
volume-4fbc1eb7-b2ab-4247-9f09-792021fc505c

[root@openstack1 ceph(keystone_admin)]# rbd –user vole -p volumes info volume-4fbc1eb7-b2ab-4247-9f09-792021fc505c
rbd image ‘volume-4fbc1eb7-b2ab-4247-9f09-792021fc505c’:
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.37cf5e5b00de
format: 2
features: layering

How to create and modify users in ceph (cephx) ?

In this article I am going to show you the method of creating the ceph users. We need to create the ceph user to integrate it with various components or segregate the responsibilities.

Step 1 : If you are having existing user in ceph cluster. You can check the acccess rights and key for that user using below command.

[root@ceph-m1 ceph]# ceph auth get client.glance
exported keyring for client.glance
[client.glance]
key = AQDJkztVMAOxORAAbVtUGdJZr91zvJVpKvKLUA==
caps mon = “allow r”
caps osd = “allow rwx pool=images”

[root@ceph-m1 ceph]# ceph auth get client.vicky
exported keyring for client.vicky
[client.vicky]
key = AQBrZ0xV2H5/EBAA4bcOLGwURXIQuFvp1yYS5Q==

Step 2 : In the previous output we have see that user vicky doesn’t have any access. To provide the access to existing user we can use below commands. I am providing all rights to user vicky.

[root@ceph-m1 ceph]# ceph auth caps client.vicky mon ‘allow *’ osd ‘allow *’
updated caps for client.vicky

After issuing above command we can again check the user vicky rights and keys.

[root@ceph-m1 ceph]# ceph auth get client.vicky
exported keyring for client.vicky
[client.vicky]
key = AQBrZ0xV2H5/EBAA4bcOLGwURXIQuFvp1yYS5Q==
caps mon = “allow *”
caps osd = “allow *”

Step 3 : If you want to remove the given rights from user vicky.

[root@ceph-m1 ceph]# ceph auth caps client.vicky mon ‘ ‘ osd ‘ ‘
updated caps for client.vicky

[root@ceph-m1 ceph]# ceph auth get client.vicky
exported keyring for client.vicky
[client.vicky]
key = AQBrZ0xV2H5/EBAA4bcOLGwURXIQuFvp1yYS5Q==
caps mon = ” ”
caps osd = ” ”

Step 4 : We can delete the user using below command. After deleting it while fetching the information for the user we will get user which is obvious.

[root@ceph-m1 ceph]# ceph auth del client.vicky
updated

[root@ceph-m1 ceph]# ceph auth get client.vicky
Error ENOENT: failed to find client.vicky in keyring

Step 5 : Suppose you want to create a new user key and you are not user whether that user is already key present or not. Below is the safest approach to generate the key.

[root@ceph-m1 ceph]# ceph auth get-or-create-key client.glance
AQDJkztVMAOxORAAbVtUGdJZr91zvJVpKvKLUA==

[root@ceph-m1 ceph]# ceph auth get-or-create-key client.vicky
AQBrZ0xV2H5/EBAA4bcOLGwURXIQuFvp1yYS5Q==

If we want to fetch the key of existing user.

[root@ceph-m1 ceph]# ceph auth print-key client.glance
AQDJkztVMAOxORAAbVtUGdJZr91zvJVpKvKLUA==

Step 6 : If we want to create a user by specifying the keyring file we may use below command.

[ceph@ceph-admin ceph-config]$ sudo ceph-authtool -C /etc/ceph/ceph.keyring -n client.ringo –cap osd ‘allow rwx’ –cap mon ‘allow rw’ –gen-key
creating /etc/ceph/ceph.keyring

[ceph@ceph-admin ceph-config]$ sudo cat /etc/ceph/ceph.keyring
[client.ringo]
key = AQBnbExVsBx5MhAAFXB77/NY8iitSoMjJQaGjQ==
caps mon = “allow rw”
caps osd = “allow rwx”

We can use the below command to modify the permissions of user without generating the new key.

[ceph@ceph-admin ceph-config]$ sudo ceph-authtool /etc/ceph/ceph.keyring -n client.ringo –cap osd ‘allow rwx’ –cap mon ‘allow rw’

[ceph@ceph-admin ceph-config]$ sudo cat /etc/ceph/ceph.keyring
[client.ringo]
key = AQBnbExVsBx5MhAAFXB77/NY8iitSoMjJQaGjQ==
caps mon = “allow rw”
caps osd = “allow rwx”

[References]

http://ceph.com/docs/master/rados/configuration/auth-config-ref/

https://ceph.com/docs/v0.79/rados/operations/auth-intro/

https://ceph.com/docs/v0.79/rados/operations/authentication/

 

Use sed and awk one liner on ceph ?

In this article I am going to show you the usage of basic awk and sed command liners on ceph for getting the desired outputs. I have created them as part of practice in my test lab.

–> In ceph cluster we will be having lot of objects if we want to see the mapping of each and every object that will be difficult manual task. I have used below one liner to list the content of every pool and every object inside the pool.

[root@ceph-m2 ~]# for i in `rados lspools`; do for j in `rbd -p $i ls`; do ceph osd map $i $j; done; done
osdmap e81 pool ‘rbd’ (2) object ‘foo_cp’ -> pg 2.85a7b54d (2.d) -> up ([2,5,4], p2) acting ([2,5,4], p2)
osdmap e81 pool ‘rbd’ (2) object ‘image1’ -> pg 2.571c6242 (2.2) -> up ([4,5,2], p4) acting ([4,5,2], p4)
osdmap e81 pool ‘pool1’ (4) object ‘foo’ -> pg 4.7fc1f406 (4.6) -> up ([4,5,2], p4) acting ([4,5,2], p4)
osdmap e81 pool ‘pool1’ (4) object ‘foo1’ -> pg 4.be9754b3 (4.13) -> up ([5,2,4], p5) acting ([5,2,4], p5)
osdmap e81 pool ‘pool1’ (4) object ‘foo_cp’ -> pg 4.85a7b54d (4.d) -> up ([2,4,5], p2) acting ([2,4,5], p2)
osdmap e81 pool ‘pool2’ (5) object ‘foo’ -> pg 5.7fc1f406 (5.6) -> up ([5,2,4], p5) acting ([5,2,4], p5)

Above output is truncated for the save of brevity.

–> We want to calculate the number of objects present on each OSD in cluster. I have used the below oneliner to determine the same. I have captured the output got in previous oneliner in file text2.txt 

[root@ceph-m2 ~]# awk -F “[][]” ‘{print $2}’ text2.txt | sed ‘s/,/\n/g’ | awk  ‘{a[$1]++} END { for (i in a) print “Number of objects on OSD.” i” is ” a[i] }’
Number of objects on OSD.4 is 10
Number of objects on OSD.5 is 10
Number of objects on OSD.2 is 10

–> I have established password less root connection to the all the servers which are contributing OSDs in cluster from one mon node. We can use the below command to see the underlying disks of every OSD. In below output ceph-4 is presenting OSD.4

[root@ceph-m2 ~]# for i in `ceph osd tree | grep -w host | awk ‘{print $4}’`
> do
> ssh $i df -h | grep -i osd
> done
/dev/sdc1               19G  124M   19G   1% /var/lib/ceph/osd/ceph-4
/dev/sdb1              2.0G   35M  2.0G   2% /var/lib/ceph/osd/ceph-0
/dev/sdb1              2.0G   34M  2.0G   2% /var/lib/ceph/osd/ceph-1
/dev/sdc1               19G  124M   19G   1% /var/lib/ceph/osd/ceph-5
/dev/sdc1               19G  188M   19G   1% /var/lib/ceph/osd/ceph-2
/dev/sdb1              2.0G   34M  2.0G   2% /var/lib/ceph/osd/ceph-3

–> If you want to see the number of PGs (placement groups) present in each pool.

[root@ceph-m2 ~]# ceph pg dump | grep ‘active+clean’ | awk ‘{print $1}’ | awk -F. ‘{count[$1]++} END { for (i in count) print “Pool Number ” i, “has “, count[i], ” PGs” done }’
dumped all in format plain
Pool Number 4 has  32  PGs
Pool Number 5 has  16  PGs
Pool Number 6 has  8  PGs
Pool Number 0 has  64  PGs
Pool Number 1 has  64  PGs
Pool Number 2 has  64  PGs

In above output we are getting the only the pool number not the pool name. We can get the pool name from below command.

[root@ceph-m2 ~]# ceph osd lspools | sed ‘s/,/\n/g’
0 data
1 metadata
2 rbd
4 pool1
5 pool2
6 images

I am still working towards more improvement on above hacks and working on some new as well 🙂

What is use of Crush Maps in Ceph ?

In this article I am going to explain about the crush map in ceph. CRUSH is an algorithm which stands for CRUSH – Controlled, Scalable, Decentralized Placement of Replicated Data. It helps to determine how to store and retrieve data by computing data storage locations. It eliminate the need of centralized server when clients want to communicate with ceph cluster.  It empowers clients to communicate with OSDs directly.

When we deploy a cluster, ceph automatically generates a default CRUSH map of your configuration. In large deployment generally we need to modify the default configuration to get the optimal results.

Below are the steps which you are used to modify the existing CRUSH map.

–> Get the crush map.

–> De-compile the crush map.

–> Edit any section of crush map.

–> Re-compile the crush map.

–> Set the crush map.

Step 1 : In the first step we are getting crush map in output file /tmp/crush-compiled.

[root@ceph-m3 ~]# ceph osd getcrushmap -o /tmp/crush-compiled
got crush map from osdmap epoch 84

Step 2 : Decompile the crush map collected in previous step.

[root@ceph-m3 ~]# crushtool -d /tmp/crush-compiled -o /crush-decomplied

Step 3 : After decompiling ASCII file has been generated.

[root@ceph-m3 ~]# file /crush-decomplied
/crush-decomplied: ASCII text

Once we have decompiled the file lets have a look into that file.

[root@ceph-m3 ~]# cat /crush-decomplied | grep ^#
# begin crush map
# devices
# types
# buckets
# rules
# end crush map

Its having mainly four sections.

devices : Containing the list of OSD (disks) present in cluster.

types : It defines the type of Buckets used in CRUSH hierarchy. Buckets consist of a hierarchical aggregation of storage locations (e.g., rows, racks, chassis, hosts, etc.) and their assigned weights.

buckets: Once you define bucket types, you must declare bucket instances for your hosts, and any other failure domain partitioning you choose.

Rules: Rules consist of the manner of selecting buckets.

Lets make some change in CRUSH decompiled file. I am going to change the CRUSH weight of an OSD.5

[root@ceph-m3 ~]# less /crush-decomplied | grep -v “#” | awk ‘/weight/{print;}’
item osd.0 weight 0.000
item osd.4 weight 0.020
item osd.1 weight 0.000
item osd.5 weight 0.020
item osd.3 weight 0.000
item osd.2 weight 0.030
item ceph-osd1 weight 0.020
item ceph-osd2 weight 0.020
item ceph-m3 weight 0.030

Below is the current output of “ceph osd tree”.

[root@ceph-m3 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      0.06996 root default
-2      0.01999         host ceph-osd1
0       0                       osd.0   up      1
4       0.01999                 osd.4   up      1
-3      0.01999         host ceph-osd2
1       0                       osd.1   up      1
5       0.01999                 osd.5   up      1
-4      0.02998         host ceph-m3
3       0                       osd.3   up      1
2       0.02998                 osd.2   up      1

I have changed the value of osd.5 weight to 3 in our decompiled file (/crush-decomplied) using vi editor.

Step 4 : I am going to recompile the modified CRUSH map.

[root@ceph-m3 ~]# crushtool -c /crush-decomplied -o /tmp/crush-compiled1

[root@ceph-m3 ~]# file /tmp/crush-compiled1
/tmp/crush-compiled1: MS Windows icon resource – 8 icons, 1-colors

Step 5 : Finally, time has come to set the recompiled CRUSH map.

[root@ceph-m3 ~]# ceph osd setcrushmap -i /tmp/crush-compiled1
set crush map

We can see the changed value of Weight in output of “ceph osd tree” for osd.5

[root@ceph-m3 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      0.06998 root default
-2      0.01999         host ceph-osd1
0       0                       osd.0   up      1
4       0.01999                 osd.4   up      1
-3      0.01999         host ceph-osd2
1       0                       osd.1   up      1
5       0.03                    osd.5   up      1
-4      0.03            host ceph-m3
3       0                       osd.3   up      1
2       0.03                    osd.2   up      1

We have given more priority to OSD.5 for saving the data.

How to find the image-format type of image in ceph ?

In previous article (How to perform various operation using rbd in ceph ?) we have seen that we were getting error while creating clone from the snapshot of image created with format 1.

I thought of digging more into it. How can we verify whats the type of format used for the image ?

Type 1 : Image created without specifying any format, by default its created with format 1. You can see that in below output.

[root@ceph-m3 ~]# rbd -p pool2 info foo
rbd image ‘foo’:
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.3685.2ae8944a
format: 1                                                                    <<<<<<

With format 1 we can create the snapshot of image but from the snapshot we can’t create the clone. Because, it doesn’t support the layering feature.

Type 2 : Image created with specifying the format type. I have created image by specifying the format 2.

[root@ceph-m3 ~]#  rbd create –image-format 2 -p pool2 foonew –size 500
[root@ceph-m3 ~]# rbd -p pool2 info foonew
rbd image ‘foonew’:
size 500 MB in 125 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.36ba74b0dc51
format: 2                                     <<<<<<
features: layering                       <<<<<<

It supports the layering feature. We can create the clone from snapshot of image.

If you want to convert existing image to different format, I didn’t find any elegant method for it, only thing I found is to export and import the image the different format.

My image is foo of type format 1. I am importing this image to a file.

[root@ceph-m3 ~]# rbd export pool2/foo@snap10 /tmp/image1
Exporting image: 100% complete…done.

Importing the same image in format 2. In below command we have not specified the pool name hence it will go default pool rbd.

[root@ceph-m3 ~]# rbd import –image-format 2 /tmp/image1
Importing image: 100% complete…done.

But the image is converted into format2. Notice the name of image is also not mentioned hence taking the filename as the image name.

[root@ceph-m3 ~]# rbd -p rbd info image1
rbd image ‘image1’:
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.379e74b0dc51
format: 2
features: layering

In below example I am specifying the pool and image name. As we are already having image with name foo in pool2 hence below command is getting fail. We need to import the image with different name.

[root@ceph-m3 ~]# rbd import -p pool2 –image-format 2 /tmp/image1 foo
rbd: image creation failed
Importing image: 0% complete…failed.
rbd: import failed: (17) File exists
2015-04-23 10:52:35.088374 7f0178a957c0 -1 librbd: rbd image foo already exists

Eventually I imported the image using name foo1 instead of foo in pool2.

[root@ceph-m3 ~]# rbd import -p pool2 –image-format 2 /tmp/image1 foo1
Importing image: 100% complete…done.

Below is the information which I have found from official ceph website.

--image-format format
Specifies which object layout to use. The default is 1.

  • format 1 – Use the original format for a new rbd image. This format is understood by all versions of librbd and the kernel rbd module, but does not support newer features like cloning.
  • format 2 – Use the second rbd format, which is supported by librbd and kernel since version 3.11 (except for striping). This adds support for cloning and is more easily extensible to allow more features in the future.

How to perform various operation using rbd in ceph ?

In this article I am going to show you the various snapshot options which can be performed on rados block device image. Not all the options are covered in this article.

Step 1 : I have created one image of size 1GB inside the pool foo.

[root@mon3 ~]# rbd create –pool pool2 –size 1024 foo

[root@mon3 ~]# rbd snap ls pool2/foo
SNAPID NAME     SIZE
2 snap1 1024 MB

I was able to create the snapshot successfully using below command.

[root@mon3 ~]# rbd snap create pool2/foo@snap1

But while creating the clone from snapshot was getting error.

[root@mon3 ~]# rbd -p pool2 clone pool2/foo@snap1 foo_cl
rbd: clone error: (22) Invalid argument
2015-04-23 11:22:43.641888 7fd11a4857c0 -1 librbd: parent image must be in new form

Step 2 : When we are creating images inside the pool they can be format 1 or of format 2 type. For the snapshot to work we need image of foramt type 2.

I created new image by specifying the format 2 option.

[root@mon3 ~]# rbd create –image-format 2 -p pool2 foonew –size 500
[root@mon3 ~]# rbd -p pool2 ls
foo
foonew

I have create snapshot with name snap2.

[root@mon3 ~]# rbd snap create pool2/foonew@snap2

We can check the snapshot for the image using below command.

[root@mon3 ~]# rbd snap ls pool2/foonew
SNAPID NAME    SIZE
3 snap2 500 MB

Step 3 : We have created snapshot now lets try to create clone from that snapshot. While doing that again I am getting the error that snapshot is not protected. But we need not to worry about this error like in step 1.

[root@mon3 ~]# rbd -p pool2 clone pool2/foonew@snap2 foo_cl
rbd: clone error: (22) Invalid argument
2015-04-23 11:25:51.851418 7f0d414577c0 -1 librbd: parent snapshot must be protected
I protect the snapshot from any kind of accidental deletion.

[root@mon3 ~]# rbd snap protect pool2/foonew@snap2

After protecting the snapshot, I am able to perform the clone from snapshot successfully.

[root@mon3 ~]# rbd -p pool2 clone pool2/foonew@snap2 foo_cl

[root@mon3 ~]# rbd -p pool2 ls
foo
foonew

Step 4 : I was trying to locate the clone created in previous step. I found that it got created in default rbd pool because I have not specified the pool name while creating the clone.

[root@mon3 ~]# rbd -p rbd ls
foo_cl

Created clone by specifying the pool name pool3.

[root@mon3 ~]# rbd -p pool2 clone pool2/foonew@snap2 pool3/foo_cl
[root@mon3 ~]# rbd -p pool3 ls
foo_cl

Step 5 : After the creation of clones from the snapshot. We can check how many children snapshot is having. We have create two clones from snapshot, we are getting same count with pool name and image name.

[root@mon3 ~]# rbd children pool2/foonew@snap2
pool3/foo_cl
rbd/foo_cl

Now if we want to delete the snapshot, it will not allow us. Remember, we have protect the snapshot.

[root@mon3 ~]# rbd snap rm pool2/foonew@snap2
rbd: snapshot ‘snap2’ is protected from removal.
2015-04-23 11:29:48.619637 7fa036b517c0 -1 librbd: removing snapshot from header failed: (16) Device or resource busy

Step 6 : To remove the dependency between the snapshot and image, we need to flatten the image using below command.

[root@mon3 ~]# rbd -p rbd flatten –image foo_cl
Image flatten: 100% complete…done.

Once again check the children for the snapshot. You will see only one in output this time because we have performed the flatten  operation for other.

[root@mon3 ~]# rbd children pool2/foonew@snap2
pool3/foo_cl
[root@mon3 ~]#

In similar way we can perform the flatten operation for other clone as well then we can unprotect the snapshot for removal.

How to access Redhat ceph storage as block device storage ?

We can mount Ceph as a thinly provisioned block device, when we write data to Ceph using a block device, Ceph automatically stripes and replicates the data across the cluster.

Step 1 : I have created one image named foo of size 1GB using pool1.

[root@ceph-m3 ~]# rbd create –pool pool1  –size 1024 foo
[root@ceph-m3 ~]# rbd –pool pool1 ls
foo

Step 2 : We can verify the information of created image using below command.

[root@ceph-m3 ~]# rbd –pool pool1 info foo
rbd image ‘foo’:
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.2851.2ae8944a
format: 1

Step 3 : I have initially created image of 1GB I have extended the size of image to 2GB.

[root@ceph-m3 ~]# rbd –pool pool1 –image foo –size 2048 resize
Resizing image: 100% complete…done.

[root@ceph-m3 ~]# rbd –pool pool1 info foo
rbd image ‘foo’:
size 2048 MB in 512 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.2851.2ae8944a
format: 1

Step 4 : We can check the newly created images (objects) using below command.

[root@ceph-m3 ~]# rados -p pool1 ls
object3
object2
object4
object5
foo.rbd                       <<<<
rbd_directory             <<<<
object1

We can also use the command “rbd -p pool1 ls” to list the images (objects).

Lets check the content of rbd_directory.

[root@ceph-m3 ~]# rados -p pool1 get rbd_directory /tmp/rbd.out

[root@ceph-m3 ~]# cat /tmp/rbd.out
foo

We can map the placement of rbd_directory and foo using below commands.

[root@ceph-m3 current]# ceph osd map pool1 rbd_directory
osdmap e49 pool ‘pool1’ (4) object ‘rbd_directory’ -> pg 4.30a98c1c (4.1c) -> up ([2,5,4], p2) acting ([2,5,4], p2)

[root@ceph-m3 4.1c_head]# ceph osd map pool1 foo.rbd
osdmap e49 pool ‘pool1’ (4) object ‘foo.rbd’ -> pg 4.61b88b98 (4.18) -> up ([4,2,5], p4) acting ([4,2,5], p4)

Object are placed in Placement groups 4.1c and 4.18.

As we know when we are creating OSD its getting mounted on server.

[root@ceph-m3 current]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        19G   58M   19G   1% /var/lib/ceph/osd/ceph-2

[root@ceph-m3 current]# pwd
/var/lib/ceph/osd/ceph-2/current/

We can grep for the specific PGs in below output.

[root@ceph-m3 current]# ls -la | grep -i 4.1c
drwxr-xr-x   2 root root    77 Apr 22 08:37 4.1c_head

[root@ceph-m3 current]# cd 4.1c_head/

[root@ceph-m3 4.1c_head]# ll
total 10244
-rw-r–r– 1 root root 10485760 Apr 22 07:34 object1__head_BAC5DEBC__4
-rw-r–r– 1 root root       19 Apr 22 08:37 rbd\udirectory__head_30A98C1C__4

Same goes for the PG (4.18) corresponding to foo.rbd

[root@ceph-m3 4.18_head]# ls -lsh
total 8.0K
8.0K -rw-r–r– 1 root root 112 Apr 22 08:38 foo.rbd__head_61B88B98__4

Step 5 : We have created the image time has come to map the image to block device.

[root@ceph-m3 current]# rbd –pool pool1 map foo

[root@ceph-m3 current]# rbd -p pool1 showmapped
id pool  image snap device
0  pool1 foo   –    /dev/rbd0

Step 6 : Created filesystem on block device and after that mounted it on /mnt/

[root@ceph-m3 4.18_head]# mkfs.ext4 /dev/rbd0

[root@ceph-m3 4.18_head]# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       2.0G  6.0M  1.8G   1% /mnt

In the PG 4.18 directory we can see the new file has introduced after filesystem creation. You can compare the output of step 4.

[root@ceph-m3 4.18_head]# ll
total 8136
-rw-r–r– 1 root root     112 Apr 22 08:38 foo.rbd__head_61B88B98__4
-rw-r–r– 1 root root 4194304 Apr 22 09:06 rb.0.2851.2ae8944a.000000000107__head_6F421E38__4