How to access Redhat ceph storage as block device storage ?

We can mount Ceph as a thinly provisioned block device, when we write data to Ceph using a block device, Ceph automatically stripes and replicates the data across the cluster.

Step 1 : I have created one image named foo of size 1GB using pool1.

[root@ceph-m3 ~]# rbd create –pool pool1  –size 1024 foo
[root@ceph-m3 ~]# rbd –pool pool1 ls
foo

Step 2 : We can verify the information of created image using below command.

[root@ceph-m3 ~]# rbd –pool pool1 info foo
rbd image ‘foo’:
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.2851.2ae8944a
format: 1

Step 3 : I have initially created image of 1GB I have extended the size of image to 2GB.

[root@ceph-m3 ~]# rbd –pool pool1 –image foo –size 2048 resize
Resizing image: 100% complete…done.

[root@ceph-m3 ~]# rbd –pool pool1 info foo
rbd image ‘foo’:
size 2048 MB in 512 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.2851.2ae8944a
format: 1

Step 4 : We can check the newly created images (objects) using below command.

[root@ceph-m3 ~]# rados -p pool1 ls
object3
object2
object4
object5
foo.rbd                       <<<<
rbd_directory             <<<<
object1

We can also use the command “rbd -p pool1 ls” to list the images (objects).

Lets check the content of rbd_directory.

[root@ceph-m3 ~]# rados -p pool1 get rbd_directory /tmp/rbd.out

[root@ceph-m3 ~]# cat /tmp/rbd.out
foo

We can map the placement of rbd_directory and foo using below commands.

[root@ceph-m3 current]# ceph osd map pool1 rbd_directory
osdmap e49 pool ‘pool1’ (4) object ‘rbd_directory’ -> pg 4.30a98c1c (4.1c) -> up ([2,5,4], p2) acting ([2,5,4], p2)

[root@ceph-m3 4.1c_head]# ceph osd map pool1 foo.rbd
osdmap e49 pool ‘pool1’ (4) object ‘foo.rbd’ -> pg 4.61b88b98 (4.18) -> up ([4,2,5], p4) acting ([4,2,5], p4)

Object are placed in Placement groups 4.1c and 4.18.

As we know when we are creating OSD its getting mounted on server.

[root@ceph-m3 current]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        19G   58M   19G   1% /var/lib/ceph/osd/ceph-2

[root@ceph-m3 current]# pwd
/var/lib/ceph/osd/ceph-2/current/

We can grep for the specific PGs in below output.

[root@ceph-m3 current]# ls -la | grep -i 4.1c
drwxr-xr-x   2 root root    77 Apr 22 08:37 4.1c_head

[root@ceph-m3 current]# cd 4.1c_head/

[root@ceph-m3 4.1c_head]# ll
total 10244
-rw-r–r– 1 root root 10485760 Apr 22 07:34 object1__head_BAC5DEBC__4
-rw-r–r– 1 root root       19 Apr 22 08:37 rbd\udirectory__head_30A98C1C__4

Same goes for the PG (4.18) corresponding to foo.rbd

[root@ceph-m3 4.18_head]# ls -lsh
total 8.0K
8.0K -rw-r–r– 1 root root 112 Apr 22 08:38 foo.rbd__head_61B88B98__4

Step 5 : We have created the image time has come to map the image to block device.

[root@ceph-m3 current]# rbd –pool pool1 map foo

[root@ceph-m3 current]# rbd -p pool1 showmapped
id pool  image snap device
0  pool1 foo   –    /dev/rbd0

Step 6 : Created filesystem on block device and after that mounted it on /mnt/

[root@ceph-m3 4.18_head]# mkfs.ext4 /dev/rbd0

[root@ceph-m3 4.18_head]# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       2.0G  6.0M  1.8G   1% /mnt

In the PG 4.18 directory we can see the new file has introduced after filesystem creation. You can compare the output of step 4.

[root@ceph-m3 4.18_head]# ll
total 8136
-rw-r–r– 1 root root     112 Apr 22 08:38 foo.rbd__head_61B88B98__4
-rw-r–r– 1 root root 4194304 Apr 22 09:06 rb.0.2851.2ae8944a.000000000107__head_6F421E38__4

Advertisements

2 thoughts on “How to access Redhat ceph storage as block device storage ?

  1. sspadmin

    nice info. I have a query. How to copy data from rbd block device to rados pool? Say If one has to copy data from mounted RBD dviece on linux to rados data pool, how can this be done?

    Reply
    1. Vikrant Post author

      Thanks for your comment. I have not tried it. I guess it may be possible if you are using both formats as backend for openstack then you will be able to copy the things using openstack GUI or commands but I have not tried it.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s