What is use of Crush Maps in Ceph ?

In this article I am going to explain about the crush map in ceph. CRUSH is an algorithm which stands for CRUSH – Controlled, Scalable, Decentralized Placement of Replicated Data. It helps to determine how to store and retrieve data by computing data storage locations. It eliminate the need of centralized server when clients want to communicate with ceph cluster.  It empowers clients to communicate with OSDs directly.

When we deploy a cluster, ceph automatically generates a default CRUSH map of your configuration. In large deployment generally we need to modify the default configuration to get the optimal results.

Below are the steps which you are used to modify the existing CRUSH map.

–> Get the crush map.

–> De-compile the crush map.

–> Edit any section of crush map.

–> Re-compile the crush map.

–> Set the crush map.

Step 1 : In the first step we are getting crush map in output file /tmp/crush-compiled.

[root@ceph-m3 ~]# ceph osd getcrushmap -o /tmp/crush-compiled
got crush map from osdmap epoch 84

Step 2 : Decompile the crush map collected in previous step.

[root@ceph-m3 ~]# crushtool -d /tmp/crush-compiled -o /crush-decomplied

Step 3 : After decompiling ASCII file has been generated.

[root@ceph-m3 ~]# file /crush-decomplied
/crush-decomplied: ASCII text

Once we have decompiled the file lets have a look into that file.

[root@ceph-m3 ~]# cat /crush-decomplied | grep ^#
# begin crush map
# devices
# types
# buckets
# rules
# end crush map

Its having mainly four sections.

devices : Containing the list of OSD (disks) present in cluster.

types : It defines the type of Buckets used in CRUSH hierarchy. Buckets consist of a hierarchical aggregation of storage locations (e.g., rows, racks, chassis, hosts, etc.) and their assigned weights.

buckets: Once you define bucket types, you must declare bucket instances for your hosts, and any other failure domain partitioning you choose.

Rules: Rules consist of the manner of selecting buckets.

Lets make some change in CRUSH decompiled file. I am going to change the CRUSH weight of an OSD.5

[root@ceph-m3 ~]# less /crush-decomplied | grep -v “#” | awk ‘/weight/{print;}’
item osd.0 weight 0.000
item osd.4 weight 0.020
item osd.1 weight 0.000
item osd.5 weight 0.020
item osd.3 weight 0.000
item osd.2 weight 0.030
item ceph-osd1 weight 0.020
item ceph-osd2 weight 0.020
item ceph-m3 weight 0.030

Below is the current output of “ceph osd tree”.

[root@ceph-m3 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      0.06996 root default
-2      0.01999         host ceph-osd1
0       0                       osd.0   up      1
4       0.01999                 osd.4   up      1
-3      0.01999         host ceph-osd2
1       0                       osd.1   up      1
5       0.01999                 osd.5   up      1
-4      0.02998         host ceph-m3
3       0                       osd.3   up      1
2       0.02998                 osd.2   up      1

I have changed the value of osd.5 weight to 3 in our decompiled file (/crush-decomplied) using vi editor.

Step 4 : I am going to recompile the modified CRUSH map.

[root@ceph-m3 ~]# crushtool -c /crush-decomplied -o /tmp/crush-compiled1

[root@ceph-m3 ~]# file /tmp/crush-compiled1
/tmp/crush-compiled1: MS Windows icon resource – 8 icons, 1-colors

Step 5 : Finally, time has come to set the recompiled CRUSH map.

[root@ceph-m3 ~]# ceph osd setcrushmap -i /tmp/crush-compiled1
set crush map

We can see the changed value of Weight in output of “ceph osd tree” for osd.5

[root@ceph-m3 ~]# ceph osd tree
# id    weight  type name       up/down reweight
-1      0.06998 root default
-2      0.01999         host ceph-osd1
0       0                       osd.0   up      1
4       0.01999                 osd.4   up      1
-3      0.01999         host ceph-osd2
1       0                       osd.1   up      1
5       0.03                    osd.5   up      1
-4      0.03            host ceph-m3
3       0                       osd.3   up      1
2       0.03                    osd.2   up      1

We have given more priority to OSD.5 for saving the data.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s