title: “Ceph on Openshift”
date: 2021-01-14T11:59:41
slug: ceph-on-openshift
Reaplio 3/2 Means: I will have 3 Replicas, the Client Receives OK after writing the second Replica. Third Replica will be written after/while sending ACK to the Client
Replication Factor 2/2 Means: Client Receives ACK if all Replicas are written
Show Status
ceph status
List Pools
ceph osd pool ls
or with id:
ceph osd lspools
List Pool Size (Replicas)
ceph osd pool get ocs-storagecluster-cephfilesystem-data0 size
List min Pool Size (when Client receives ACK)
ceph osd pool get ocs-storagecluster-cephfilesystem-data0 min\_size
List Numbers of Placement Groups
ceph osd pool get ocs-storagecluster-cephfilesystem-data0 pg\_num
Other Possible Parameters:
size|min\_size|pg\_num|pgp\_num|pgp\_num\_actual|crush\_rule|
hashpspool|nodelete|nopgchange|nosizechange|write\_fadvise\_dontneed|noscrub|nodeep-
scrub|hit\_set\_type|hit\_set\_period|hit\_set\_count|hit\_set\_fpp|use\_gmt\_hitset|target\_max\_
bytes|target\_max\_objects|cache\_target\_dirty\_ratio|cache\_target\_dirty\_high\_ratio|cache\_
target\_full\_ratio|cache\_min\_flush\_age|cache\_min\_evict\_age|min\_read\_recency\_for\_promote|
min\_write\_recency\_for\_promote|fast\_read|hit\_set\_grade\_decay\_rate|hit\_set\_search\_last\_n|
scrub\_min\_interval|scrub\_max\_interval|deep\_scrub\_interval|recovery\_priority|recovery\_
op\_priority|scrub\_priority|compression\_mode|compression\_algorithm|compression\_required\_
ratio|compression\_max\_blob\_size|compression\_min\_blob\_size|csum\_type|csum\_min\_block|
csum\_max\_block|allow\_ec\_overwrites|fingerprint\_algorithm|pg\_autoscale\_mode|pg\_
autoscale\_bias|pg\_num\_min|target\_size\_bytes|target\_size\_ratio {--yes-i-really-mean-it}
List OSDs
ceph osd ls
Get OSD Tree and Status (UP/DOWN)
ceph osd tree
Show PlacementGroups and their OSDs
ceph pg dump pgs\_brief
PG\_STAT STATE UP UP\_PRIMARY ACTING ACTING\_PRIMARY
3.d active+clean [2,0,1] 2 [2,0,1] 2
1.f active+clean [2,1,0] 2 [2,1,0] 2
PG_STAT: Inique Identifier
STATE: State
UP:
UP_PRIMARY:
ACTING: Includes these OSDs
ACTING_PRIMARY: This is the Primary OSD
More Dump Objects
all|summary|sum|delta|pools|osds|pgs|pgs\_brief [all|summary|sum|delta|pools
Exec into a OSD
oc rsh rook-ceph-osd-0-65547d878c-dfd8g
Put a File (text.txt) into a pool
Exec into ToolBox Pod and:
rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data put test.txt ./test.txt
List all Files in a Pool:
rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data ls
test.txt
Get a File (test.txt) from Ceph Storage and write it into new_file
rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data get test.txt new\_file
Monitormap:
ceph mon getmap > /tmp/map
monmaptool --print /tmp/map
monmaptool: monmap file /tmp/map
epoch 3
fsid 709412c8-caf8-4958-885d-66d8a918ba0e
last\_changed 2021-01-13 10:10:19.834120
created 2021-01-13 10:09:39.685560
min\_mon\_release 14 (nautilus)
0: [v2:172.31.132.253:3300/0,v1:172.31.132.253:6789/0] mon.a
1: [v2:172.31.186.223:3300/0,v1:172.31.186.223:6789/0] mon.b
2: [v2:172.31.147.23:3300/0,v1:172.31.147.23:6789/0] mon.c
OSDMap:
ceph osd getmap > /tmp/osdmap
osdmaptool --print /tmp/osdmap
osdmaptool: osdmap file '/tmp/osdmap'
epoch 40
fsid 709412c8-caf8-4958-885d-66d8a918ba0e
created 2021-01-13 10:09:40.731069
modified 2021-01-13 10:11:41.718711
flags sortbitwise,recovery\_deletes,purged\_snapdirs,pglog\_hardlimit
crush\_version 14
full\_ratio 0.85
backfillfull\_ratio 0.8
nearfull\_ratio 0.75
require\_min\_compat\_client luminous
min\_compat\_client jewel
require\_osd\_release nautilus
pool 1 'ocs-storagecluster-cephblockpool' replicated size 3 min\_size 2 crush\_rule 1 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 17 flags hashpspool,selfmanaged\_snaps stripe\_width 0 target\_size\_ratio 0.49 application rbd
removed\_snaps [1~3]
pool 2 'ocs-storagecluster-cephobjectstore.rgw.control' replicated size 3 min\_size 2 crush\_rule 2 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 13 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 3 'ocs-storagecluster-cephfilesystem-metadata' replicated size 3 min\_size 2 crush\_rule 3 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 15 flags hashpspool stripe\_width 0 pg\_autoscale\_bias 4 pg\_num\_min 16 recovery\_priority 5 application cephfs
pool 4 'ocs-storagecluster-cephfilesystem-data0' replicated size 3 min\_size 2 crush\_rule 4 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 16 flags hashpspool stripe\_width 0 target\_size\_ratio 0.49 application cephfs
pool 5 'ocs-storagecluster-cephobjectstore.rgw.meta' replicated size 3 min\_size 2 crush\_rule 5 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 16 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 6 'ocs-storagecluster-cephobjectstore.rgw.log' replicated size 3 min\_size 2 crush\_rule 6 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 21 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 7 'ocs-storagecluster-cephobjectstore.rgw.buckets.index' replicated size 3 min\_size 2 crush\_rule 7 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 26 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 8 'ocs-storagecluster-cephobjectstore.rgw.buckets.non-ec' replicated size 3 min\_size 2 crush\_rule 8 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 31 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 9 '.rgw.root' replicated size 3 min\_size 2 crush\_rule 9 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 36 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 10 'ocs-storagecluster-cephobjectstore.rgw.buckets.data' replicated size 3 min\_size 2 crush\_rule 10 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 40 flags hashpspool stripe\_width 0 application rook-ceph-rgw
max\_osd 3
osd.0 up in weight 1 up\_from 13 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.128.8.21:6800/79773,v1:10.128.8.21:6801/79773] [v2:10.128.8.21:6802/79773,v1:10.128.8.21:6803/79773] exists,up cfba7128-87ee-4af3-a5eb-8539153df4e6
osd.1 up in weight 1 up\_from 11 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.129.8.12:6800/77176,v1:10.129.8.12:6801/77176] [v2:10.129.8.12:6802/77176,v1:10.129.8.12:6803/77176] exists,up 682f2bbf-d4e0-4bdc-9bff-c78e5642fd17
osd.2 up in weight 1 up\_from 14 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.130.8.21:6800/80313,v1:10.130.8.21:6801/80313] [v2:10.130.8.21:6802/80313,v1:10.130.8.21:6803/80313] exists,up 61f186ca-ecd8-48f7-9c96-addcfee773b9
Calculate PlacementGroup (pg) by Poolname and Filename
ceph osd map ocs-storagecluster-cephfilesystem-data0 file2.txt
osdmap e40 pool 'ocs-storagecluster-cephfilesystem-data0' (4) object 'file2.txt' -> pg 4.2a096a1d (4.1d) -> up ([1,0,2], p1) acting ([1,0,2], p1)
Ceph Bluestore Tool
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
{
"/var/lib/ceph/osd/ceph-0/block": {
"osd\_uuid": "cfba7128-87ee-4af3-a5eb-8539153df4e6",
"size": 2199023255552,
"btime": "2021-01-13 10:10:50.648298",
"description": "main",
"bluefs": "1",
"ceph\_fsid": "709412c8-caf8-4958-885d-66d8a918ba0e",
"kv\_backend": "rocksdb",
"magic": "ceph osd volume v026",
"mkfs\_done": "yes",
"osd\_key": "AQApx/5fOpgaJRAAOSndBV7CVzBhCz/XT1Fs0A==",
"ready": "ready",
"require\_osd\_release": "14",
"whoami": "0"
}
}
Mounting the Bluestore Storage with ceph-objectstore-tool
1st Step: The OSD bust be stopped, otherwise you receive an error that the block device cannot be locked
Tell the ceph monitors not to “out” any OSDs from the crush map and not to start recovery and re-balance activities, to maintain the replica count.
ceph osd set noout
ceph -s
cluster:
id: 709412c8-caf8-4958-885d-66d8a918ba0e
health: HEALTH\_WARN
noout flag(s) set
(unset wit "ceph osd unset noout")
Stop (scale down) the Rook-Ceph Operator
oc scale --replicas=0 deployment/rook-ceph-operator
Now Restart the desired OSD Pod with a simple Sleep Comamnd as the OSD Process must not be run for mounting the Bluestore
oc edit deployment rook-ceph-osd-2
containers:
- args:
- -c
- 'while true; do sleep 3600; done'
command:
- /bin/sh
AND REMOVE LIVENESS PROBE!!!!
Exec into the OSD Pod
oc rsh rook-ceph-osd-2-6f5d645d4f-765wq
Execute Commands
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op list
Mount Bluestore to /mnt (Its running in foreground by default, so add a &)
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fuse --mountpoint /mnt/ &
List the Content:
sh-4.4# ls -l /mnt
total 0
drwx------. 0 root root 0 Jan 1 1970 1.0\_head
drwx------. 0 root root 0 Jan 1 1970 1.10\_head
drwx------. 0 root root 0 Jan 1 1970 1.11\_head
drwx------. 0 root root 0 Jan 1 1970 1.12\_head
drwx------. 0 root root 0 Jan 1 1970 1.13\_head
drwx------. 0 root root 0 Jan 1 1970 1.14\_head
drwx------. 0 root root 0 Jan 1 1970 1.15\_head
drwx------. 0 root root 0 Jan 1 1970 1.16\_head
drwx------. 0 root root 0 Jan 1 1970 1.17\_head
drwx------. 0 root root 0 Jan 1 1970 1.18\_head
List Volume
ceph fs subvolumegroup ls ocs-storagecluster-cephfilesystem (This will display csi)
ceph fs subvolume ls ocs-storagecluster-cephfilesystem csi
ceph fs subvolume info ocs-storagecluster-cephfilesystem csi-vol-96671632-5680-11eb-9d1a-0a580a80041a csi
