---

title: “Ceph on Openshift”
date: 2021-01-14T11:59:41
slug: ceph-on-openshift

---

Reaplio 3/2 Means: I will have 3 Replicas, the Client Receives OK after writing the second Replica. Third Replica will be written after/while sending ACK to the Client
Replication Factor 2/2 Means: Client Receives ACK if all Replicas are written

Show Status

ceph status

List Pools

ceph osd pool ls
or with id:
ceph osd lspools

List Pool Size (Replicas)

ceph osd pool get ocs-storagecluster-cephfilesystem-data0 size

List min Pool Size (when Client receives ACK)

ceph osd pool get ocs-storagecluster-cephfilesystem-data0 min\_size

List Numbers of Placement Groups

ceph osd pool get ocs-storagecluster-cephfilesystem-data0 pg\_num

Other Possible Parameters:

size|min\_size|pg\_num|pgp\_num|pgp\_num\_actual|crush\_rule|
 hashpspool|nodelete|nopgchange|nosizechange|write\_fadvise\_dontneed|noscrub|nodeep-
 scrub|hit\_set\_type|hit\_set\_period|hit\_set\_count|hit\_set\_fpp|use\_gmt\_hitset|target\_max\_
 bytes|target\_max\_objects|cache\_target\_dirty\_ratio|cache\_target\_dirty\_high\_ratio|cache\_
 target\_full\_ratio|cache\_min\_flush\_age|cache\_min\_evict\_age|min\_read\_recency\_for\_promote|
 min\_write\_recency\_for\_promote|fast\_read|hit\_set\_grade\_decay\_rate|hit\_set\_search\_last\_n|
 scrub\_min\_interval|scrub\_max\_interval|deep\_scrub\_interval|recovery\_priority|recovery\_
 op\_priority|scrub\_priority|compression\_mode|compression\_algorithm|compression\_required\_
 ratio|compression\_max\_blob\_size|compression\_min\_blob\_size|csum\_type|csum\_min\_block|
 csum\_max\_block|allow\_ec\_overwrites|fingerprint\_algorithm|pg\_autoscale\_mode|pg\_
 autoscale\_bias|pg\_num\_min|target\_size\_bytes|target\_size\_ratio {--yes-i-really-mean-it}

List OSDs

ceph osd ls

Get OSD Tree and Status (UP/DOWN)

ceph osd tree

Show PlacementGroups and their OSDs

ceph pg dump pgs\_brief
PG\_STAT STATE UP UP\_PRIMARY ACTING ACTING\_PRIMARY
3.d active+clean [2,0,1] 2 [2,0,1] 2
1.f active+clean [2,1,0] 2 [2,1,0] 2

PG_STAT: Inique Identifier
STATE: State
UP:
UP_PRIMARY:
ACTING: Includes these OSDs
ACTING_PRIMARY: This is the Primary OSD

More Dump Objects

all|summary|sum|delta|pools|osds|pgs|pgs\_brief [all|summary|sum|delta|pools

Exec into a OSD

oc rsh rook-ceph-osd-0-65547d878c-dfd8g

Put a File (text.txt) into a pool
Exec into ToolBox Pod and:

rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data put test.txt ./test.txt

List all Files in a Pool:

rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data ls
test.txt

Get a File (test.txt) from Ceph Storage and write it into new_file

rados -p ocs-storagecluster-cephobjectstore.rgw.buckets.data get test.txt new\_file

Monitormap:

ceph mon getmap > /tmp/map

monmaptool --print /tmp/map
monmaptool: monmap file /tmp/map
epoch 3
fsid 709412c8-caf8-4958-885d-66d8a918ba0e
last\_changed 2021-01-13 10:10:19.834120
created 2021-01-13 10:09:39.685560
min\_mon\_release 14 (nautilus)
0: [v2:172.31.132.253:3300/0,v1:172.31.132.253:6789/0] mon.a
1: [v2:172.31.186.223:3300/0,v1:172.31.186.223:6789/0] mon.b
2: [v2:172.31.147.23:3300/0,v1:172.31.147.23:6789/0] mon.c

OSDMap:

ceph osd getmap > /tmp/osdmap

osdmaptool --print /tmp/osdmap
osdmaptool: osdmap file '/tmp/osdmap'
epoch 40
fsid 709412c8-caf8-4958-885d-66d8a918ba0e
created 2021-01-13 10:09:40.731069
modified 2021-01-13 10:11:41.718711
flags sortbitwise,recovery\_deletes,purged\_snapdirs,pglog\_hardlimit
crush\_version 14
full\_ratio 0.85
backfillfull\_ratio 0.8
nearfull\_ratio 0.75
require\_min\_compat\_client luminous
min\_compat\_client jewel
require\_osd\_release nautilus

pool 1 'ocs-storagecluster-cephblockpool' replicated size 3 min\_size 2 crush\_rule 1 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 17 flags hashpspool,selfmanaged\_snaps stripe\_width 0 target\_size\_ratio 0.49 application rbd
 removed\_snaps [1~3]
pool 2 'ocs-storagecluster-cephobjectstore.rgw.control' replicated size 3 min\_size 2 crush\_rule 2 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 13 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 3 'ocs-storagecluster-cephfilesystem-metadata' replicated size 3 min\_size 2 crush\_rule 3 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 15 flags hashpspool stripe\_width 0 pg\_autoscale\_bias 4 pg\_num\_min 16 recovery\_priority 5 application cephfs
pool 4 'ocs-storagecluster-cephfilesystem-data0' replicated size 3 min\_size 2 crush\_rule 4 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 16 flags hashpspool stripe\_width 0 target\_size\_ratio 0.49 application cephfs
pool 5 'ocs-storagecluster-cephobjectstore.rgw.meta' replicated size 3 min\_size 2 crush\_rule 5 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 16 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 6 'ocs-storagecluster-cephobjectstore.rgw.log' replicated size 3 min\_size 2 crush\_rule 6 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 21 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 7 'ocs-storagecluster-cephobjectstore.rgw.buckets.index' replicated size 3 min\_size 2 crush\_rule 7 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 26 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 8 'ocs-storagecluster-cephobjectstore.rgw.buckets.non-ec' replicated size 3 min\_size 2 crush\_rule 8 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 31 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 9 '.rgw.root' replicated size 3 min\_size 2 crush\_rule 9 object\_hash rjenkins pg\_num 8 pgp\_num 8 autoscale\_mode on last\_change 36 flags hashpspool stripe\_width 0 pg\_num\_min 8 application rook-ceph-rgw
pool 10 'ocs-storagecluster-cephobjectstore.rgw.buckets.data' replicated size 3 min\_size 2 crush\_rule 10 object\_hash rjenkins pg\_num 32 pgp\_num 32 autoscale\_mode on last\_change 40 flags hashpspool stripe\_width 0 application rook-ceph-rgw

max\_osd 3
osd.0 up in weight 1 up\_from 13 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.128.8.21:6800/79773,v1:10.128.8.21:6801/79773] [v2:10.128.8.21:6802/79773,v1:10.128.8.21:6803/79773] exists,up cfba7128-87ee-4af3-a5eb-8539153df4e6
osd.1 up in weight 1 up\_from 11 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.129.8.12:6800/77176,v1:10.129.8.12:6801/77176] [v2:10.129.8.12:6802/77176,v1:10.129.8.12:6803/77176] exists,up 682f2bbf-d4e0-4bdc-9bff-c78e5642fd17
osd.2 up in weight 1 up\_from 14 up\_thru 38 down\_at 0 last\_clean\_interval [0,0) [v2:10.130.8.21:6800/80313,v1:10.130.8.21:6801/80313] [v2:10.130.8.21:6802/80313,v1:10.130.8.21:6803/80313] exists,up 61f186ca-ecd8-48f7-9c96-addcfee773b9

Calculate PlacementGroup (pg) by Poolname and Filename

ceph osd map ocs-storagecluster-cephfilesystem-data0 file2.txt
osdmap e40 pool 'ocs-storagecluster-cephfilesystem-data0' (4) object 'file2.txt' -> pg 4.2a096a1d (4.1d) -> up ([1,0,2], p1) acting ([1,0,2], p1)

Ceph Bluestore Tool

ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
{
 "/var/lib/ceph/osd/ceph-0/block": {
 "osd\_uuid": "cfba7128-87ee-4af3-a5eb-8539153df4e6",
 "size": 2199023255552,
 "btime": "2021-01-13 10:10:50.648298",
 "description": "main",
 "bluefs": "1",
 "ceph\_fsid": "709412c8-caf8-4958-885d-66d8a918ba0e",
 "kv\_backend": "rocksdb",
 "magic": "ceph osd volume v026",
 "mkfs\_done": "yes",
 "osd\_key": "AQApx/5fOpgaJRAAOSndBV7CVzBhCz/XT1Fs0A==",
 "ready": "ready",
 "require\_osd\_release": "14",
 "whoami": "0"
 }
}

Mounting the Bluestore Storage with ceph-objectstore-tool
1st Step: The OSD bust be stopped, otherwise you receive an error that the block device cannot be locked

Tell the ceph monitors not to “out” any OSDs from the crush map and not to start recovery and re-balance activities, to maintain the replica count.

ceph osd set noout
ceph -s
 cluster:
 id: 709412c8-caf8-4958-885d-66d8a918ba0e
 health: HEALTH\_WARN
 noout flag(s) set

(unset wit "ceph osd unset noout")

Stop (scale down) the Rook-Ceph Operator

oc scale --replicas=0 deployment/rook-ceph-operator

Now Restart the desired OSD Pod with a simple Sleep Comamnd as the OSD Process must not be run for mounting the Bluestore

oc edit deployment rook-ceph-osd-2
 containers:
 - args:
 - -c
 - 'while true; do sleep 3600; done'
 command:
 - /bin/sh
AND REMOVE LIVENESS PROBE!!!!

Exec into the OSD Pod

oc rsh rook-ceph-osd-2-6f5d645d4f-765wq

Execute Commands

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op list

Mount Bluestore to /mnt (Its running in foreground by default, so add a &)

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fuse --mountpoint /mnt/ &

List the Content:

sh-4.4# ls -l /mnt
total 0
drwx------. 0 root root 0 Jan 1 1970 1.0\_head
drwx------. 0 root root 0 Jan 1 1970 1.10\_head
drwx------. 0 root root 0 Jan 1 1970 1.11\_head
drwx------. 0 root root 0 Jan 1 1970 1.12\_head
drwx------. 0 root root 0 Jan 1 1970 1.13\_head
drwx------. 0 root root 0 Jan 1 1970 1.14\_head
drwx------. 0 root root 0 Jan 1 1970 1.15\_head
drwx------. 0 root root 0 Jan 1 1970 1.16\_head
drwx------. 0 root root 0 Jan 1 1970 1.17\_head
drwx------. 0 root root 0 Jan 1 1970 1.18\_head

List Volume

ceph fs subvolumegroup ls ocs-storagecluster-cephfilesystem (This will display csi)
ceph fs subvolume ls ocs-storagecluster-cephfilesystem csi
ceph fs subvolume info ocs-storagecluster-cephfilesystem csi-vol-96671632-5680-11eb-9d1a-0a580a80041a csi

---

title: “Install Ceph Toolbox on Openshift”
date: 2021-01-14T11:56:15
slug: install-ceph-toolbox-on-openshift

---

oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

---

title: “failed to create newCsiDriverClient: driver name rook-ceph.cephfs.csi.ceph.com not found in the list of registered CSI drivers”
date: 2020-05-27T09:35:00
slug: failed-to-create-newcsidriverclient-driver-name-rook-ceph-cephfs-csi-ceph-com-not-found-in-the-list-of-registered-csi-drivers

---

Bei Fehlermeldung:

failed to create newCsiDriverClient: driver name rook-ceph.cephfs.csi.ceph.com not found in the list of registered CSI drivers

Müssen alle Kubelets restarted werden.

---

title: “Pool Operations”
date: 2020-05-03T18:33:40
slug: pool-operations

---

Create Pool:
ceph osd pool create nextcloud 32 32
pool ‘nextcloud’ created

Delete Pool:
ceph osd pool delete 32 32 –yes-i-really-really-mean-it
pool ’32’ removed

---

title: “Troubleshooting Techniques”
date: 2020-04-22T09:15:36
slug: troubleshooting-techniques

---

https://github.com/rook/rook/blob/master/Documentation/ceph-common-issues.md

---

title: “Edit replication size”
date: 2020-04-18T09:15:29
slug: edit-replication-size

---

kubectl edit CephFilesystem myfs -n rook-ceph
spec:
 dataPools:
 - compressionMode: ""
 crushRoot: ""
 deviceClass: ""
 erasureCoded:
 algorithm: ""
 codingChunks: 0
 dataChunks: 0
 failureDomain: ""
 replicated:
 requireSafeReplicaSize: false
 size: 3
 targetSizeRatio: 0
 metadataPool:
 compressionMode: ""
 crushRoot: ""
 deviceClass: ""
 erasureCoded:
 algorithm: ""
 codingChunks: 0
 dataChunks: 0
 failureDomain: ""
 replicated:
 requireSafeReplicaSize: false
 size: 3
 targetSizeRatio: 0

---

title: “Replace OSD”
date: 2020-04-18T08:54:15
slug: replace-osd

---

Exec in Toolbox:

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

Useful Commands:

ceph status
ceph osd tree
ceph osd ls (for getting OSD ID)

Remove faulty OSD (in example osd with ID 1):

# ceph osd out osd.1
marked out osd.1.

# ceph osd crush remove osd.1
removed item id 1 name 'osd.1' from crush map

# ceph auth del osd.1
updated

# ceph osd rm osd.1
removed osd.1

Remove Deployment for faulty OSD (will be created again in next step)

# kubectl delete deployment -n rook-ceph rook-ceph-osd-1
deployment.extensions "rook-ceph-osd-1" deleted

Scale Operator Down and up again to detect noes OSDs:

# kubectl scale deployment rook-ceph-operator --replicas=0 -n rook-ceph
deployment.extensions/rook-ceph-operator scaled

# kubectl get pods --all-namespaces -o wide|grep operator

# kubectl scale deployment rook-ceph-operator --replicas=1 -n rook-ceph
deployment.extensions/rook-ceph-operator scaled

# kubectl get pods --all-namespaces -o wide|grep operator
rook-ceph-system rook-ceph-operator-76cf7f88f-g9pxr 0/1 ContainerCreating 0 2s kube-ceph02

New OSD prepare PODs should be created:

kubectl get pods -n rook-ceph -o wide

Check with:

ceph status
ceph osd tree
ceph osd list

In Case of problems:

On OSD Node:

view /var/lib/rook/rook-ceph/log/ceph-volume.log

Check Container logs:

kubectl logs -n rook-ceph rook-ceph-osd-prepare-k8s-node01-hl4rj

Check MGR Container for Recovery:

kubectl logs -f -n rook-ceph rook-ceph-mgr-a-7cb4ccffc6-dnz2q
debug 2020-04-18 08:57:59.352 7ffb18b9b700 0 log\_channel(cluster) log [DBG] : pgmap v940: 64 pgs: 2 active+undersized+degraded+remapped+backfilling, 55 active+clean, 7 active+undersized+degraded+remapped+backfill\_wait; 14 GiB data, 39 GiB used, 708 GiB / 750 GiB avail; 799 KiB/s rd, 5.0 MiB/s wr, 231 op/s; 2648/33588 objects degraded (7.884%); 5.7 MiB/s, 4 objects/s recovering
192.168.1.7 - - [18/Apr/2020:08:58:01] "GET / HTTP/1.1" 200 155 "" "kube-probe/1.18"
debug 2020-04-18 08:58:01.352 7ffb18b9b700 0 log\_channel(cluster) log [DBG] : pgmap v942: 64 pgs: 2 active+undersized+degraded+remapped+backfilling, 55 active+clean, 7 active+undersized+degraded+remapped+backfill\_wait; 14 GiB data, 39 GiB used, 708 GiB / 750 GiB avail; 738 KiB/s rd, 6.0 MiB/s wr, 270 op/s; 2648/33588 objects degraded (7.884%); 7.1 MiB/s, 5 objects/s recovering

---

title: “Erase Ceph Disc”
date: 2020-04-11T21:12:04
slug: erase-ceph-disc

---

sgdisk --zap-all /dev/vdb
dd if=/dev/zero of=/dev/vdb bs=1M count=100 oflag=direct,dsync
ls /dev/mapper/ceph-\* | xargs -I% -- dmsetup remove %
rm -rf /dev/ceph-\*
rm -rf /var/lib/rook
fdisk -l