9 views
# Ceph Ansible vs Rook differences This document highlights just the differences of the to deployments. If you want to see the full comparison you can find it [here](https://input.scs.community/iONqFJduQS-aN0URauTxqA). The 2nd Level headlines are the commands that were run to get the information in the tables below. In the Tables below in the headings the prefix `A` means Ansible and `R` means Rook. ## `ceph -s` **cluster** |cluster|ceph-ansible/osism|rook| |-------|------------------|----| |id|11111111-1111-1111-1111-111111111111|cb4de109-3c6e-4916-8cfc-725de573a56e| |health|HEALTH_OK|HEALTH_OK| **services** |services|ceph-ansible/osism|rook| |--------|------------------|----| |mon|3 daemons, quorum testbed-node-0,testbed-node-1,testbed-node-2 (age 6d)|3 daemons, quorum a,b,c (age 46h)| mgr|testbed-node-0(active, since 6d), standbys: testbed-node-1, testbed-node-2|a(active, since 46h), standbys: b| |mds|1/1 daemons up, 2 standby|| osd|6 osds: 6 up (since 6d), 6 in (since 6d)|6 osds: 6 up (since 46h), 6 in (since 46h)| |rgw|3 daemons active (3 hosts, 1 zones)|| ## `ceph config dump` |WHO|MASK|LEVEL|OPTION|A VALUE|A RO|R VALUE|R RO| |---|----|-----|------|-------|----|-------|----| |global||advanced|cluster_network|||192.168.16.0/20|\*| |global||basic|log_to_file|||true|| |global||advanced|mon_allow_pool_delete|||true|| |global||advanced|mon_allow_pool_size_one|||true|| |global||advanced|mon_cluster_log_file||||| |global||basic|ms_client_mode|||secure|\*| |global||basic|ms_cluster_mode|||secure|\*| |global||advanced|ms_osd_compress_mode|||force|| |global||basic|ms_service_mode|||secure|\*| |global||advanced|public_network|||192.168.16.0/20|\*| |global||advanced|rbd_default_map_options|||ms_mode=secure|\*| |mon||advanced|auth_allow_insecure_global_id_reclaim|||false|| |mgr||advanced|mgr/balancer/mode|||upmap|| |mgr||advanced|mgr/dashboard/PROMETHEUS_API_SSL_VERIFY|||false|\*| |mgr||advanced|mgr/dashboard/RGW_API_ACCESS_KEY|||QS5ENJ9SNPKH41FCSPSX|\*| |mgr||advanced|mgr/dashboard/RGW_API_SECRET_KEY|||K24kZDv96OUHWzxeiS0oKeZtenJKFVAkuot1pp8g|\*| |mgr||advanced|mgr/dashboard/server_addr|0.0.0.0|\*||| |mgr||advanced|mgr/dashboard/server_port|7000|\*|8443|\*| |mgr||advanced|mgr/dashboard/ssl|false|\*||| |mgr||advanced|mgr/dashboard/standby_behaviour|redirect|\*||| |mgr||advanced|mgr/dashboard/standby_error_status_code|503|\*||| |mgr||advanced|mgr/prometheus/rbd_stats_pools||||\*| |mds.cephfs-a||basic|mds_join_fs|||cephfs|| |mds.cephfs-b||basic|mds_join_fs|||cephfs|| |mds.cephfs-c||basic|mds_join_fs|||cephfs|| |mds.cephfs-d||basic|mds_join_fs|||cephfs|| |client.rgw.rgw.a||advanced|rgw_enable_usage_log|||true|| |client.rgw.rgw.a||advanced|rgw_log_nonexistent_bucket|||true|| |client.rgw.rgw.a||advanced|rgw_log_object_name_utc|||true|| |client.rgw.rgw.a||advanced|rgw_run_sync_thread|||true|| |client.rgw.rgw.a||advanced|rgw_zone|||rgw|\*| |client.rgw.rgw.a||advanced|rgw_zonegroup|||rgw|\*| ## `ceph config show mon.xxx` |NAME|A VALUE|A SOURCE|A OVERRIDES|A IGNORES|R VALUE|R SOURCE|R OVERRIDES|R IGNORES| |----|-------|--------|-----------|---------|-------|--------|-----------|---------| |auth_allow_insecure_global_id_reclaim|false|file|||false|mon||| |cluster_network|192.168.16.0/20|file||||default||mon| |err_to_stderr|true|cmdline|||true|default||| |keyring|$mon_data/keyring|default|||/etc/ceph/keyring-store/keyring|override|default[$mon_data/keyring]|| |log_to_file|false|default|||true|mon|default[false]|| |mon_allow_pool_delete|true|file|||true|mon||| |mon_allow_pool_size_one|||||true|mon||| |mon_cluster_log_file||||||mon||| |mon_data|/var/lib/ceph/mon/ceph-testbed-node-0|cmdline||||||| |mon_host|[v2:192.168.16.10:3300,v1:192.168.16.10:6789],[v2:192.168.16.11:3300,v1:192.168.16.11:6789],[v2:192.168.16.12:3300,v1:192.168.16.12:6789]|file|||[v2:192.168.16.12:3300]|override||| |mon_initial_members|testbed-node-0,testbed-node-1,testbed-node-2|file|||a|cmdline||| |ms_client_mode|||||secure|mon||| |ms_cluster_mode|||||secure|mon||| |ms_osd_compress_mode|||||force|mon||| |ms_service_mode|||||secure|mon||| |osd_pool_default_crush_rule|-1|file||||||| |osd_pool_default_min_size|0|file||||||| |osd_pool_default_size|2|file||||||| |public_addr|v2:192.168.16.10:0/0|cmdline|||v2:192.168.16.12:0/0|cmdline||| |public_network|192.168.16.0/20|file||||default||mon| |rbd_default_map_options||||||default||mon| |setuser_match_path|||||/var/lib/ceph/mon/ceph-a/store.db|cmdline||| ## `ceph config show mgr.xxx` |NAME|A VALUE|A SOURCE|A OVERRIDES|A IGNORES|R VALUE|R SOURCE|R OVERRIDES|R IGNORES |----|-------|--------|-----------|---------|-------|--------|-----------|--------- |auth_allow_insecure_global_id_reclaim|false|file||||||| |cluster_network|192.168.16.0/20|file|||192.168.16.0/20|mon||| |err_to_stderr|true|cmdline|||true|default||| |keyring|$mgr_data/keyring|default|||/etc/ceph/keyring-store/keyring|override|default[$mgr_data/keyring]|| |log_stderr_prefix|||||debug|default||| |log_to_file|false|default|||true|mon|default[false]|| |mon_allow_pool_delete|||||true|mon||| |mon_allow_pool_size_one|||||true|mon||| |mon_cluster_log_file||||||mon||| |mon_cluster_log_to_file|||||false|default||| |mon_cluster_log_to_stderr|||||true|default||| |mon_host|[v2:192.168.16.10:3300,v1:192.168.16.10:6789],[v2:192.168.16.11:3300,v1:192.168.16.11:6789],[v2:192.168.16.12:3300,v1:192.168.16.12:6789]|file|||[v2:192.168.16.12:3300],[v2:192.168.16.11:3300],[v2:192.168.16.10:3300]|override||| |mon_initial_members|testbed-node-0,testbed-node-1,testbed-node-2|file|||a,b,c|cmdline||| |ms_client_mode|||||secure|mon||| |ms_cluster_mode|||||secure|mon||| |ms_osd_compress_mode|||||force|mon||| |ms_service_mode|||||secure|mon||| |osd_pool_default_crush_rule|-1|file||||||| |osd_pool_default_min_size|0|file||||||| |osd_pool_default_size|2|file||||||| |public_addr|v2:192.168.16.10:0/0|override|||v2:192.168.16.12:0/0|override||| |public_network|192.168.16.0/20|file|||192.168.16.0/20|mon||| |rbd_default_map_options|||||ms_mode=secure|mon||| ## `ceph config show mds.xxx` |NAME|A VALUE|A SOURCE|A OVERRIDES|A IGNORES|R VALUE|R SOURCE|R OVERRIDES|R IGNORES| |----|-----|------|---------|-------|-----|------|---------|-------| |auth_allow_insecure_global_id_reclaim|false|file||||||| |cluster_network|192.168.16.0/20|file|||192.168.16.0/20|mon||| |err_to_stderr|true|cmdline|||true|default||| |keyring|$mds_data/keyring|default|||/etc/ceph/keyring-store/keyring|override|default[$mds_data/keyring]|| |log_stderr_prefix|||||debug|default||| |log_to_file|false|default|||true|mon|default[false]|| |mds_join_fs|||||cephfs|mon||| |mon_allow_pool_delete|||||true|mon||| |mon_allow_pool_size_one|||||true|mon||| |mon_cluster_log_file||||||mon||| |mon_cluster_log_to_file|||||false|default||| |mon_cluster_log_to_stderr|||||true|default||| |mon_host|[v2:192.168.16.10:3300,v1:192.168.16.10:6789],[v2:192.168.16.11:3300,v1:192.168.16.11:6789],[v2:192.168.16.12:3300,v1:192.168.16.12:6789]|file|||[v2:192.168.16.12:3300],[v2:192.168.16.11:3300],[v2:192.168.16.10:3300]|override||| |mon_initial_members|testbed-node-0,testbed-node-1,testbed-node-2|file|||a,b,c|cmdline||| |ms_client_mode|||||secure|mon||| |ms_cluster_mode|||||secure|mon||| |ms_osd_compress_mode|||||force|mon||| |ms_service_mode|||||secure|mon||| |osd_pool_default_crush_rule|-1|file||||||| |osd_pool_default_min_size|0|file||||||| |osd_pool_default_size|2|file||||||| |public_network|192.168.16.0/20|file|||192.168.16.0/20|mon||| |rbd_default_map_options|||||ms_mode=secure|mon||| ## `ceph config show osd.0` |NAME|A VALUE|A SOURCE|A OVERRIDES|A IGNORES|R VALUE|R SOURCE|R OVERRIDES|R IGNORES |----|-------|--------|-----------|---------|-------|--------|-----------|--------- |auth_allow_insecure_global_id_reclaim|false|file||||||| |cluster_network|192.168.16.0/20|file||| 192.168.16.0/20|mon||| |crush_location|||||root=defaulthost=testbed-node-0|cmdline||| |log_stderr_prefix|||||debug|default||| |log_to_file|false|default||| true|mon|default[false]|| |log_to_stderr|true|default||| true|default||| |mon_allow_pool_delete|||||true|mon||| |mon_allow_pool_size_one|||||true|mon||| |mon_cluster_log_file||||||mon||| |mon_cluster_log_to_file|||||false|default||| |mon_cluster_log_to_stderr|||||true|default||| |mon_host|[v2:192.168.16.10:3300,v1:192.168.16.10:6789],[v2:192.168.16.11:3300,v1:192.168.16.11:6789],[v2:192.168.16.12:3300,v1:192.168.16.12:6789]|file||| [v2:192.168.16.12:3300],[v2:192.168.16.11:3300],[v2:192.168.16.10:3300]|override||| |mon_initial_members|testbed-node-0,testbed-node-1,testbed-node-2|file||||||| |ms_client_mode|||||secure|mon||| |ms_cluster_mode|||||secure|mon||| |ms_osd_compress_mode|||||force|mon||| |ms_service_mode|||||secure|mon||| |osd_memory_target|11775980339|file|||26917057331|default||| |osd_pool_default_crush_rule|-1|file||||||| |osd_pool_default_min_size|0|file||||||| |osd_pool_default_size|2|file||||||| |public_network|192.168.16.0/20|file|||192.168.16.0/20|mon||| |rbd_default_map_options|||||ms_mode=secure|mon||| ## `radosgw-admin --show-config` |NAME|A VALUE|R VALUE| |----|-------|-------| |erasure_code_dir|/usr/lib/ceph/erasure-code|/usr/lib64/ceph/erasure-code| |host|e9b083485367|rook-ceph-tools-d894ddcfc-gfltt| |keyring|/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin|/etc/ceph/keyring| |mgr_disabled_modules|diskprediction_local|| |mon_host|192.168.16.10,192.168.16.11,192.168.16.12|192.168.16.12:6789,192.168.16.11:6789,192.168.16.10:6789| |osd_class_dir|/usr/lib/rados-classes|/usr/lib64/rados-classes| |plugin_dir|/usr/lib/ceph|/usr/lib64/ceph| ## location of settings Links listed here will point to the locations where settings are made for the osism/ceph-ansible setup. - [`mgr/dashboard`](https://github.com/osism/container-image-ceph-ansible/blob/main/files/playbooks/quincy/ceph-bootstrap-dashboard.yml) - [`auth_allow_insecure_global_id_reclaim`](https://github.com/osism/testbed/blob/main/environments/custom/playbook-workarounds-ceph.yml) - [`cluster_network`](https://github.com/osism/testbed/blob/main/environments/ceph/configuration.yml) - [`ceph_osd_pool_default_crush_rule`](https://github.com/ceph/ceph-ansible/blob/main/roles/ceph-defaults/vars/main.yml) and [task](https://github.com/ceph/ceph-ansible/blob/main/roles/ceph-osd/tasks/crush_rules.yml) - [`mon_host`](https://github.com/ceph/ceph-ansible/blob/main/roles/ceph-defaults/defaults/main.yml) - [`public_network`](https://github.com/osism/testbed/blob/main/environments/ceph/configuration.yml) - [`setgroup`](https://github.com/ceph/ceph-ansible/blob/main/roles/ceph-mon/tasks/deploy_monitors.yml) - [`setuser`](https://github.com/ceph/ceph-ansible/blob/main/roles/ceph-mon/tasks/deploy_monitors.yml)