diff --git a/knowledge/technical_manual/Ceph/.gitignore b/knowledge/technical_manual/Ceph/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Calamari/.gitignore b/knowledge/technical_manual/Ceph/Calamari/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Call-Home-Agent/.gitignore b/knowledge/technical_manual/Ceph/Call-Home-Agent/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Ansible/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Ansible/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Dashboard/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Dashboard/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Disk/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Disk/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Installer/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Installer/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Medic/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Medic/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Metrics/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Metrics/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Mgr-Plugings/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Mgr-Plugings/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Nano/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Nano/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Ceph-Volume/.gitignore b/knowledge/technical_manual/Ceph/Ceph-Volume/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Cephadm/.gitignore b/knowledge/technical_manual/Ceph/Cephadm/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Cephadm/administration/.gitignore b/knowledge/technical_manual/Ceph/Cephadm/administration/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Cephadm/administration/attribution.txt b/knowledge/technical_manual/Ceph/Cephadm/administration/attribution.txt new file mode 100644 index 000000000..25144f446 --- /dev/null +++ b/knowledge/technical_manual/Ceph/Cephadm/administration/attribution.txt @@ -0,0 +1,4 @@ +Title of work: Ceph administration knowledge +Link to work: https://www.ibm.com/docs/en/storage-ceph/7.1 +License of the work: Apache 2.0 +Creators name: IBM Corporation \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/Cephadm/administration/qna.yaml b/knowledge/technical_manual/Ceph/Cephadm/administration/qna.yaml new file mode 100644 index 000000000..d3b6a2a7e --- /dev/null +++ b/knowledge/technical_manual/Ceph/Cephadm/administration/qna.yaml @@ -0,0 +1,197 @@ +version: 3 +created_by: Pavan Govindraj +domain: opensource_storage +seed_examples: + - context: | + ## Administration + + Learn how to manage processes, monitor cluster states, manage users, and add and remove daemons for + IBM Storage Ceph. + + ### Key Administrative Tasks + - Manage processes: Start, stop, and restart Ceph daemons as needed. + - Monitor cluster health: Keep track of overall and component health within IBM Storage Ceph clusters. + - Stretch clusters: Configure stretch clusters for 2-site setups. + - Override Ceph behavior: Use runtime overrides to change cluster options. + - Manage Ceph users: Handle authentication and access control for Ceph users. + - Use ceph-volume: Manage OSDs with operations like prepare, list, create, activate, zap, and migrate. + - Performance benchmarking: Run benchmarks to assess cluster performance. + - Performance counters: Gather internal performance metrics and visualize them with various tools. + - mClock OSD scheduler: Implement quality of service (QoS) via the mClock queueing scheduler. + - BlueStore: The backend object store for OSD daemons, placing objects directly on the block device. + - Crimson (Technology Preview): A replacement for ceph-osd suited to modern low-latency, high-throughput + storage tech. + + questions_and_answers: + - question: | + How can a storage administrator manage Ceph daemons in IBM Storage Ceph? + answer: | + A storage administrator can start, stop, and restart Ceph daemons by type or instance using process + management tools. + - question: | + What is BlueStore in IBM Storage Ceph? + answer: | + BlueStore is the backend object store for OSD daemons, enabling direct object placement on the block device. + - question: | + What is the role of the mClock OSD scheduler in IBM Storage Ceph? + answer: | + The mClock OSD scheduler helps implement quality of service (QoS) in IBM Storage Ceph, ensuring better + resource management. + - context: | + ## High-level monitoring + + As a storage administrator, you can monitor the health of the Ceph daemons to ensure that they are up and running. + + High-level monitoring involves checking the storage cluster capacity to prevent exceeding its full ratio. + The IBM Storage Ceph Dashboard is commonly used for this, but command-line interfaces, Ceph admin sockets, + or Ceph APIs can also be used to monitor the cluster. + + ### Key Monitoring Tasks + - Check storage cluster health: Verify the cluster's health before reading or writing data. + - Watch cluster events: Use the CLI to watch events in real-time. + - Data usage calculation: The used value represents the actual amount of raw storage used. + - Check usage stats: Use the df command to see data usage and distribution. + - Check cluster status: Use the CLI to check the status of IBM Storage Ceph clusters. + - Monitor OSD utilization: Use the ceph osd df command to view OSD stats. + - Monitor Ceph Monitor status: Check the Ceph Monitor quorum status after starting the cluster. + + questions_and_answers: + - question: | + What tools can you use for high-level monitoring in IBM Storage Ceph? + answer: | + You can use the IBM Storage Ceph Dashboard, command-line interfaces, admin socket, or Ceph API + for high-level monitoring. + - question: | + How do you check the health of the storage cluster in IBM Storage Ceph? + answer: | + You can check the health of the storage cluster by using the IBM Storage Ceph Dashboard or + command-line interface tools. + - question: | + What does the used value represent in Ceph storage monitoring? + answer: | + The used value reflects the actual amount of raw storage consumed in the Ceph storage cluster. + - context: | + ## Using cephadm-ansible modules + + Use cephadm-ansible modules in Ansible playbooks to administer your IBM Storage Ceph cluster. + These modules wrap cephadm calls, + letting you write unique playbooks for managing your cluster. + + Note: cephadm-ansible modules support only the most critical tasks. For unsupported operations, + use the command or shell Ansible modules in your playbooks. + + ### Available cephadm-ansible modules + - cephadm_bootstrap: Bootstraps a storage cluster using Ansible. + - ceph_orch_host: Adds or removes hosts from the storage cluster. + - ceph_config: Sets or retrieves configuration options for IBM Storage Ceph. + - ceph_orch_apply: Applies service specifications to deploy Ceph services (mon, crash, mds, mgr, osd, etc.). + - ceph_orch_daemon: Manages Ceph daemon states (start, stop, restart). + - cephadm_registry_login: Logs into the Ceph registry. + + questions_and_answers: + - question: | + What is the purpose of cephadm-ansible modules in IBM Storage Ceph? + answer: | + The cephadm-ansible modules simplify writing Ansible playbooks by wrapping cephadm calls to administer + IBM Storage Ceph clusters. + - question: | + Which cephadm-ansible module allows bootstrapping a storage cluster? + answer: | + The cephadm_bootstrap module is used to bootstrap a storage cluster with Ansible. + - question: | + How can you manage Ceph daemon states using cephadm-ansible modules? + answer: | + You can manage Ceph daemon states (start, stop, restart) using the ceph_orch_daemon module in your + Ansible playbooks. + - context: | + ## Difference between Crimson and Classic Ceph OSD architecture + + The Crimson and Ceph OSD architectures differ, particularly with the use of the Seastar reactor in Crimson. + + ### Classic Ceph OSD architecture + In classic ceph-osd, a messenger thread reads a client message, places it in the OP queue, and an osd-op + thread-pool processes the message. The transaction is passed to BlueStore’s kv_queue, which waits for + rocksdb to commit. After committing, the finisher thread queues the message for the messenger thread to send. + + This architecture involves significant inter-thread coordination, with locks and queues that add latency, + especially as processor usage and the number of tasks scale. + + ### Crimson architecture + Unlike classic ceph-osd, Crimson avoids inter-thread coordination by completing I/O operations on a + single core without context switches when possible. It uses the Seastar framework, which pre-allocates one + thread per core, allowing operations to run on a single core without blocking. + + Seastar eliminates the need for locks and context-switches by ensuring nonblocking tasks own the CPU until + completion. This design fits Ceph OSD well, where PG shards handle I/Os. + + Note: Crimson-osd does not daemonize itself. Systemd handles daemonization on supported Linux + distributions. + + questions_and_answers: + - question: | + What is the main difference between Crimson and Classic Ceph OSD architecture? + answer: | + Crimson uses the Seastar reactor to avoid inter-thread coordination and context switches, + unlike Classic Ceph OSD. + - question: | + How does Crimson handle I/O operations compared to Classic Ceph OSD? + answer: | + Crimson completes I/O operations on a single core without blocking or context switches, + improving efficiency. + - question: | + What framework does Crimson use to handle I/O operations? + answer: | + Crimson uses the Seastar framework, an asynchronous engine that allocates one thread per core for I/O + operations. + - question: | + Why does Crimson-osd not daemonize itself? + answer: | + Crimson-osd doesn't daemonize as systemd handles this on supported Linux distributions. + - context: | + ## BlueStore fragmentation tool + + As a storage administrator, you can periodically check the fragmentation level of your BlueStore OSDs + with a simple command, whether the OSD is online or offline. + + Over time, free space on BlueStore OSDs becomes fragmented. Some fragmentation is normal, but excessive + fragmentation can degrade performance. + + The BlueStore fragmentation tool generates a fragmentation score ranging from 0 to 1. A score of 0 indicates no + fragmentation, while a score of 1 indicates severe fragmentation. + + ### Table 1. Fragmentation score range + + - 0.0 - 0.4: None to tiny fragmentation + - 0.4 - 0.7: Small, acceptable fragmentation + - 0.7 - 0.9: Considerable, but safe fragmentation + - 0.9 - 1.0: Severe fragmentation, leading to performance issues + + **Note**: If severe fragmentation is detected, contact IBM Support for assistance. + + You can check for fragmentation while the BlueStore OSD process is online or offline. + + questions_and_answers: + - question: | + What does a BlueStore fragmentation score of 1 indicate? + answer: | + A score of 1 indicates severe fragmentation, which can cause performance issues. + - question: | + How often should fragmentation checks be performed for BlueStore OSDs? + answer: | + As a storage administrator, you should periodically check fragmentation levels for both online and offline + BlueStore OSDs. + - question: | + What is considered an acceptable fragmentation score for BlueStore OSDs? + answer: | + A score between 0.0 and 0.7 is considered acceptable, with small to moderate levels of fragmentation. + - question: | + What should you do if a BlueStore OSD has severe fragmentation? + answer: | + If severe fragmentation is detected, contact IBM Support for assistance. + +document_outline: "Teach the Large Language Model about the Ceph administration" +document: + repo: git@github.ibm.com:Pavan-Govindraj/IBM-Ceph-data-Instructlab-taxanomy.git + commit: 7446af5 + patterns: + - IBM_Ceph_7.1/administration*.md diff --git a/knowledge/technical_manual/Ceph/Cephfs/.gitignore b/knowledge/technical_manual/Ceph/Cephfs/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Cephfs/smb/.gitignore b/knowledge/technical_manual/Ceph/Cephfs/smb/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Container/.gitignore b/knowledge/technical_manual/Ceph/Container/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Distribution/.gitignore b/knowledge/technical_manual/Ceph/Distribution/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/NFS-Ganesha/.gitignore b/knowledge/technical_manual/Ceph/NFS-Ganesha/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/NVMeOF-gateway/.gitignore b/knowledge/technical_manual/Ceph/NVMeOF-gateway/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/NVMeOF-gateway/attribution.txt b/knowledge/technical_manual/Ceph/NVMeOF-gateway/attribution.txt new file mode 100644 index 000000000..5c291427b --- /dev/null +++ b/knowledge/technical_manual/Ceph/NVMeOF-gateway/attribution.txt @@ -0,0 +1,4 @@ +Title of work: NVMEoF gateway Ceph knowledge +Link to work: https://www.ibm.com/docs/en/storage-ceph/7.1 +License of the work: Apache 2.0 +Creators name: IBM Corporation \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/NVMeOF-gateway/qna.yaml b/knowledge/technical_manual/Ceph/NVMeOF-gateway/qna.yaml new file mode 100644 index 000000000..03b8e35ac --- /dev/null +++ b/knowledge/technical_manual/Ceph/NVMeOF-gateway/qna.yaml @@ -0,0 +1,169 @@ +version: 3 +created_by: Pavan Govindraj +domain: opensource_storage +seed_examples: + - context: | + Storage administrators can install and configure an NVMe over Fabrics (NVMe-oF) gateway for an IBM Storage + Ceph cluster. With the Ceph NVMe-oF gateway, you can effectively run a fully integrated block storage + infrastructure with all features and benefits of a conventional Storage Area Network (SAN). + The NVMe-oF gateway integrates IBM Storage Ceph with the NVMe over TCP (NVMe/TCP) protocol to provide an + NVMe/TCP target that exports RADOS Block Device (RBD) images. The NVMe/TCP protocol allows clients, which + are known as initiators, to send NVMe-oF commands to storage devices, which are known as targets , over an + Internet Protocol network. Initiators can be Linux clients, VMWare clients, or both. For VMWare clients, the + NVMe/TCP volumes are shown as VMFS Datastore and for Linux clients, the NVMe/TCP volumes are shown as + block devices + questions_and_answers: + - question: What is the role of the NVMe-oF gateway in an IBM Storage Ceph cluster? + answer: | + The NVMe-oF gateway enables storage administrators to install and configure a fully integrated block storage + infrastructure for an IBM Storage Ceph cluster, providing features similar to a + conventional Storage Area Network (SAN). + - question: Which protocol does the Ceph NVMe-oF gateway use to provide block-level access? + answer: | + The Ceph NVMe-oF gateway uses the NVMe over TCP (NVMe/TCP) protocol to provide block-level access by + exporting RADOS Block Device (RBD) images. + - question: What types of clients are supported by the NVMe-oF gateway? + answer: | + The NVMe-oF gateway supports both Linux clients and VMware clients, where NVMe/TCP volumes appear as VMFS + Datastore for VMware clients and as block devices for Linux clients. + - context: | + The IBM Storage Ceph NVMe-oF gateway also supports NVMe Discovery. Each NVMe-oF gateway that runs in the Ceph + cluster also runs a Discovery Controller. The Discovery Controller reports all IP addresses of each of the + gateways in the group that are defined with listeners. For configuring information, see Configuring the + NVMe-oF gateway initiator. + Networking requirements: NVMe over Fabrics with the TCP protocol requires proper sizing of network bandwidth + and latency.NVMe-oF gateway performance best practices: Use best practices to ensure optimal gateway + performance. + Installing the NVMe-oF gateway: Before utilizing the Ceph NVMe-oF gateway, install the required software + packages via the command-line interface. + Configuring the NVMe-oF gateway target: Configure targets, LUNs, and clients by using the Ceph gateway + nvmeof-cli command. + Configuring the NVMe-oF gateway initiator: Configure the initiator to allow the NVMe/TCP protocol to + send NVMe-oF commands to targets over an Internet Protocol network. + Managing NVMe-oF gateways: Use these instructions to manage your NVMe-oF gateways. + questions_and_answers: + - question: | + What is the function of the Discovery Controller in the NVMe-oF gateway? + answer: | + The Discovery Controller reports the IP addresses of each gateway in the Ceph cluster defined with + listeners. + - question: | + What are the networking requirements for NVMe over Fabrics with the TCP protocol? + answer: | + The NVMe over Fabrics TCP protocol requires proper sizing of network bandwidth and low latency + to perform optimally. + - question: | + How do you configure the NVMe-oF gateway target? + answer: | + You configure the NVMe-oF gateway target, LUNs, and clients using the nvmeof-cli command-line utility. + - context: | + Installing the NVMe-oF gateway: Before you can utilize the benefits of the Ceph NVMe-oF gateway, + you must install the required software packages. You can install the Ceph NVMe-oF gateway by using the + command-line interface. + Each NVMe-oF gateway runs the Storage Performance Development Kit (SPDK) to provide NVMe-oF protocol support. + SPDK uses a + user-space implementation of the NVMe-oF protocol to interact with the Ceph librbd library, exposing RBD + images to NVMe-oF clients. This allows running a fully integrated block storage infrastructure with features + of a conventional SAN. + Deploying the NVMe-oF gateway: The Ceph NVMe-oF gateway is the NVMe-oF target node and also a Ceph client node. + It can either be a stand-alone node or collocated on a Ceph Object Storage Daemon (OSD) node. + Configuring mTLS authentication: Configure mutual TLS (mTLS) to secure connections between the command-line + interface gRPC client and the Ceph NVMe-oF gateway gRPC server. + questions_and_answers: + - question: | + What software is required to install the NVMe-oF gateway? + answer: | + You need to install the required software packages via the command-line interface to use the NVMe-oF gateway. + - question: | + What is the function of SPDK in the Ceph NVMe-oF gateway? + answer: | + SPDK provides NVMe-oF protocol support and interacts with the Ceph librbd library to expose RBD images to + NVMe-oF clients. + - question: | + Where can the Ceph NVMe-oF gateway be deployed? + answer: | + The Ceph NVMe-oF gateway can be deployed as a stand-alone node or collocated on a + Ceph Object Storage Daemon (OSD) node. + - context: | + Managing load balancing with scale-up and scale-down: Conduct load balancing after adding and before + removing gateways. + It is important to run load balancing commands after certain events: + 1. After adding new gateway nodes (scale-up): Rearrange the load balancing groups of the namespaces + to ensure they are + equally distributed across all existing gateways. + 2. Before removing a gateway node (scale-down): Change the namespace load balancing group IDs of the + gateways that are planned to be removed to avoid inaccessible namespaces. + Use the change_load_balancing_group command to update the namespaces affected by the scaling changes. + Example command: + ``` + nvmeof-cli --server-address GATEWAY_IP --server-port SERVER_PORT namespace change_load_balancing_group + --subsystem SUBSYSTEM [--nsid NAMESPACE_NSID | --uuid NAMESPACE_UUID] -l LOAD_BALANCING_GROUP + ``` + Use either --NSID or --UUID parameters to specify the namespace. + questions_and_answers: + - question: | + When should load balancing be performed in NVMe-oF gateway management? + answer: | + Load balancing should be done after adding new gateway nodes (scale-up) and before removing gateways + (scale-down). + - question: | + What command is used to change the load balancing group in NVMe-oF gateways? + answer: | + The command `change_load_balancing_group` is used to change the load balancing group of affected + namespaces. + - question: | + Why is it necessary to update load balancing groups before scaling down gateways? + answer: | + To prevent namespaces from becoming inaccessible by reassigning their load balancing group IDs before + removing gateways. + - context: | + Use the commands listed here to manage your SPDK logs. + + ### Before you begin + Managing NVMe-oF subsystems with the CLI requires the nvmeof-cli alias setup. For more details, + see step 1 of Defining an NVMe-oF subsystem. + + ### Getting the SPDK log level + Use the `spdk_log_level get` command to retrieve the SPDK version in JSON format. + Example: + ``` + nvmeof-cli --server-address NODE_IP --server-port SERVER_PORT spdk_log_level get --format json --output stdio + ``` + + ### Setting the SPDK log level + Use the `spdk_log_level set` command to set the log level for the SPDK output and NVMe over Fabrics + target (nvmf) log flags. Available log levels are DEBUG, INFO, NOTICE, WARNING, and ERROR. + Example: + ``` + nvmeof-cli --server-address NODE_IP --server-port SERVER_PORT spdk_log_level set --level ERROR --print + ERROR --format json --output stdio + ``` + + ### Disabling SPDK log flags + Use the `spdk_log_level disable` command to remove the log level for the SPDK output and NVMe over Fabrics + target (nvmf) log flags. + Example: + ``` + nvmeof-cli --server-address NODE_IP --server-port SERVER_PORT spdk_log_level disable --format json --output + stdio + ``` + questions_and_answers: + - question: | + How can you retrieve the SPDK log level in JSON format? + answer: | + Use the command `spdk_log_level get` with `nvmeof-cli`, specifying the server address and port, + and the format as JSON. + - question: | + What are the available log levels for the SPDK NVMe over Fabrics target (nvmf)? + answer: | + The available SPDK nvmf log levels are DEBUG, INFO, NOTICE, WARNING, and ERROR. + - question: | + How do you disable the SPDK log flags for NVMe over Fabrics target (nvmf)? + answer: | + Use the command `spdk_log_level disable` with `nvmeof-cli`, specifying the server address and + port in JSON format. +document_outline: Teach the Large Language Model about the NVMEoF feature of Ceph +document: + repo: git@github.ibm.com:Pavan-Govindraj/IBM-Ceph-data-Instructlab-taxanomy.git + commit: 7446af5 + patterns: [IBM_Ceph_7.1/nvmeof*.md] diff --git a/knowledge/technical_manual/Ceph/RADOS/.gitignore b/knowledge/technical_manual/Ceph/RADOS/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/RBD-Mirror/.gitignore b/knowledge/technical_manual/Ceph/RBD-Mirror/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/RBD/.gitignore b/knowledge/technical_manual/Ceph/RBD/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/RGW-Multisite/.gitignore b/knowledge/technical_manual/Ceph/RGW-Multisite/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/RGW/.gitignore b/knowledge/technical_manual/Ceph/RGW/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Rook/.gitignore b/knowledge/technical_manual/Ceph/Rook/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/Security/.gitignore b/knowledge/technical_manual/Ceph/Security/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Calamari/.gitignore b/knowledge/technical_manual/Ceph/upstream/Calamari/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Call-Home-Agent/.gitignore b/knowledge/technical_manual/Ceph/upstream/Call-Home-Agent/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Ansible/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Ansible/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Dashboard/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Dashboard/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Disk/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Disk/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Installer/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Installer/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Medic/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Medic/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Metrics/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Metrics/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Mgr-Plugings/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Mgr-Plugings/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Nano/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Nano/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Ceph-Volume/.gitignore b/knowledge/technical_manual/Ceph/upstream/Ceph-Volume/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Cephadm/.gitignore b/knowledge/technical_manual/Ceph/upstream/Cephadm/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Cephadm/administration/.gitignore b/knowledge/technical_manual/Ceph/upstream/Cephadm/administration/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/.gitignore b/knowledge/technical_manual/Ceph/upstream/Cephfs/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/attribution.txt b/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/attribution.txt new file mode 100644 index 000000000..d3f9b0974 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/attribution.txt @@ -0,0 +1,4 @@ +Title of work: Deploying Metadata Servers +Link to work: https://docs.ceph.com/en/squid/cephfs/add-remove-mds/ +License of the work: Apache 2.0 +Creators name: Amarnath \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/qna.yaml b/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/qna.yaml new file mode 100644 index 000000000..a27570c2c --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/Cephfs/add_remove_mds/qna.yaml @@ -0,0 +1,142 @@ +version: 3 +created_by: Amarnath +domain: opensource_storage +seed_examples: + - context: | + Deploying Metadata Servers (MDS) is essential for each CephFS file system. The cluster operator generally uses + automated tools like Rook or Ansible (via ceph-ansible playbooks) to deploy MDS servers. Systemd commands may + also be used for deployment on bare-metal systems. Refer to the MDS Configuration Reference for details on + configuring metadata servers. + questions_and_answers: + - question: | + What tools are recommended for deploying MDS in CephFS? + answer: | + Rook and Ansible (via ceph-ansible playbooks) are recommended for deploying MDS in CephFS. + - question: | + Can systemd commands be used for deploying MDS? + answer: | + Yes, systemd commands can be used, especially for bare-metal deployments. + - question: | + Why is the MDS Configuration Reference important? + answer: | + The MDS Configuration Reference provides details on how to configure metadata servers effectively. + - context: | + Provisioning hardware for an MDS requires careful consideration of CPU and RAM. MDS is single-threaded and + CPU-bound, typically requiring 2-3 CPU cores under heavy loads. High-performance CPUs are recommended for + optimal performance, as future versions may utilize multiple cores. RAM is critical for managing a distributed + metadata cache. The default cache size is 4GB, but 8GB of RAM is recommended, and 64GB or more may be necessary + for large clusters with over 1000 clients. Over-provisioning hardware is a best practice for flexibility and + scalability in bare-metal deployments. + questions_and_answers: + - question: | + How many CPU cores are typically required by an MDS under heavy loads? + answer: | + An MDS typically requires 2-3 CPU cores under heavy loads. + - question: | + Why is high-performance CPU important for MDS? + answer: | + High-performance CPUs ensure optimal MDS performance and are necessary for handling client requests \ + efficiently. + - question: | + What is the recommended amount of RAM for an MDS? + answer: | + At least 8GB of RAM is recommended, with 64GB or more for large clusters. + - question: | + What is the default cache size for MDS? + answer: | + The default cache size for MDS is 4GB. + - question: | + Why is over-provisioning hardware recommended for MDS? + answer: | + Over-provisioning allows flexibility to scale and handle future performance demands. + + - context: | + Adding an MDS to a CephFS cluster involves the following steps: + 1. Create the directory `/var/lib/ceph/mds/ceph-${id}` to store the MDS keyring. + 2. Generate the authentication key using the command: + ``` + sudo ceph auth get-or-create mds.${id} mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' \ + /var/lib/ceph/mds/ceph-${id}/keyring + ``` + 3. Start the MDS service using: + ``` + sudo systemctl start ceph-mds@${id} + ``` + 4. Optionally, configure the file system the MDS should join: + ``` + ceph config set mds.${id} mds_join_fs ${fs} + ``` + questions_and_answers: + - question: | + What is the first step in adding an MDS to a CephFS cluster? + answer: | + Create the directory `/var/lib/ceph/mds/ceph-${id}` to store the MDS keyring. + - question: | + How is the authentication key for MDS generated? + answer: | + Use the command `sudo ceph auth get-or-create mds.${id}` to generate the authentication key. + - question: | + How do you start the MDS service after adding it? + answer: | + Start the service with the command `sudo systemctl start ceph-mds@${id}`. + - question: | + What command is used to configure the file system an MDS should join? + answer: | + Use `ceph config set mds.${id} mds_join_fs ${fs}` to configure the file system. + + - context: | + Removing an MDS from a CephFS cluster involves: + 1. Optionally, add a replacement MDS to avoid downtime. + 2. Stop the MDS to be removed using: + ``` + sudo systemctl stop ceph-mds@${id} + ``` + 3. Remove the MDS directory: + ``` + sudo rm -rf /var/lib/ceph/mds/ceph-${id} + ``` + The MDS notifies the Ceph monitors of its shutdown, enabling failover to a standby MDS if available. + questions_and_answers: + - question: | + What is the first step in removing an MDS from a CephFS cluster? + answer: | + Optionally, add a replacement MDS to avoid downtime. + - question: | + How do you stop the MDS to be removed? + answer: | + Use the command `sudo systemctl stop ceph-mds@${id}` to stop the MDS. + - question: | + How do you remove the MDS directory after stopping the service? + answer: | + Use the command `sudo rm -rf /var/lib/ceph/mds/ceph-${id}` to remove the directory. + - question: | + What happens when the MDS is stopped? + answer: | + The MDS notifies the Ceph monitors, enabling failover to a standby MDS if available. + + - context: | + CephFS is designed for high availability, supporting standby MDS for rapid failover. To maximize this benefit, + deploy MDS daemons across multiple nodes. Co-locating MDS with other Ceph daemons is effective if configured + within hardware limits. Proper provisioning of CPU and RAM is necessary for ensuring optimal performance. + questions_and_answers: + - question: | + What is the role of standby MDS in CephFS? + answer: | + Standby MDS provides rapid failover and ensures high availability. + - question: | + Why should MDS daemons be distributed across multiple nodes? + answer: | + Distributing MDS daemons prevents single-node failures and ensures high availability. + - question: | + Can MDS be co-located with other Ceph daemons? + answer: | + Yes, MDS can be co-located with other Ceph daemons if hardware limits are respected. + - question: | + Why is high availability important for CephFS? + answer: | + High availability ensures that the file system remains accessible even in case of hardware failures. +document_outline: Teach the Large Language Model about the Deploying MDS of CephFS +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/cephfs/add-remove-mds*.md] diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/.gitignore b/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/attribution.txt b/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/attribution.txt new file mode 100644 index 000000000..01a495d09 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/attribution.txt @@ -0,0 +1,4 @@ +Title of work: SMB Ceph knowledge +Link to work: https://docs.ceph.com/en/squid/cephadm/services/smb/#smb-service +License of the work: Apache 2.0 +Creators name: Mohit Bisht diff --git a/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/qna.yaml b/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/qna.yaml new file mode 100644 index 000000000..1634f1f53 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/Cephfs/smb/qna.yaml @@ -0,0 +1,292 @@ +version: 3 +created_by: Mohit Bisht +domain: opensource_storage +seed_examples: + - context: | + Storage administrators can install and configure an SMB with ceph. + The Ceph MGR module for SMB provides an interface for deploying and controlling clusters of Samba services and + managing SMB shares. In the SMB manager module, a cluster is a logical management unit that maps to one or more + managed Samba services that itself is a cluster and may or may not use any high-availability mechanisms. + Enabling the SMB services: + Before communicating with the Ceph MGR module for SMB, it must be enabled first: + ```bash + $ ceph mgr module enable smb + ``` + Usually, SMB is layered on CephFS subvolumes or subvolume groups. Do not create any SMB shares on top of CephFS + root volumes. You can share a subvolume once or more within a single SMB cluster context. However, defining only + one SMB share per CephFS subvolume is recommended to keep complexity low. + questions_and_answers: + - question: | + What is Samba and how does it integrate with Ceph? + answer: | + Samba is an open-source implementation of the Server Message Block (SMB) protocol that provides file and + print services across different operating systems, such as Windows, Mac OS (X), and Linux. The Ceph MGR + module for SMB integrates Samba with Ceph, providing an interface for deploying and managing clusters of + Samba services and SMB shares. + - question: | + How do you enable the SMB services for Ceph, and what restrictions apply to SMB shares? + answer: | + To enable SMB services for Ceph, you run the command: `$ ceph mgr module enable smb`. SMB should only be + layered on CephFS subvolumes or subvolume groups. It is not recommended to create SMB shares on CephFS + root volumes, and each CephFS subvolume should have only one SMB share to keep complexity low. + - question: | + What are the protocol support limitations for Ceph managed SMB services? + answer: | + Ceph managed SMB services only support SMB2 and SMB3. The older SMB1 protocol, also known as CIFS, is not + supported. Additionally, mixed protocol support for NFS and SMB on the same subvolume is not allowed as it + may cause data corruption. + - context: | + The Ceph MGR module for SMB offers a way to deploy and manage clusters of Samba services and configure SMB + shares.A cluster in the SMB manager module refers to a logical management unit that can map to one or more + managed Samba services, possibly with high-availability features. + Managing SMB clusters involves creating, listing, and removing clusters using imperative commands. + To create a new cluster, the `ceph smb cluster create` command is used, specifying the cluster ID and + authentication mode, either as user or active-directory. Listing clusters ensures efficient management + and uses the `ceph smb cluster ls`command. Clusters can be removed using `ceph smb cluster rm`. + SMB shares can be created and managed by mapping CephFS volumes and paths to specific clusters. The `ceph smb + share create` command is used to establish shares, specifying parameters like cluster ID, share ID, and CephFS + volume. Shares can be removed with the `ceph smb share rm` command, and shares can be listed using + `ceph smb share ls`. + questions_and_answers: + - question: | + How can you create an SMB cluster using Ceph? + answer: | + To create an SMB cluster, use `ceph smb cluster create`, specifying the cluster ID and authentication mode. + For example, specify 'user' or 'active-directory' authentication and provide necessary credentials. + - question: | + What are the commands to manage SMB shares in Ceph? + answer: | + To create a share, use `ceph smb share create` with details like cluster ID, share ID, and CephFS volume. Use + `ceph smb share rm` to remove a share and `ceph smb share ls` to list shares. + - question: | + What is the purpose of listing SMB clusters in Ceph? + answer: | + Listing SMB clusters ensures efficient management and robust data protection. Use `ceph smb cluster ls` to + view the clusters, with the option to format the output in JSON or YAML. + - question: | + What parameters are required to create an SMB cluster in Ceph? + answer: | + Parameters include cluster ID (a unique identifier), authentication mode (user or active-directory), and + options like domain realm, user credentials, custom DNS, placement, and clustering preferences. + - question: | + How can you remove an SMB share from a Ceph cluster? + answer: | + To remove an SMB share, use the command `ceph smb share rm`, specifying the cluster ID and share ID. This + removes the specified share from the cluster. + - question: | + How do you ensure Ceph's SMB shares are read-only? + answer: | + To create a read-only SMB share, use the `--readonly` flag with the `ceph smb share create` command. This + prevents clients from modifying the shared data. + - question: | + Can you mix NFS and SMB file services on the same CephFS subvolume? + answer: | + No, mixing NFS and SMB file services on the same CephFS subvolume is not supported. Doing so can potentially + cause data corruption. + - context: | + Integrating Active Directory with SMB using the imperative method requires setting the `auth_mode` parameter to + active-directory and specifying the cluster ID for the AD setup. The `ceph smb cluster create` command creates a + new logical cluster identified by the cluster ID and configures it to use Active Directory authentication. + Parameters like domain realm, AD join user credentials, custom DNS, placement, and clustering options need to be + specified to complete the setup. + questions_and_answers: + - question: | + How do you set up an SMB cluster with Active Directory integration using Ceph? + answer: | + Use `ceph smb cluster create` with parameters like cluster ID, `auth_mode: active-directory`, domain realm, + AD join credentials, custom DNS, placement, and clustering. + - question: | + What command is used to create an SMB cluster with Active Directory in Ceph? + answer: | + Command `ceph smb cluster create smb1 active-directory --domain_realm DOMAIN_REALM --domain_join_user_pass + % --custom_dns --placement --clustering `. + - question: | + What parameters are necessary for integrating Active Directory with SMB in Ceph? + answer: | + Necessary parameters include the cluster ID, `auth_mode: active-directory`, domain realm, domain join user + credentials, custom DNS, placement details, and clustering options. + - context: | + Composing resource spec for the declarative method involves learning how to create SMB clusters and shares + using resource spec files in either YAML or JSON format.This method is an alternative to the imperative + approach and is supported by the SMB manager module. You use the `ceph smb apply` command to process these + specifications, similar to how service specifications are handled with Ceph orchestration using cephadm. + To create SMB clusters and shares declaratively: + - Compose a resource specification file in YAML or JSON. + - Apply the resource file using `ceph smb apply`. + Example YAML list of resources: + ```yaml + - resource_type: ceph.smb.cluster + cluster_id: rhumba + # ... other fields skipped for brevity ... + - resource_type: ceph.smb.cluster + cluster_id: salsa + # ... other fields skipped for brevity ... + - resource_type: ceph.smb.share + cluster_id: salsa + share_id: foo + ``` + YAML format supporting multiple documents: + ```yaml + --- + resource_type: ceph.smb.cluster + cluster_id: rhumba + # ... other fields skipped for brevity ... + --- + resource_type: ceph.smb.cluster + cluster_id: salsa + # ... other fields skipped for brevity ... + --- + resource_type: ceph.smb.share + cluster_id: salsa + share_id: foo + ``` + questions_and_answers: + - question: | + What is the purpose of composing resource specifications for SMB clusters? + answer: | + Resource specifications provide a declarative way to configure SMB clusters and shares, as an alternative + to the imperative method. + - question: | + How can you create an SMB cluster using the declarative method? + answer: | + Use a resource specification file with `resource_type: ceph.smb.cluster` and apply it using `ceph smb apply`. + - question: | + What are the formats supported for resource specifications in the declarative method? + answer: | + The formats supported are YAML and JSON. + - question: | + How do you view SMB cluster details using the declarative method? + answer: | + Use the `ceph smb show` command to display the resources applied to the Ceph cluster configuration. + - question: | + How can you remove an SMB share using the declarative method? + answer: | + Set `intent: removed` in the resource specification for the SMB share and apply it to remove the share. + - context: | + Integrating Active Directory with SMB using the declarative method involves specifying `auth_mode` as + active-directory in the resource specification file. Here's an example: + ```yaml + resource_type: ceph.smb.cluster + cluster_id: tango + auth_mode: active-directory + domain_settings: + realm: DOMAIN1.SINK.TEST + join_sources: + - source_type: resource + ref: join1-admin + custom_dns: + - "192.168.76.204" + placement: + count: 1 + ``` + Declarative method configuration example: Setting up an SMB cluster and share involves writing a YAML file + with the required configurations and applying it. + Example YAML: + ```yaml + resources: + - resource_type: ceph.smb.cluster + cluster_id: tango + auth_mode: active-directory + domain_settings: + realm: DOMAIN1.SINK.TEST + join_sources: + - source_type: resource + ref: join1-admin + custom_dns: + - "192.168.76.204" + placement: + count: 1 + label: ilovesmb + - resource_type: ceph.smb.join.auth + auth_id: join1-admin + auth: + username: Administrator + password: Passw0rd + - resource_type: ceph.smb.share + cluster_id: tango + share_id: cache + cephfs: + volume: cephfs + subvolumegroup: smb1 + subvolume: cache + path: / + - resource_type: ceph.smb.share + cluster_id: tango + share_id: sp1 + name: "Staff Pics" + cephfs: + volume: cephfs + path: /pics + subvolumegroup: smb1 + subvolume: staff + ``` + Save this to `resources.yaml` and run: + ```bash + $ ceph smb apply -i resources.yaml + ``` + To remove resources, use `removed.yaml`: + ```yaml + resources: + - resource_type: ceph.smb.cluster + cluster_id: tango + intent: removed + - resource_type: ceph.smb.join.auth + auth_id: join1-admin + intent: removed + - resource_type: ceph.smb.share + cluster_id: tango + share_id: cache + intent: removed + - resource_type: ceph.smb.share + cluster_id: tango + share_id: sp1 + intent: removed + ``` + Remove resources with: + ```bash + $ ceph smb apply -i removed.yaml + ``` + SMB File System share access and control: Clients can access shares using SMB support in Windows, MacOS X, + or Linux. Use the format `\\\`, ensuring IP addresses are added in the DNS. If the + cluster and client are in the same AD Domain, single sign-on occurs; otherwise, enter a username and password. + questions_and_answers: + - question: | + How do you specify Active Directory authentication for an SMB cluster? + answer: | + Set `auth_mode: active-directory` in the resource specification file for the SMB cluster. + - question: | + What is an example configuration for setting up an SMB cluster using Active Directory? + answer: | + ```yaml + resource_type: ceph.smb.cluster + cluster_id: tango + auth_mode: active-directory + domain_settings: + realm: DOMAIN1.SINK.TEST + join_sources: + - source_type: resource + ref: join1-admin + custom_dns: + - "192.168.76.204" + placement: + count: 1 + ``` + - question: | + How do you apply the resource specification file to create an SMB cluster and shares? + answer: | + Use the command `ceph smb apply -i resources.yaml` to apply the resource file. + - question: | + How can you remove SMB resources using the declarative method? + answer: | + Create a `removed.yaml` file specifying the intent as `removed` for each resource and use + `ceph smb apply -i removed.yaml`. + - question: | + How can clients connect to an SMB share? + answer: | + Use `\\\` in Windows, MacOS X, or Linux. Ensure IP addresses are added in the DNS, + and enter credentials if needed. +document_outline: Teach the Large Language Model about the SMB feature of Ceph +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/cephadm/services/smb*.md] diff --git a/knowledge/technical_manual/Ceph/upstream/Container/.gitignore b/knowledge/technical_manual/Ceph/upstream/Container/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Distribution/.gitignore b/knowledge/technical_manual/Ceph/upstream/Distribution/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/NFS-Ganesha/.gitignore b/knowledge/technical_manual/Ceph/upstream/NFS-Ganesha/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/.gitignore b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/attribution.txt new file mode 100644 index 000000000..f8f7e1b62 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/attribution.txt @@ -0,0 +1,4 @@ +Title of work: NVMe/TCP Initiator for VMware ESX +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-initiator-esx/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/qna.yaml new file mode 100644 index 000000000..77f01e425 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/esxi/qna.yaml @@ -0,0 +1,85 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: opensource_storage +seed_examples: + - context: | + To configure an ESXi host to use the Ceph NVMe-oF gateway, ensure the prerequisites are met. These include an + ESXi host running VMware vSphere Hypervisor (version 7.0U3 or later), a deployed Ceph NVMe-oF gateway, a Ceph + cluster configured for NVMe-oF, and a defined subsystem within the gateway. Proper setup enables the ESXi host + to achieve high-performance storage with reduced latency and increased bandwidth. + questions_and_answers: + - question: What version of VMware ESXi is required for configuring NVMe-oF? + answer: | + VMware vSphere Hypervisor (ESXi) 7.0U3 or later is required. + - question: What is needed within the Ceph cluster for NVMe-oF functionality? + answer: | + The Ceph cluster must be configured for NVMe-oF. + - question: What does the Ceph NVMe-oF gateway provide? + answer: | + The Ceph NVMe-oF gateway enables ESXi hosts to use high-performance NVMe-oF storage. + - question: What is the purpose of defining a subsystem in the Ceph NVMe-oF gateway? + answer: | + The subsystem acts as a logical representation for NVMe-oF services, enabling communication with initiators. + - context: | + Configuring VMware ESXi for NVMe/TCP involves enabling NVMe/TCP on a NIC and tagging a VMKernel NIC to permit + NVMe/TCP traffic. This setup is critical to ensure the ESXi host can communicate with NVMe-oF gateways. + Enabling NVMe/TCP enhances storage performance by reducing latency and increasing bandwidth. + questions_and_answers: + - question: How do you enable NVMe/TCP on a NIC in VMware ESXi? + answer: | + Use the command: `esxcli nvme fabric enable --protocol TCP --device vmnicN`, replacing `N` with the NIC number. + - question: How can you tag a VMKernel NIC for NVMe/TCP traffic? + answer: | + Use the command: `esxcli network ip interface tag add --interface-name vmkN --tagname NVMeTCP`, replacing `N` + with the VMkernel ID. + - question: Why is enabling NVMe/TCP important for VMware ESXi hosts? + answer: | + Enabling NVMe/TCP reduces latency and improves bandwidth, enabling high-performance storage access. + - context: | + Listing and discovering NVMe-oF adapters and subsystems are the initial steps in configuring an ESXi host for + NVMe-oF. These steps provide details about available adapters and subsystems, which are essential for connecting + the host to the storage infrastructure. + questions_and_answers: + - question: Which command lists the available NVMe-oF adapters on the ESXi host? + answer: | + Use the command: `esxcli nvme adapter list`. + - question: How do you discover NVMe-oF subsystems from an NVMe/TCP adapter? + answer: | + Use the command: `esxcli nvme fabric discover -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 4420`. + - question: What information is needed to discover an NVMe-oF subsystem? + answer: | + You need the NVMe/TCP adapter name, the gateway IP address, and the port number (default is 4420). + - context: | + Connecting to an NVMe-oF gateway subsystem allows VMware ESXi hosts to access storage resources. The connection + is established using the subsystem's NQN and gateway details. Properly connecting ensures the storage can be + utilized by the host. + questions_and_answers: + - question: What command connects to the NVMe-oF gateway subsystem? + answer: | + Use the command: `esxcli nvme connect -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 4420 -s SUBSYSTEM_NQN`. + - question: What is the role of the NQN in connecting to an NVMe-oF gateway? + answer: | + The NQN uniquely identifies the subsystem, enabling the host to establish a connection to the correct resource. + - question: How can you verify if the connection to the NVMe-oF gateway is established? + answer: | + Run the command: `esxcli nvme controller list` and check if the "Connected" column shows "true". + - context: | + After connecting to an NVMe-oF gateway, listing the controllers and namespaces verifies the setup. It also + provides details about available storage resources. The vSphere client can then confirm the proper configuration + and readiness of NVMe/TCP disks. + questions_and_answers: + - question: How do you list NVMe/TCP controllers in VMware ESXi? + answer: | + Use the command: `esxcli nvme controller list`. + - question: Which command lists the namespaces available in an NVMe-oF subsystem? + answer: | + Use the command: `esxcli nvme namespace list`. + - question: How can you verify NVMe/TCP disks in the vSphere client? + answer: | + Navigate to the ESXi host, go to the "Storage" page under the "Devices" tab, and check if the NVMe/TCP disks + are listed. +document_outline: NVMe/TCP Initiator for VMware ESX +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-initiator-esx.md] diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/attribution.txt new file mode 100644 index 000000000..53751ae44 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/attribution.txt @@ -0,0 +1,4 @@ +Title of work: NVMe/TCP Initiator for Linux +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-initiator-linux.md/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/qna.yaml new file mode 100644 index 000000000..544b9fd71 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/Initiators/rhel/qna.yaml @@ -0,0 +1,75 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: NVMe/TCP Initiator for Linux +seed_examples: +- context: | + The NVMe/TCP initiator for Linux requires Kernel 5.0 or later and supported distributions such as RHEL 9.2, + Ubuntu 24.04, or SLES 15 SP3. These prerequisites ensure compatibility with the NVMe-oF protocol for block + storage access. + questions_and_answers: + - question: What is the minimum required kernel version for NVMe/TCP initiator? + answer: | + Kernel version 5.0 or later is required. + - question: Which Linux distributions are supported for NVMe/TCP initiator? + answer: | + RHEL 9.2 or later, Ubuntu 24.04 or later, and SLES 15 SP3 or later. + - question: Why are prerequisites important for the NVMe/TCP initiator? + answer: | + They ensure compatibility with the NVMe-oF protocol for accessing block storage. +- context: | + To set up the NVMe/TCP initiator on Linux, you need to install the `nvme-cli` package, load the NVMe-oF kernel + module, and verify the target's reachability using the `nvme discover` command. + questions_and_answers: + - question: How do you install the NVMe command-line utility on Linux? + answer: | + Use the command `yum install nvme-cli`. + - question: Which kernel module needs to be loaded for NVMe-oF? + answer: | + The `nvme-fabrics` module must be loaded using the `modprobe` command. + - question: How can you verify if the NVMe/TCP target is reachable? + answer: | + Use the `nvme discover -t tcp -a GATEWAY_IP -s 4420` command. +- context: | + After verifying the NVMe/TCP target, you can connect to it using the `nvme connect` command with the target + gateway IP and subsystem NQN. This establishes the initiator-target link. + questions_and_answers: + - question: What command is used to connect to the NVMe/TCP target? + answer: | + Use the command `nvme connect -t tcp -a GATEWAY_IP -n SUBSYSTEM_NQN`. + - question: What does the `-n` option specify in the `nvme connect` command? + answer: | + It specifies the target subsystem NQN for the connectio. + - question: Why is the gateway IP required in the connection command? + answer: | + The gateway IP identifies the NVMe/TCP target to which the initiator connects. +- context: | + After connecting to the NVMe/TCP target, you can list block devices using `nvme list`, create a filesystem on + the NVMe node path, and mount it to a directory for use. + questions_and_answers: + - question: How do you list the NVMe block devices after connection? + answer: | + Use the command `nvme list` to view all NVMe block devices. + - question: How can you create a filesystem on the NVMe node? + answer: | + Use the command `mkfs.ext4 NVME_NODE_PATH` to create an ext4 filesystem. + - question: What is the purpose of mounting the NVMe node path? + answer: | + Mounting makes the NVMe block device accessible as a usable file system. +- context: | + Once the filesystem is mounted, you can interact with the NVMe-oF storage by creating and verifying files in the + mounted directory. For example, create a text file and confirm its contents. + questions_and_answers: + - question: How do you create a directory for mounting NVMe storage? + answer: | + Use the command `mkdir /mnt/nvmeof` to create the mount point. + - question: How can you verify the contents of a file on the NVMe storage? + answer: | + Use the command `cat /mnt/nvmeof/hello.text` to view the file's contents. + - question: What command can you use to create a file on the NVMe-oF storage? + answer: | + Use the command `echo "Hello NVME-oF" > /mnt/nvmeof/hello.text`. +document_outline: NVMe/TCP Initiator for Linux +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-initiator-linux.md] diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/attribution.txt new file mode 100644 index 000000000..f8f7e1b62 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/attribution.txt @@ -0,0 +1,4 @@ +Title of work: NVMe/TCP Initiator for VMware ESX +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-initiator-esx/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/attribution.txt new file mode 100644 index 000000000..685830bc6 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/attribution.txt @@ -0,0 +1,4 @@ +Title of work: Ceph NVMe-oF Overview +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-overview/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/qna.yaml new file mode 100644 index 000000000..079b7e9ba --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_overview/qna.yaml @@ -0,0 +1,77 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: Ceph NVMe-oF Overview +seed_examples: +- context: | + The Ceph NVMe-oF Gateway exports RADOS Block Device (RBD) images as NVMe namespaces, enabling clients without + native Ceph support to access Ceph block storage. It uses the NVMe-oF protocol to send NVMe commands over a + TCP/IP network, integrating Ceph into conventional storage infrastructures. + questions_and_answers: + - question: What does the Ceph NVMe-oF Gateway export? + answer: | + It exports RADOS Block Device (RBD) images as NVMe namespaces. + - question: How does the NVMe-oF protocol benefit clients? + answer: | + It allows clients without native Ceph support to send NVMe commands over a TCP/IP network. + - question: What type of infrastructure can the Ceph NVMe-oF Gateway provision? + answer: | + A fully integrated block-storage infrastructure, similar to a conventional SAN. +- context: | + Each Ceph NVMe-oF gateway consists of an SPDK NVMe-oF target with `bdev_rbd` and a control daemon. This combination + enables efficient communication between RBD images and NVMe-oF clients for storage access. + questions_and_answers: + - question: What are the components of the Ceph NVMe-oF gateway? + answer: | + It includes an SPDK NVMe-oF target with `bdev_rbd` and a control daemon. + - question: What does the `bdev_rbd` module do in the Ceph NVMe-oF gateway? + answer: | + It allows RBD images to be used as NVMe namespaces for storage access. + - question: How does the NVMe-oF gateway facilitate storage access for clients? + answer: | + By enabling NVMe-oF protocol communication with RBD images. +- context: | + The NVMe-oF protocol allows clients to access storage over TCP/IP networks, making Ceph block storage accessible + to various operating systems without requiring native Ceph client support. This expands Ceph's usability across + diverse platforms. + questions_and_answers: + - question: What network protocol does the NVMe-oF gateway use for communication? + answer: | + It uses the NVMe-oF protocol over TCP/IP networks. + - question: How does the NVMe-oF gateway benefit operating systems without native Ceph support? + answer: | + It enables them to access Ceph block storage via the NVMe-oF protocol. + - question: What key advantage does NVMe-oF bring to Ceph storage? + answer: | + It makes Ceph block storage accessible to a wider range of platforms and operating systems. +- context: | + The Ceph NVMe-oF gateway integrates with the public network and optionally the cluster network. It ensures seamless + connectivity between NVMe-oF initiators and the Ceph storage cluster, providing reliable block storage operations. + questions_and_answers: + - question: Which networks does the Ceph NVMe-oF gateway integrate with? + answer: | + It integrates with the public network and optionally the cluster network. + - question: What role does the public network play in the NVMe-oF gateway setup? + answer: | + It facilitates communication between NVMe-oF initiators and the Ceph cluster. + - question: Why is optional integration with the cluster network useful? + answer: | + It provides an additional layer of connectivity for enhanced performance and reliability. +- context: | + Configuring the Ceph NVMe-oF gateway involves setting up requirements such as NVMe-oF target and initiators. The + setup ensures compatibility between the RBD module and various operating systems, enabling efficient storage access. + questions_and_answers: + - question: What are the key components required for configuring the Ceph NVMe-oF gateway? + answer: | + NVMe-oF targets, initiators, and a properly configured RBD module. + - question: How does the gateway ensure compatibility with operating systems? + answer: | + Through its RBD module, which works with various operating systems. + - question: Why is configuring NVMe-oF initiators important? + answer: | + It enables clients to communicate effectively with the NVMe-oF targets. +document_outline: Ceph NVMe-oF Overview +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-overview.md] + diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/attribution.txt new file mode 100644 index 000000000..ab4adfeb7 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/attribution.txt @@ -0,0 +1,4 @@ +Title of work: NVME-oF Gateway Requirements +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-requirements/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/qna.yaml new file mode 100644 index 000000000..c0a4650d4 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_requirements/qna.yaml @@ -0,0 +1,79 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: NVME-oF Gateway Requirements +seed_examples: + - context: | + To implement a highly-available Ceph NVMe/TCP solution, provision at least two NVMe/TCP gateways on different + nodes. This ensures fault tolerance and seamless operation by eliminating single points of failure. Proper + node distribution and redundancy planning are critical for reliability and high availability. + questions_and_answers: + - question: Why are two NVMe/TCP gateways recommended? + answer: | + Two gateways ensure high availability by providing fault tolerance and eliminating single points of failure. + - question: What does deploying gateways on different nodes achieve? + answer: | + It enhances fault tolerance and ensures seamless operation in case one node fails. + - question: What is critical for achieving high availability in a Ceph NVMe/TCP solution? + answer: | + Proper node distribution and redundancy planning are critical for reliability and high availability. + - context: | + NVMe/TCP gateways require a minimum of a single 10Gb Ethernet link in the Ceph public network to operate + efficiently. A high-speed network ensures optimal data transmission and minimal latency, critical for storage + performance. Network provisioning directly impacts the efficiency of the NVMe/TCP solution. + questions_and_answers: + - question: What is the minimum network requirement for an NVMe/TCP gateway? + answer: | + A single 10Gb Ethernet link in the Ceph public network is the minimum requirement. + - question: Why is a 10Gb Ethernet link recommended? + answer: | + It ensures sufficient bandwidth for optimal performance and efficient data transmission. + - question: How does network provisioning affect the NVMe/TCP solution? + answer: | + Proper network provisioning minimizes latency and enhances the solution's overall efficiency. + - context: | + The memory usage on NVMe-oF gateways grows with the number of mapped RBD images. A higher number of mapped + images requires more memory, so it is essential to plan memory allocation accordingly. Adequate memory ensures + reliable gateway performance even under heavy workloads. + questions_and_answers: + - question: What impacts the memory usage of an NVMe-oF gateway? + answer: | + The memory usage scales with the number of mapped RBD images. + - question: How should memory be planned for NVMe-oF gateways? + answer: | + Memory should be allocated based on the expected number of RBD images to be mapped. + - question: Why is sufficient memory important for NVMe-oF gateways? + answer: | + It ensures reliable operation and prevents performance issues under high workloads. + - context: | + Proper hardware selection is vital for NVMe-oF gateways to meet performance and scalability needs. This includes + sufficient memory and network capacity to handle workloads efficiently. Refer to hardware recommendation guides + for detailed specifications to ensure optimal hardware configuration. + questions_and_answers: + - question: What factors should be considered for NVMe-oF gateway hardware? + answer: | + Memory and network capacity should be considered to meet performance and scalability requirements. + - question: How does hardware affect gateway efficiency? + answer: | + Proper hardware ensures that the gateway can handle workloads and data traffic efficiently. + - question: Where can detailed hardware recommendations be found? + answer: | + Detailed specifications can be found in the hardware recommendation guide. + - context: | + Deploying NVMe-oF gateways successfully involves adhering to best practices. These include distributing gateways + across nodes for fault tolerance, ensuring sufficient memory and network bandwidth, and regularly monitoring + performance. Following these practices ensures reliable and high-performance operation. + questions_and_answers: + - question: Why should gateways be distributed across different nodes? + answer: | + Distributing gateways across nodes ensures fault tolerance and high availability. + - question: What are two key requirements for optimal gateway performance? + answer: | + Sufficient memory and high-speed network bandwidth are critical for optimal performance. + - question: How can NVMe-oF gateway performance be maintained over time? + answer: | + Performance can be maintained through regular monitoring and updates. +document_outline: NVME-oF Gateway Requirements +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-requirements.md] diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/attribution.txt b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/attribution.txt new file mode 100644 index 000000000..7350bdef7 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/attribution.txt @@ -0,0 +1,4 @@ +Title of work: Installing and Configuring NVMe-oF Targets +Link to work: https://docs.ceph.com/en/squid/rbd/nvmeof-target-configure/ +License of the work: Apache 2.0 +Creators name: Krishna Ramaswamy \ No newline at end of file diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/qna.yaml new file mode 100644 index 000000000..523f5ebd1 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/nvmeof_target_configure/qna.yaml @@ -0,0 +1,76 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: Installing and Configuring NVMe-oF Targets +seed_examples: +- context: | + Traditionally, block-level access to Ceph was limited to QEMU, librbd, and the Linux kernel client. Starting with + the Ceph Squid release, NVMe/TCP support has been added, enabling wider platform usage and potential new use cases. + questions_and_answers: + - question: What new feature was introduced in the Ceph Squid release? + answer: | + NVMe/TCP support was added, expanding block-level access capabilities. + - question: What were the traditional methods of block-level access to Ceph? + answer: | + QEMU, librbd (common in OpenStack environments), and the Linux kernel client. + - question: How does NVMe/TCP support benefit Ceph storage clusters? + answer: | + It enables wider platform usage and opens up new use cases. +- context: | + The prerequisites for installing and configuring NVMe-oF targets include RHEL/CentOS 8.0 or newer, Linux kernel + v4.16 or newer, a working Ceph Squid or later cluster deployed with cephadm, NVMe-oF gateways, and separate network + subnets for front-end and back-end traffic. + questions_and_answers: + - question: What operating system versions are required for NVMe-oF targets? + answer: | + RHEL/CentOS 8.0 or newer are required. + - question: Why is it necessary to use separate subnets for NVMe-oF traffic? + answer: | + To segregate NVMe-oF front-end traffic from Ceph back-end traffic for better performance. + - question: What is the minimum Linux kernel version required for NVMe-oF targets? + answer: | + Linux kernel v4.16 or newer is required. +- context: | + The Ceph NVMe-oF gateway serves as a translator between Ceph's RBD interface and the NVMe-oF protocol. It can run + as a standalone service or be colocated with other Ceph daemons, such as OSD nodes. Sufficient CPU and memory are + essential when colocating with other services. + questions_and_answers: + - question: What is the function of the Ceph NVMe-oF gateway? + answer: | + It acts as a translator between Ceph's RBD interface and the NVMe-oF protocol. + - question: Can the NVMe-oF gateway run on OSD nodes? + answer: | + Yes, but sufficient CPU and memory must be available. + - question: Why is it important to allocate sufficient CPU and memory for colocated gateways? + answer: | + To ensure reliable performance when colocated with other Ceph daemons. +- context: | + To install the NVMe-oF gateway, create a Ceph pool, enable RBD on the pool, and deploy gateway daemons on specific + nodes. Use cephadm orchestration commands to manage the deployment efficiently. + questions_and_answers: + - question: What command creates a pool for NVMe-oF gateways? + answer: | + The command is `ceph osd pool create NVME-OF_POOL_NAME`. + - question: How is RBD enabled on the NVMe-oF pool? + answer: | + Use the command `rbd pool init NVME-OF_POOL_NAME`. + - question: How are NVMe-oF gateway daemons deployed on nodes? + answer: | + Use the command `ceph orch apply nvmeof NVME-OF_POOL_NAME --placement="host01, host02"`. +- context: | + Configuring NVMe subsystems involves downloading the `nvmeof-cli` container, creating a subsystem, defining + gateway ports, adding host NQNs, and creating NVMe namespaces for block storage operations. + questions_and_answers: + - question: How do you download the `nvmeof-cli` container for configuration? + answer: | + Use the command `podman pull quay.io/ceph/nvmeof-cli:latest`. + - question: What is the purpose of defining a subsystem in NVMe-oF configuration? + answer: | + The subsystem acts as a logical representation for NVMe-oF services, enabling communication with initiators. + - question: How is a new NVMe namespace created? + answer: | + Use the command `podman run -it quay.io/ceph/nvmeof-cli:latest --server-address GATEWAY_IP --server-port GATEWAY_PORT 5500 namespace add`. +document_outline: NVME-oF Gateway Requirements +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-target-configure.md] diff --git a/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/qna.yaml b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/qna.yaml new file mode 100644 index 000000000..77f01e425 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/NVMeOF-gateway/qna.yaml @@ -0,0 +1,85 @@ +version: 3 +created_by: Krishna Ramaswamy +domain: opensource_storage +seed_examples: + - context: | + To configure an ESXi host to use the Ceph NVMe-oF gateway, ensure the prerequisites are met. These include an + ESXi host running VMware vSphere Hypervisor (version 7.0U3 or later), a deployed Ceph NVMe-oF gateway, a Ceph + cluster configured for NVMe-oF, and a defined subsystem within the gateway. Proper setup enables the ESXi host + to achieve high-performance storage with reduced latency and increased bandwidth. + questions_and_answers: + - question: What version of VMware ESXi is required for configuring NVMe-oF? + answer: | + VMware vSphere Hypervisor (ESXi) 7.0U3 or later is required. + - question: What is needed within the Ceph cluster for NVMe-oF functionality? + answer: | + The Ceph cluster must be configured for NVMe-oF. + - question: What does the Ceph NVMe-oF gateway provide? + answer: | + The Ceph NVMe-oF gateway enables ESXi hosts to use high-performance NVMe-oF storage. + - question: What is the purpose of defining a subsystem in the Ceph NVMe-oF gateway? + answer: | + The subsystem acts as a logical representation for NVMe-oF services, enabling communication with initiators. + - context: | + Configuring VMware ESXi for NVMe/TCP involves enabling NVMe/TCP on a NIC and tagging a VMKernel NIC to permit + NVMe/TCP traffic. This setup is critical to ensure the ESXi host can communicate with NVMe-oF gateways. + Enabling NVMe/TCP enhances storage performance by reducing latency and increasing bandwidth. + questions_and_answers: + - question: How do you enable NVMe/TCP on a NIC in VMware ESXi? + answer: | + Use the command: `esxcli nvme fabric enable --protocol TCP --device vmnicN`, replacing `N` with the NIC number. + - question: How can you tag a VMKernel NIC for NVMe/TCP traffic? + answer: | + Use the command: `esxcli network ip interface tag add --interface-name vmkN --tagname NVMeTCP`, replacing `N` + with the VMkernel ID. + - question: Why is enabling NVMe/TCP important for VMware ESXi hosts? + answer: | + Enabling NVMe/TCP reduces latency and improves bandwidth, enabling high-performance storage access. + - context: | + Listing and discovering NVMe-oF adapters and subsystems are the initial steps in configuring an ESXi host for + NVMe-oF. These steps provide details about available adapters and subsystems, which are essential for connecting + the host to the storage infrastructure. + questions_and_answers: + - question: Which command lists the available NVMe-oF adapters on the ESXi host? + answer: | + Use the command: `esxcli nvme adapter list`. + - question: How do you discover NVMe-oF subsystems from an NVMe/TCP adapter? + answer: | + Use the command: `esxcli nvme fabric discover -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 4420`. + - question: What information is needed to discover an NVMe-oF subsystem? + answer: | + You need the NVMe/TCP adapter name, the gateway IP address, and the port number (default is 4420). + - context: | + Connecting to an NVMe-oF gateway subsystem allows VMware ESXi hosts to access storage resources. The connection + is established using the subsystem's NQN and gateway details. Properly connecting ensures the storage can be + utilized by the host. + questions_and_answers: + - question: What command connects to the NVMe-oF gateway subsystem? + answer: | + Use the command: `esxcli nvme connect -a NVME_TCP_ADAPTER -i GATEWAY_IP -p 4420 -s SUBSYSTEM_NQN`. + - question: What is the role of the NQN in connecting to an NVMe-oF gateway? + answer: | + The NQN uniquely identifies the subsystem, enabling the host to establish a connection to the correct resource. + - question: How can you verify if the connection to the NVMe-oF gateway is established? + answer: | + Run the command: `esxcli nvme controller list` and check if the "Connected" column shows "true". + - context: | + After connecting to an NVMe-oF gateway, listing the controllers and namespaces verifies the setup. It also + provides details about available storage resources. The vSphere client can then confirm the proper configuration + and readiness of NVMe/TCP disks. + questions_and_answers: + - question: How do you list NVMe/TCP controllers in VMware ESXi? + answer: | + Use the command: `esxcli nvme controller list`. + - question: Which command lists the namespaces available in an NVMe-oF subsystem? + answer: | + Use the command: `esxcli nvme namespace list`. + - question: How can you verify NVMe/TCP disks in the vSphere client? + answer: | + Navigate to the ESXi host, go to the "Storage" page under the "Devices" tab, and check if the NVMe/TCP disks + are listed. +document_outline: NVMe/TCP Initiator for VMware ESX +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/rbd/nvmeof-initiator-esx.md] diff --git a/knowledge/technical_manual/Ceph/upstream/RADOS/.gitignore b/knowledge/technical_manual/Ceph/upstream/RADOS/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/RBD-Mirror/.gitignore b/knowledge/technical_manual/Ceph/upstream/RBD-Mirror/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/RBD/.gitignore b/knowledge/technical_manual/Ceph/upstream/RBD/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/RGW-Multisite/.gitignore b/knowledge/technical_manual/Ceph/upstream/RGW-Multisite/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/RGW/.gitignore b/knowledge/technical_manual/Ceph/upstream/RGW/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/RGW/keystone/attribution.txt b/knowledge/technical_manual/Ceph/upstream/RGW/keystone/attribution.txt new file mode 100644 index 000000000..3865a1f8b --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/RGW/keystone/attribution.txt @@ -0,0 +1,4 @@ +Title of work: Ceph integration with OpenStack Keystone +Link to work: https://docs.ceph.com/en/latest/radosgw/keystone/ +License of the work: Apache 2.0 +Creators name: Pragadeeswaran Sathyanarayanan diff --git a/knowledge/technical_manual/Ceph/upstream/RGW/keystone/qna.yaml b/knowledge/technical_manual/Ceph/upstream/RGW/keystone/qna.yaml new file mode 100644 index 000000000..1dad52652 --- /dev/null +++ b/knowledge/technical_manual/Ceph/upstream/RGW/keystone/qna.yaml @@ -0,0 +1,90 @@ +--- +version: 1 +created_by: Pragadeeswaran Sathyanarayanan +domain: opensource_storage +document_outline: Teach the Large Language Model about integrating Ceph with OpenStack Keystone service. +document: + repo: git@github.com:pavankumarag/ceph-instructlab-taxonomy-data.git + commit: b60d0d0 + patterns: [upstream/doc/radosgw/keystone*.md] +seed_examples: + - context: | + It is possible to integrate the Ceph Object Gateway with Keystone, the OpenStack identity service. This sets up + the gateway to accept Keystone as the users' authority. A user that Keystone authorizes to access the gateway + will also be automatically created on the Ceph Object Gateway if they didn't exist beforehand. A token that + Keystone validates will be considered valid by the gateway. + questions_and_answers: + - question: | + What is the purpose of integrating Keystone with the Ceph Object Gateway? + answer: | + Integrating Keystone allows the Ceph Object Gateway to accept Keystone as the user authority and validates + tokens authorized by Keystone. + - question: | + What happens when Keystone authorizes a user for the Ceph Object Gateway? + answer: | + The user is automatically created on the Ceph Object Gateway if they didn’t exist beforehand. + + - context: | + The following configuration options are available for Keystone integration: + - rgw keystone api version: Specifies the Keystone API version. + - rgw keystone url: Keystone server URL and admin port. + - rgw keystone admin token: Token for admin operations (use token path for security). + - rgw keystone accepted roles: Defines the user roles accepted. + - rgw keystone token cache size: Determines the number of tokens to cache. + - rgw keystone implicit tenants: Creates private tenants for new users when set to true. + questions_and_answers: + - question: | + What is the purpose of the `rgw keystone api version` option? + answer: | + It specifies the version of the Keystone API to be used for integration. + - question: | + How can you securely configure the Keystone admin token? + answer: | + Use the `rgw keystone admin token path` option to securely provide the admin token. + + - context: | + For Keystone v2.0 API, you can configure a service tenant, user, and password to avoid using the shared + `rgw keystone admin token`. These credentials must have admin privileges. For v3 API, replace + `rgw keystone admin tenant` with `rgw keystone admin domain` and `rgw keystone admin project`. This setup + avoids the need for the shared secret token in production. + questions_and_answers: + - question: | + How can you avoid using the shared Keystone admin token in production environments? + answer: | + By configuring service tenant credentials (user and password) with admin privileges instead of the shared + admin token. + - question: | + What should be configured for Keystone v3 API instead of `rgw keystone admin tenant`? + answer: | + Use `rgw keystone admin domain` and `rgw keystone admin project` for Keystone v3 API. + + - context: | + Keystone must be configured to point to the Ceph Object Gateway as an object-storage endpoint. Use the `openstack + service create` and `openstack endpoint create` commands to define the Swift service and endpoint URLs. If the + `rgw swift account in url` is set to true, include the `/v1/AUTH_%(tenant_id)s` suffix in endpoint URLs. + questions_and_answers: + - question: | + What does the `rgw swift account in url` configuration option do? + answer: | + It requires object-store endpoint URLs to include the suffix `/v1/AUTH_%(tenant_id)s` for tenant access. + - question: | + How do you configure Keystone to recognize the Ceph Object Gateway? + answer: | + Use the `openstack service create` and `openstack endpoint create` commands to configure the Swift service + and endpoints. + + - context: | + Additional features of Keystone integration include S3 API support and service token handling. Enable the S3 API + integration by setting `rgw s3 auth use keystone`. Service tokens allow expired user tokens in requests when + paired with valid service tokens. The expiration cache can be tuned with `rgw keystone expired token cache + expiration`. + questions_and_answers: + - question: | + How can Keystone support authentication for S3 API in Ceph Object Gateway? + answer: | + By setting the `rgw s3 auth use keystone` option in the configuration. + - question: | + What configuration enables service token support in Keystone integration? + answer: | + Enable `rgw keystone service token enabled` and specify accepted roles with `rgw keystone service token + accepted roles`. diff --git a/knowledge/technical_manual/Ceph/upstream/Rook/.gitignore b/knowledge/technical_manual/Ceph/upstream/Rook/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/Security/.gitignore b/knowledge/technical_manual/Ceph/upstream/Security/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/upstream/vSphere-Plugin/.gitignore b/knowledge/technical_manual/Ceph/upstream/vSphere-Plugin/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/knowledge/technical_manual/Ceph/vSphere-Plugin/.gitignore b/knowledge/technical_manual/Ceph/vSphere-Plugin/.gitignore new file mode 100644 index 000000000..e69de29bb