Prerequisites:

1. RoCE required Compliant Ethernet. Currently, I am using Mellanox ConnectX-6 Cards
2. RoCE required a Compliant Switch. I used Mellanox 100G Switch.

The Difference between Traditional Ethernet Communication and RoCE can be explained very clearly in the diagram taken by Huawei’s Basic Knowledge and Differences of RoCE, IB, and TCP Networks

Some Key Pointers on the difference between TCP/IP and RDMA

1. The Traditional TCP/IP network communication uses the Kernel to send messages which have high data movement and data replication overhead.
2. RDMA can bypass the kernel and access the memory directly which allows low-latency network communication.

There are 3 types of RDMA network technologies is so neatly presented in Basic Knowledge and Differences of RoCE, IB, and TCP Networks

References:

1. Basic Knowledge and Differences of RoCE, IB, and TCP Networks