Configuration and Debugging of PCIe Endpoint (EP Mode) on OK3588-C Linux 5.10.66 Buildroot

48 Views, 0 Favorites, 0 Comments

Configuration and Debugging of PCIe Endpoint (EP Mode) on OK3588-C Linux 5.10.66 Buildroot

This article will provide a detailed introduction on how to complete the kernel configuration, U-Boot settings, and performance testing of the EP device on the RK3588 development board, helping developers quickly set up an efficient and stable PCIe Endpoint environment.

The document testing is based on the RK3588 development board.

During the testing, both the RC (Root Complex, host device) and the EP (Endpoint, peripheral device) are Linux devices (RK3588 development boards).
This article focuses on describing the configuration and testing process of the EP device. For detailed configuration information, please refer to the document: Rockchip_Developer_Guide_PCIe_EP_Stardard_Card_CN.pdf.
Software：Linux 5.10.66 4G+32G

Kernel version: OK3588 linux 5.10.66 version kernel (Android12 Linux ForlinxDesktop20.04 R5 release)

Uboot version: OK3588 protection encryption uboot (Android12 Linux ForlinxDesktop20.04 R4 release)

Hardware Connection

For the hardware connection, only one PCIe 3x4 lane male-to-male extension cable is required.

Software Settings

Applying patches

To set the PCIe-EP mode on the 3588, Rockchip officially provides the configuration reference document and source code patches for the PCIe EP mode. The document is located in the docs directory.

Note:

1. The patch file is the basic configuration for the EP card. There is also an update package in the data package. The update package contains update patches based on the base patch, mainly for optimizing the base patch and fixing issues. First, apply the base patch, and then replace the corresponding files in the SDK with the files in the update package directory.

2. Both the patch directory and the source code directory contain necessary modifications. Users can choose to directly apply the patch file or replace the corresponding source code files, either method is acceptable.

Kernel

The kernel patch file is located at:

Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\linux_base\Patch directory\kernel

Modify the kernel according to the patch file.

Uboot

The U-Boot patch file is located at:

\u-boot

After applying the patch, replace the files in the u-boot in the update package.

Rkbin

It is a bit cumbersome to use the patch method to modify the files under rkbin. Directly replace the source code files:

Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\linux_base\Source code directory\rkbin

After replacing the files in the folder, replace the files in the update package to the corresponding positions.

The above configurations are required for both the EP card and the RC card.

General Kernel Configuration

The following configuration items need to be completed during the actual testing process to enable the functions of the EP card. It is recommended to make modifications according to the following configurations. If there are any objections, please refer to the Rockchip official document:

Rockchip_Developer_Guide_PCIe_EP_Stardard_Card_CN.pdf

1. SRNS

In the SRNS mode, the Refclock uses its own internal clock. Set this option so that the RC card and the EP card each use their own 100M clock generators.

uboot

diff --git a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

index 7a24a1d..f1ce41a 100755

--- a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

+++ b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

@@ -81,7 +81,7 @@

#define PCIe_ATU_CPU_ADDR_HIGH 0x18

/* SRNS: Use Separate refclk(internal clock) instead of from RC */

-// #define PCIe_ENABLE_SRNS_PLL_REFCLK

+ #define PCIe_ENABLE_SRNS_PLL_REFCLK

kernel

There is a patch file for the SRNS mode in the kernel configuration:

Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\SRNS

You can directly make corrections under the kernel.

2. Hugepage

Hugepage is a way to allocate memory through the linux HugePageSize. The official Rockchip document recommends using the HugePage method for memory allocation to meet the requirement of large-block memory for high-speed PCIe data transmission.

The kernel configuration file defconfig:

diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig

index 59b4bf7ca..62a77acf7 100644

--- a/arch/arm64/configs/OK3588-Linux_defconfig

+++ b/arch/arm64/configs/OK3588-Linux_defconfig

@@ -734,6 +734,7 @@ CONFIG_VIRTIO_NET=y

CONFIG_NLMON=y

CONFIG_POSIX_MQUEUE=y

CONFIG_PCIe_DW_ROCKCHIP_EP=y

+CONFIG_HUGETLBFS=y

Device tree file：

diff --git a/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts b/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts

index 8ea120342..651f81533 100644

--- a/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts

+++ b/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts

@@ -5,7 +5,8 @@

compatible = "forlinx,ok3588", "rockchip,rk3588";

chosen: chosen {

- bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 root=PARTUUID=614e0000-0000 rw rootwait";

+ // bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 root=PARTUUID=614e0000-0000 rw rootwait";

+ bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 clk_gate.always_on=1pm_domains.always_on=1 root=PARTUUID=614e0000-0000 rw rootwaitdefault_hugepagesz=32M hugepagesz=32M hugepages=16 ";

};

cspmu: cspmu@fd10c000 {

3. BAR

The setup of BAR follows the official rk documentation

uboot

diff --git a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

index 7a24a1d..f1ce41a 100755

--- a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

+++ b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c

@@ -201,14 +201,14 @@ static void PCIe_bar_init(void *dbi_base)

writel(val, dbi_base + 0x7c);

/* Resize BAR0 to support 4M 32bits */

- resbar_base = dbi_base + PCI_RESBAR;

- writel(0x40, resbar_base + 0x4);

+ resbar_base = dbi_base + 0x2e8;

+ writel(0xfffff0, resbar_base + 0x4);

writel(0x2c0, resbar_base + 0x8);

/* BAR2: 64M 64bits */

- writel(0x400, resbar_base + 0x14);

+ writel(0xfffff0, resbar_base + 0x14);

writel(0x6c0, resbar_base + 0x18);

/* BAR4: Fixed for EP wired register, 1M 32bits */

- writel(0x10, resbar_base + 0x24);

+ writel(0xfffff0, resbar_base + 0x24);

writel(0xc0, resbar_base + 0x28);

/* Set flags */

rockchip_PCIe_ep_set_bar_flag(dbi_base, 0, PCI_BASE_ADDRESS_MEM_TYPE_32);

kernel

diff --git a/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c b/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c

index 1addd6c38..55e7756d4 100755

--- a/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c

+++ b/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c

@@ -532,18 +532,18 @@ static void rockchip_PCIe_resize_bar(struct rockchip_PCIe *rockchip)

/* Resize BAR0 4M 32bits, BAR2 64M 64bits-pref, BAR4 1MB 32bits */

bar = BAR_0;

- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x40);

+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);

dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0x2c0);

rockchip_PCIe_ep_set_bar_flag(rockchip, bar, PCI_BASE_ADDRESS_MEM_TYPE_32);

bar = BAR_2;

- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x400);

+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);

dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0x6c0);

rockchip_PCIe_ep_set_bar_flag(rockchip, bar,

PCI_BASE_ADDRESS_MEM_PREFETCH | PCI_BASE_ADDRESS_MEM_TYPE_64);

bar = BAR_4;

- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x10);

+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);

dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0xc0);

rockchip_PCIe_ep_set_bar_flag(rockchip, bar, PCI_BASE_ADDRESS_MEM_TYPE_32)

The above configurations are required for both the EP card and the RC card.

Uboot General Settings

Uboot is configured in defconfig

diff --git a/configs/OK3588-Linux_defconfig b/configs/OK3588-Linux_defconfig

index cde4d18..e6228b8 100644

--- a/configs/OK3588-Linux_defconfig

+++ b/configs/OK3588-Linux_defconfig

+CONFIG_SPL_PCIe_EP_SUPPORT=y

+CONFIG_SPL_RAM_SUPPORT=y

+CONFIG_SPL_RAM_DEVICE=y

The above configurations are required for both the EP card and the RC card.

RC Kernel Configuration

The configuration on the RC side is simple.Only need to add the configuration in defconfig.

diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig

index 3506488cf..26d8f2ae8 100644

--- a/arch/arm64/configs/OK3588-Linux_defconfig

+++ b/arch/arm64/configs/OK3588-Linux_defconfig

@@ -733,3 +733,8 @@ CONFIG_VETH=y

CONFIG_VIRTIO_NET=y

CONFIG_NLMON=y

CONFIG_POSIX_MQUEUE=y

+CONFIG_PCIe_FUNC_RKEP=y

EP Kernel Configuration

The kernel of the EP (Endpoint) side needs to be configured. Note that if the `CONFIG_STRICT_DEVMEM` option is configured in the kernel, it should be removed.

diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig

index 3506488cf..59b4bf7ca 100644

--- a/arch/arm64/configs/OK3588-Linux_defconfig

+++ b/arch/arm64/configs/OK3588-Linux_defconfig

@@ -733,3 +733,8 @@ CONFIG_VETH=y

CONFIG_VIRTIO_NET=y

CONFIG_NLMON=y

CONFIG_POSIX_MQUEUE=y

+CONFIG_PCIe_DW_ROCKCHIP_EP=y

Device tree configuration

diff --git a/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi b/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi

index 6dc06d53e..1848560bb 100644

--- a/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi

+++ b/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi

@@ -46,8 +46,15 @@

#address-cells =;

#size-cells =;

ranges;

- dma_trans: dma-trans@3c000000 {

+ /* dma_trans: dma-trans@3c000000 {

reg =;

+ };*/

+ bar0_region: bar0-region@3c000000 {

+ reg =;

+ };

+ bar2_region: bar2-region@40000000 {

+ reg =;// # Bar大小配置：0x04000000Bytes

};

/* Reserve 256MB memory for hdmirx-controller@fdee0000 */

@@ -954,8 +961,11 @@

};

&PCIe3x4 {

+ compatible = "rockchip,rk3588-PCIe-std-ep";

reset-gpios = <&gpio4 RK_PB6 GPIO_ACTIVE_HIGH>;

- memory-region = <&dma_trans>;

+// memory-region = <&dma_trans>;

+ memory-region = <&bar0_region>, <&bar2_region>;

+ memory-region-names = "bar0", "bar2";

vPCIe3v3-supply = <&vcc3v3_PCIe30>;

status = "okay";

};

Test

1. Verification

After the configuration in the above steps, the following phenomena should be observed on the development board:

RC Side

PCIe-related nodes can be seen under the /dev directory.

Successful registration and usage of the rkep driver can be queried.

EP Side

PCIe dependent nodes can be seen under dev.

2. Demo Test

There are many test demos in the data package provided by Rockchip. Here, the speed-testing demo is selected:

Rockchip_PCIe_EP_Stardard_Card_20231215\examples\PCIe_speed_test

This demo needs to be compiled under Linux. Copy the compiled executable file to the development board for testing.

According to the information in the README.txt file in the examples directory, the resources required by the demo need to be downloaded.

Considering that the dynamic libraries and resources on which the test application depends are relatively large, a resource directory is created to specifically store the test resources.

Download the resource directory to the examples/ directory via Baidu Netdisk.

Link: https://pan.baidu.com/s/1dyXEXMUfWREZii0kHbkOFw?pwd=u1h7

Extraction code: u1h7

Directory Structure

examples/

└── resource

├── PCIe_camera_test

├── PCIe_speed_test

└── PCIe_video_test

After downloading the resources, copy the entire resource package to any directory in the Linux source code. Configure the compilation toolchain in the compilation tool script according to the README.txt document in the speed-testing demo.

The main modifications are as follows:

## build_ep.sh file

#/bin/sh!

EP_CROSS_COMPILE=/home/forlinx/aarch64-buildroot-linux-gnu_sdk-buildroot/bin/aarch64-linux- //Configure the toolchain

# hugepage/rkdrm

EP_MEM_CONFIG=hugepage

make -f ep.mk CROSS_COMPILE=${EP_CROSS_COMPILE} MEM_CONFIG=${EP_MEM_CONFIG} $1

## build_rc.sh file

#/bin/sh!

#RC_ARCH=x86

#RC_CROSS_COMPILE=

RC_ARCH=aarch64

RC_CROSS_COMPILE=/home/forlinx/aarch64-buildroot-linux-gnu_sdk-buildroot/bin/aarch64-linux- //Configure the toolchain

# hugepage/rkep/rkdrm

RC_MEM_CONFIG=hugepage

RC_HOST=${RC_ARCH}-linux

make -f rc.mk CROSS_COMPILE=${RC_CROSS_COMPILE} HOST=${RC_HOST} MEM_CONFIG=${RC_MEM_CONFIG} $1

!!! The path of the above toolchain should be modified according to the actual path in your development environment !!!

Execute the two .sh files respectively in the script directory, and you will get the speed-testing demo executable files for the RC (Root Complex) side and the EP (Endpoint) side.

Copy them to the corresponding boards for speed testing.

The README.txt in the speed-testing demo folder tells you how to use the executable files:

Rockchip_PCIe_EP_Stardard_Card_20231215\examples\PCIe_speed_test

EP side test script running

RC side test script running

Actual speed-testing results on the EP side

Actual speed-testing results on the RC side

Problems

1. The board fails to enter the system

It is found that the board gets stuck during startup. The phenomenon on the terminal interface is as follows:

After waiting for some time, the blue light on the core board flashes abnormally.

This is because the configuration in RK3588MINIALL.ini was modified during the configuration process.

It should be set to all 0s by default.

If the EP card still cannot enter the system after trying the above method, try to connect the RC card.

2. An error occurs when using the speed-testing demo on the RC side

The above error is summarized as a problem that the program encounters when allocating huge-page memory.

At this time, you can check the number of hugepages:

cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

You can see that the number is 0. Set a number for it using echo:

echo 1000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

This can solve the above error.

PS: This configuration also needs to be checked on the EP side.

The above information can also be found in /proc/meminfo:

cat /proc/meminfo | grep Huge

If it is 0, you need to set the number of hugepages.