最近一个项目使用到了瑞芯微的rk3568-evb1-ddr4-v10开发板,用起来挺好,功耗低,发热少,一般情况下也不用装小风扇散热,还有SATA、PCIE接口啥的,来当小NAS、NVR啥的完全足够了。

image-20220527005234171.webp

实际上整个RK3568的软硬件资料在网上都很全的,但是官方默认提供的Linux系统中内核是不支持Docker的(很多第三方开发板显然是支持的),所以需要自己配置一下内核来适配,我顺手记录了一下,少走点弯路,所以有了本文。

在下文描述的时候我将拉取到的源码放到了/home/cyqsd/Project/rk356x目录中,拉取、烧录的步骤我就省略了,因为到处都有资料,就不赘述了,编译的环境是Ubuntu_18.04,编译的目标设备系统是Debian,内核版本是:4.19.172

第一次编译

在拉取代码后最好先进行一次完整的编译,再来修改内核设置,适配Docker。请注意一下实际操作的顺序,可以稍作调整。

1.首先同步一次代码(可省略)。

cyqsd@ubuntu:~/Project/rk356x$ .repo/repo/repo sync -l
Updating files: 100% (2413/2413), done.
Updating files: 100% (16985/16985), done./app/storage_managerUpdating files:   2% (401/16985)
Updating files: 100% (914/914), done.inux/buildrootUpdating files:  47% (432/914)
Updating files: 100% (1072/1072), done.ux/debianUpdating files:  18% (203/1072)
Updating files: 100% (202/202), done.inux/device/rockchipUpdating files:   5% (11/202)
Updating files: 100% (176/176), done.inux/bsp/docsUpdating files:  21% (37/176)
Updating files: 100% (574/574), done.inux/external/CallFunIpcUpdating files:  99% (570/574)
Updating files: 100% (348/348), done.inux/external/alsa-configUpdating files:   9% (32/348)
Updating files: 100% (54/54), done. linux/external/appUpdating files:  24% (13/54)
Updating files: 100% (162/162), done.inux/external/ble_wificonfigUpdating files:   9% (16/162)
Updating files: 100% (968/968), done.inux/bsp/external/broadcom_bsaUpdating files:  50% (485/968)
Updating files: 100% (188/188), done.inux/external/camera_engine_rkaiqUpdating files:   1% (3/188)
Updating files: 100% (69/69), done. linux/external/common_algorithmUpdating files:  82% (57/69)
Updating files: 100% (128/128), done.inux/external/libglCompositorUpdating files:  39% (50/128)
Updating files: 100% (251/251), done.k/rknn-toolkit2Updating files:  80% (203/251)
Updating files: 100% (774/774), done.inux/external/rknn_demoUpdating files:   6% (52/774)
Updating files: 100% (933/933), done.inux/external/rknpuUpdating files:   6% (60/933)
Updating files: 100% (304/304), done.inux/external/rkwifibtUpdating files:  76% (233/304)
Updating files: 100% (447/447), done.inux/rockitUpdating files:   1% (6/447)
Updating files: 100% (11764/11764), done./external/softapDemoUpdating files:  26% (3087/11764)
Updating files: 100% (72292/72292), done./external/wifiAutoSetupUpdating files:   8% (6358/72292)
Updating files: 100% (17864/17864), done.rnelUpdating files:  29% (5307/17864)
Checking out projects:  85% (76/89) rk/prebuilts/gcc-buildroot-9.3.0-2020.03-x86_64_aarch64-rockchip-linuUpdating files: 100% (7165/7165), done.
Updating files: 100% (200/200), done.k/rkbinUpdating files:  53% (107/200)
Updating files: 100% (13390/13390), done./toolsUpdating files:  83% (11183/13390)
Checking out projects: 100% (89/89), done.
repo sync has finished successfully.

2.切换要编译的设备,比如我手里的设备就是rk3568-evb1-ddr4-v10

cyqsd@ubuntu:~/Project/rk356x$ ./build.sh device/rockchip/rk356x/BoardConfig-rk3568-evb1-ddr4-v10.mk

You're building on Linux
Lunch menu...pick a combo:

0. default BoardConfig.mk
1. BoardConfig-rk3566-evb2-lp4x-v10-32bit.mk
2. BoardConfig-rk3566-evb2-lp4x-v10.mk
3. BoardConfig-rk3568-evb1-ddr4-v10-32bit.mk
4. BoardConfig-rk3568-evb1-ddr4-v10-spi-nor-64M.mk
5. BoardConfig-rk3568-evb1-ddr4-v10.mk
6. BoardConfig-rk3568-nvr-spi-nand.mk
7. BoardConfig-rk3568-nvr.mk
8. BoardConfig-rk3568-uvc-evb1-ddr4-v10.mk
9. BoardConfig.mk
Which would you like? [0]: 5
switching to board: /home/cyqsd/Project/rk356x/device/rockchip/rk356x/BoardConfig-rk3568-evb1-ddr4-v10.mk
processing option: device/rockchip/rk356x/BoardConfig-rk3568-evb1-ddr4-v10.mk
switching to board: /home/cyqsd/Project/rk356x/device/rockchip/rk356x/BoardConfig-rk3568-evb1-ddr4-v10.mk

3.环境变量中配置默认编译的文件系统

默认是buildroot,所以我们设置一下。

export RK_ROOTFS_SYSTEM=debian
# 也可以后面进入内核目录,手动使用下面的命令编译
RELEASE=buster TARGET=desktop ARCH=arm64 ./mk-base-debian.sh

4.电压域确认

进行内核编译的时候会提醒电压域配置确认,一般第一次编译内核都会弹出选项来选择,要是没有弹出,或者想要手动检查一下,可以通过/home/cyqsd/Project/rk356x/kernel/arch/arm64/boot/dts/rockchip/rk3568-evb1-ddr4-v10-linux.dts 的节点 [pmu_io_domains]来查看,具体如下:

 PLEASE CHECK BOARD GPIO POWER DOMAIN CONFIGURATION !!!!!
 <<< ESPECIALLY Wi-Fi/Flash/Ethernet IO power domain >>> !!!!!
 Check Node [pmu_io_domains] in the file: /home/cyqsd/Project/rk356x/kernel/arch/arm64/boot/dts/rockchip/rk3568-evb1-ddr4-v10-linux.dts

 请再次确认板级的电源域配置!!!!!!
 <<< 特别是Wi-Fi,FLASH,以太网这几路IO电源的配置 >>> !!!!!
 检查内核文件 /home/cyqsd/Project/rk356x/kernel/arch/arm64/boot/dts/rockchip/rk3568-evb1-ddr4-v10-linux.dts 的节点 [pmu_io_domains]

我的配置文件是rk3568-evb.dtsi,与rk3568-evb1-ddr4-v10-linux.dts同目录下,找到该文件,查看内容,我修改为了下面的配置:

 /*
  * There are 10 independent IO domains in RK3566/RK3568, including PMUIO[0:2] and VCCIO[1:7].
  * 1/ PMUIO0 and PMUIO1 are fixed-level power domains which cannot be configured;
  * 2/ PMUIO2 and VCCIO1,VCCIO[3:7] domains require that their hardware power supply voltages
  *    must be consistent with the software configuration correspondingly
  *    a/ When the hardware IO level is connected to 1.8V, the software voltage configuration
  *       should also be configured to 1.8V accordingly;
  *    b/ When the hardware IO level is connected to 3.3V, the software voltage configuration
  *       should also be configured to 3.3V accordingly;
  * 3/ VCCIO2 voltage control selection (0xFDC20140)
  *    BIT[0]: 0x0: from GPIO_0A7 (default)
  *    BIT[0]: 0x1: from GRF
  *    Default is determined by Pin FLASH_VOL_SEL/GPIO0_A7:
  *    L:VCCIO2 must supply 3.3V
  *    H:VCCIO2 must supply 1.8V
  */
&pmu_io_domains {
    status = "okay";
    pmuio2-supply = <&vcc3v3_pmu>;
    vccio1-supply = <&vccio_acodec>;
    vccio3-supply = <&vccio_sd>;
    vccio4-supply = <&vcc_1v8>;
    vccio5-supply = <&vcc_3v3>;
    vccio6-supply = <&vcc_1v8>;
    vccio7-supply = <&vcc_3v3>;
};

原厂提供了Rockchip_RK356X_Introduction_IO_Power_Domains_Configuration_CN.pdf(RK3566 RK3568 IO 电源域配置指南),根据自己手里的硬件,我就改成了:VCCIO4和VCCIO6配置成1V8,其余的配置成3V3

使用编译内核命令时,要注意内核编译时所采用的具体是那个配置文件和所处的平台,如下面所示是rockchip_linux_defconfig配置文件,arm64平台。

cyqsd@ubuntu:~/Project/rk356x$ ./build.sh kernel
processing option: kernel
============Start building kernel============
TARGET_ARCH          =arm64
TARGET_KERNEL_CONFIG =rockchip_linux_defconfig
TARGET_KERNEL_DTS    =rk3568-evb1-ddr4-v10-linux
TARGET_KERNEL_CONFIG_FRAGMENT =
==========================================

然后使用./build.sh对所有模块(u-Boot,kernel,rootfs,recovery)进行编译就行了,具体首次编译中遇到的问题请参考其他教程。

内核配置修改

完成了上一个小节的首次编译后,所有模块都应该是可以正常编译生成了,烧录到设备也是可以正常运行的,这时就可以继续看这一小节。Docker不能正常运行的原因是内核问题,这就需要改变原有内核配置,来适配运行,好在moby(一个组件容器框架)提供了Docker的环境检查工具,可以从Github上单独下载这个检测脚本文件。https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh,通过这个工具就可以知道当前内核的配置文件是否支持Docker正常运行。

此处参考了:Verify your Linux Kernel for Container Compatibility · Docker Pirates ARMed with explosive stuff (hypriot.com)

cd /home/cyqsd/Project/rk356x/kernel
chmod +x check-config.sh
./check-config.sh .config

kernel目录下的.config文件是自动生成的,每次编译这个文件都将被覆盖

实际的配置文件在上一小节首次编译的时候已经提到了,是rockchip_linux_defconfig,在/home/cyqsd/Project/rk356x/kernel/arch/arm64/configs目录下,注意是arm64架构,不要选择错了,不同架构下可能存在相同名字的配置文件。

我比较推荐在内核根目录下使用check-config.sh,检查每次修改好的文件后,拷贝回这个目录

一般情况下输出的结果会是下面这样,有许多项目都是检测不通过的,按照下面的内容进行修改:

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: missing
- CONFIG_KEYS: enabled
- CONFIG_VETH: missing
- CONFIG_BRIDGE: missing
- CONFIG_BRIDGE_NETFILTER: missing
- CONFIG_IP_NF_FILTER: missing
- CONFIG_IP_NF_TARGET_MASQUERADE: missing
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: missing
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: missing
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_NETFILTER_XT_MARK: missing
- CONFIG_IP_NF_NAT: missing
- CONFIG_NF_NAT: missing
- CONFIG_POSIX_MQUEUE: missing
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: missing
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: missing
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: missing
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: missing
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: present
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

截图是这样,很明显, 修改好的是绿色,没改好的是红色。

image-20220422125925827.webp

杂项配置

除了Optional Features中的选项是可选开启的,其他项目后是missing的都需要手动开启一下,因为配置项目比较多,我就没一一截图了。推荐两种途径来找配置文件中的配置项和图形化配置中的对应关系。推荐使用同型号已经配置过的配置文件(文末我提供了我修改后的配置文件)来跳过这一步骤,直接就可以进入后面两个小节,如果你觉得还是手动操作更靠谱,那可以接着看。

  1. 查看配置选项的翻译文档来对应一下,Linux-4.4-x86_64 内核配置选项简介 [金步国] (jinbuguo.com)

比如,要开启CONFIG_BRIDGE,那就看一下文档里面怎么写的:

image-20220526173442537.webp

就把父选项开启就行了,颜色和缩进代表了父子选项的层级关系。但是因为各种原因限制,上面肯定是没有所有选项的,所以用的时候还需注意。

  1. 除了看文档还可以使用kernelconfig.io提供的在线查询工具,比如要开启CONFIG_IP_VS

    IP virtual server support - CONFIG_IP_VS - ip_vs.ko - kernelconfig.io

image-20220526174326227.webp

主线版本和平台架构也记得选择一下。

image-20220526174345690.webp

3.在配置界面直接输入/来搜索配置项。

除了上面的方法,剩下的就只能自己捣鼓了。

下面有些零散的步骤截图,仅供参考,涉及到先后顺序的可能比较乱,可以在网上额外查证对照一下:

先打开IP virtual server support,再回到选项里面打开后面的项目:

IP virtual server support - CONFIG_IP_VS - ip_vs.ko - kernelconfig.io

image-20220527013156473.webp

image-20220516020313189.webp

image-20220516020718801.webp

image-20220516023004358.webp

先开启 802.1Q/802.1ad VLAN Support,然后打开上面的VLAN filtering

image-20220516141323292.webp

CONFIG_CGROUP_BPF

bpf_prog_attach-and-bpf_prog_detachFerrisEllis.com

image-20220527013041689.webp

image-20220527013109511.webp

image-20220527013118677.webp

image-20220527013125903.webp

image-20220527013147647.webp

image-20220527013156473.webp

image-20220527013205763.webp

image-20220527013218223.webp

image-20220527013226236.webp

image-20220527013235872.webp

加入aufs4支持

因为aufs的配置和依赖没有并入到Linux内核的主线里面(我看有人说是Linus觉得aufs的代码质量太差了?),源码可以从官网拉取:aufs.sourceforge.net

有如下内容供选择,我们使用aufs4-standalone

image-20220425154141388.webp

这次使用的版本的是4.19.232,所以拉取的就是这个版本,在/home/cyqsd/Project目录执行拉取命令。此处参考了aufs的官方文档,GitHub - lfsid/kernel_configRk3568 buildroot Linux4.19 Docker 支持_江南安抚使2022的博客

cyqsd@ubuntu:~/Project$ git clone https://github.com/sfjro/aufs4-standalone.git
cyqsd@ubuntu:~/aufs4/aufs4-standalone$ git checkout origin/aufs4.19
HEAD is now at 0f5cf975 4.19 20200622

进入目录,此时目录内的文件是这样:

cyqsd@ubuntu:~/Project/aufs4-standalone$ ls
aufs4-base.patch    aufs4-loopback.patch  aufs4-standalone.patch  COPYING        fs       lockdep-debug.patch  proc_mounts.patch  tmpfs-idr.patch
aufs4-kbuild.patch  aufs4-mmap.patch      config.mk               Documentation  include  Makefile             README             vfs-ino.patch

复制Documentation,fs,两个文件夹到内核目录/home/cyqsd/Project/rk356x/kernel

cyqsd@ubuntu:~/Project/aufs4-standalone$ cp -R {Documentation,fs} /home/cyqsd/Project/rk356x/kernel

复制下面的文件到指定位置:

cyqsd@ubuntu:~/Project/aufs4-standalone$ cp include/uapi/linux/aufs_type.h /home/cyqsd/Project/rk356x/kernel/include/uapi/linux/
cyqsd@ubuntu:~/Project/aufs4-standalone$ cp aufs4-base.patch /home/cyqsd/Project/rk356x/kernel/
cyqsd@ubuntu:~/Project/aufs4-standalone$ cp aufs4-kbuild.patch /home/cyqsd/Project/rk356x/kernel/
cyqsd@ubuntu:~/Project/aufs4-standalone$ cp aufs4-mmap.patch /home/cyqsd/Project/rk356x/kernel/
cyqsd@ubuntu:~/Project/aufs4-standalone$ cp aufs4-standalone.patch /home/cyqsd/Project/rk356x/kernel/

进入cd /home/cyqsd/Project/rk356x/kernel目录,执行下面的命令:

添加日志:

cyqsd@ubuntu:~/Project/rk356x/kernel$ patch -p1 < aufs4-kbuild.patch            
patching file fs/Kconfig
Hunk #1 succeeded at 259 (offset 4 lines).
patching file fs/Makefile
Hunk #1 succeeded at 132 (offset 4 lines).
cyqsd@ubuntu:~/Project/rk356x/kernel$ patch -p1 < aufs4-base.patch
patching file MAINTAINERS
patching file drivers/block/loop.c
Hunk #1 succeeded at 763 (offset 24 lines).
patching file fs/dcache.c
Hunk #1 succeeded at 1223 (offset -15 lines).
patching file fs/fcntl.c
patching file fs/inode.c
Hunk #1 succeeded at 1666 (offset 9 lines).
patching file fs/namespace.c
Hunk #1 succeeded at 772 (offset 2 lines).
patching file fs/read_write.c
Hunk #1 succeeded at 490 (offset 1 line).
patching file fs/splice.c
patching file fs/sync.c
patching file include/linux/fs.h
Hunk #1 succeeded at 1333 (offset 47 lines).
Hunk #2 succeeded at 1828 (offset 80 lines).
Hunk #3 succeeded at 1912 (offset 92 lines).
Hunk #4 succeeded at 2368 (offset 117 lines).
Hunk #5 succeeded at 2658 (offset 119 lines).
patching file include/linux/lockdep.h
patching file include/linux/mnt_namespace.h
patching file include/linux/splice.h
patching file kernel/locking/lockdep.c
cyqsd@ubuntu:~/Project/rk356x/kernel$ patch -p1 < aufs4-mmap.patch
patching file fs/proc/base.c
Hunk #1 succeeded at 2036 with fuzz 1 (offset 20 lines).
patching file fs/proc/nommu.c
patching file fs/proc/task_mmu.c
Hunk #1 succeeded at 359 (offset 54 lines).
Hunk #2 succeeded at 1833 (offset 103 lines).
patching file fs/proc/task_nommu.c
patching file include/linux/mm.h
Hunk #1 succeeded at 1509 (offset 69 lines).
patching file include/linux/mm_types.h
Hunk #1 succeeded at 245 (offset 6 lines).
Hunk #2 succeeded at 327 (offset 13 lines).
patching file kernel/fork.c
Hunk #1 succeeded at 517 (offset 12 lines).
patching file mm/Makefile
Hunk #1 succeeded at 42 (offset 3 lines).
patching file mm/filemap.c
Hunk #1 succeeded at 2798 (offset 98 lines).
patching file mm/mmap.c
Hunk #1 succeeded at 175 (offset -5 lines).
Hunk #2 succeeded at 900 (offset -5 lines).
Hunk #3 succeeded at 1829 (offset 8 lines).
Hunk #4 succeeded at 2655 (offset 14 lines).
Hunk #5 succeeded at 2674 (offset 14 lines).
Hunk #6 succeeded at 2837 (offset 15 lines).
Hunk #7 succeeded at 2912 (offset 15 lines).
Hunk #8 succeeded at 3239 (offset 16 lines).
patching file mm/nommu.c
Hunk #1 succeeded at 629 (offset 4 lines).
Hunk #2 succeeded at 767 (offset 4 lines).
Hunk #3 succeeded at 1290 (offset 4 lines).
Hunk #4 succeeded at 1365 (offset 4 lines).
patching file mm/prfile.c
cyqsd@ubuntu:~/Project/rk356x/kernel$ patch -p1 < aufs4-standalone.patch
patching file fs/dcache.c
Hunk #1 succeeded at 1328 (offset -15 lines).
Hunk #2 succeeded at 2820 (offset -18 lines).
patching file fs/exec.c
patching file fs/fcntl.c
patching file fs/file_table.c
Hunk #3 succeeded at 372 (offset 5 lines).
patching file fs/inode.c
Hunk #1 succeeded at 1675 (offset 9 lines).
patching file fs/namespace.c
Hunk #1 succeeded at 438 (offset 1 line).
Hunk #2 succeeded at 778 (offset 2 lines).
Hunk #3 succeeded at 1900 (offset 66 lines).
patching file fs/notify/group.c
patching file fs/notify/mark.c
Hunk #1 succeeded at 285 (offset 22 lines).
Hunk #2 succeeded at 440 (offset 22 lines).
Hunk #3 succeeded at 656 (offset 22 lines).
Hunk #4 succeeded at 780 (offset 23 lines).
patching file fs/open.c
Hunk #1 succeeded at 69 with fuzz 2 (offset 5 lines).
patching file fs/read_write.c
Reversed (or previously applied) patch detected!  Assume -R? [n] y
Hunk #2 FAILED at 499.
Hunk #3 FAILED at 511.
Hunk #4 succeeded at 579 (offset -2 lines).
2 out of 4 hunks FAILED -- saving rejects to file fs/read_write.c.rej
patching file fs/splice.c
patching file fs/sync.c
patching file fs/xattr.c
Hunk #1 succeeded at 332 (offset 37 lines).
patching file kernel/locking/lockdep.c
patching file kernel/task_work.c
patching file security/device_cgroup.c
patching file security/security.c
Hunk #1 succeeded at 538 (offset -4 lines).
Hunk #2 succeeded at 555 (offset -4 lines).
Hunk #3 succeeded at 564 (offset -4 lines).
Hunk #4 succeeded at 592 (offset -4 lines).
Hunk #5 succeeded at 600 (offset -4 lines).
Hunk #6 succeeded at 608 (offset -4 lines).
Hunk #7 succeeded at 709 (offset -4 lines).
Hunk #8 succeeded at 881 (offset -4 lines).
Hunk #9 succeeded at 941 (offset -4 lines).

到此步时,aufs已经加入到了内核配置中,但是还得要手动开启支持一下。

依旧执行命令进入配置make ARCH=arm64 menuconfig

image-20220526170858106.webp

image-20220526174912554.webp

记得配置文件要拷贝回去,不然会自动覆盖。

zfs command

安装一下sudo apt-get install zfs-fuseHow to Fix the “zpool command not found” Error in Debian (linuxhint.com)

image-20220526171332481.webp

最终完成后的配置检查应该是这样:

cyqsd@ubuntu:~/Project/rk356x/kernel$ ./check-config.sh .config
info: reading kernel config from .config ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
- CONFIG_NETFILTER_XT_MARK: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: enabled
    - CONFIG_NF_CONNTRACK_TFTP: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: present
    - zfs command: available
    - zpool command: available

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

截图:

image-20220527004151841.webp

image-20220527004227074.webp

烧写并运行

1.按照正常步骤烧录系统,安装Docker即可。

image-20220527011926319.webp

可以使用一键安装命令curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

也可以手动下载安装:

从Docker的下载站点下载3个需要的deb包:Index of linux/debian/dists/buster/pool/stable/arm64/ (docker.com)

containerd.io_1.6.4-1_arm64.deb  
docker-ce-cli_20.10.9~3-0~debian-buster_arm64.deb  
docker-ce_20.10.9~3-0~debian-buster_arm64.deb

使用dpkg -i命令手动安装也行,注意设备架构是arm64,不是armhf

2.安装Docker,并运行后会提示错误,使用journalctl -xe查看详细错误信息,大概内容如下(篇幅有限只截取了一部分,大概相同即可):

May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.790201082Z" level=info msg="Starting up"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.797091708Z" level=info msg="parsed scheme: \"unix\"" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.797280708Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.797476125Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" m
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.797553708Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.802857375Z" level=info msg="parsed scheme: \"unix\"" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.803763000Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.803952001Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" m
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.804097251Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.871118174Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.882983467Z" level=warning msg="Your kernel does not support swap memory limit"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.883135425Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.883192008Z" level=warning msg="Unable to find blkio cgroup in mounts"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.883241300Z" level=warning msg="Unable to find pids cgroup in mounts"
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.884278759Z" level=info msg="Loading containers: start."
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.898813385Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.911342220Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `iptables/1.8.2 Failed to initialize nft: Protocol not
May 16 14:46:44 linaro-alip dockerd[870]: time="2022-05-16T14:46:44.948387973Z" level=warning msg="Failed to read iptables version: exit status 1"
May 16 14:46:45 linaro-alip dockerd[870]: time="2022-05-16T14:46:45.279166590Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
May 16 14:46:45 linaro-alip dockerd[870]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -
May 16 14:46:45 linaro-alip dockerd[870]:  (exit status 1)
May 16 14:46:45 linaro-alip systemd[1]: docker.service: Failed with result 'exit-code'.

警告现在就先暂时不管,主要看fail,注意到提示iptables报错导致无法完成网络设置(篇幅有限只截取了一部分,大概相同即可):

failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -

使用raspberry pi - iptables/1.8.2 Failed to initialize nft: Protocol not supported - Super User,提供的解决办法切换为旧版本:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy

如果进行到此步骤,你的设备也报错了,但是报错的内容不是这个,而是文件系统或者其他内容,说明前面几步中的内核配置有配置错误或者有漏掉的项目,需要返回去检查。

修改后,重启。

systemctl daemon-reload
systemctl restart docker
systemctl status docker

此时已经可以正常启动了。

image-20220527001853926.webp

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["http://hub-mirror.c.163.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

本文到这里就结束了,后面还要不要添加国内镜像源,就看自己了,操作和平时相同。有了Docker支持以后,在RK3568上捣鼓一些程序就方便多了,最近因为种种原因写博客的时间变少了,我有时候在想,是不是写博客花的时间比做Vlog更花时间来着,但是本来写博客都懒,做Vlog要注重的细节更多,那就更懒了。如果本文对你有所帮助,欢迎在文后评论支持一下,反正也不花钱。。。

文章目录