- Compass-ci - mailweb.openeuler.org

test mail to lsd
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

test mail to wl
by Luan Shengde 07 Feb '22

07 Feb '22

test msgg

1 0

test mail
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

reproducible-build TODO list
by Lu Weitao 07 Feb '22

07 Feb '22

## TODO list - TODO1.1: script support set env for build env --- compass-ci jihui - TODO1.2: script record buildinfo file in testbox --- IT/compass-ci - based on debian buildinfo - TODO1.3: 输出用户指定的息into buildinfo? new lib/buildinfo.sh --- IT func1() func2() func3() save to $result_root/buildinfo.yaml modify rpmbuild script to call the above funcs. --- compass-ci wangyong - TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao - TODO2: 比较构建结果是否相同 --- compass-ci weitao compass ci reuse data API es-find compare in client side 格式和进一步优化: IT client: ask use python? - TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT - TODO4 toolset --- IT/compass-ci option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong --------- Thanks, Lu Weitao

1 0

test mail 0207
by Lu Weitao 07 Feb '22

07 Feb '22

test msg 0207

1 0

Re: test mail to cxl
by Wu Fengguang 29 Jan '22

29 Jan '22

Nice to meet you, Shengde! :)

1 0

Re: test mail to lsd
by Wu Fengguang 29 Jan '22

29 Jan '22

Weitao, Got it. :) Thanks, Fengguang

1 0

test mail to lsd
by Lu Weitao 29 Jan '22

29 Jan '22

test mail bbb

1 0

test mail to cxl
by Luan Shengde 29 Jan '22

29 Jan '22

test msg aaa

1 0

test mail to cxl
by Luan Shengde 29 Jan '22

29 Jan '22

test msg

1 0

test mail
by Luan Shengde 29 Jan '22

29 Jan '22

test msg aaa

1 0

test mail 0129-e
by luanshd0525 29 Jan '22

29 Jan '22

test msg

1 0

test mail 0129-a
by Luan Shengde 29 Jan '22

29 Jan '22

test msg

1 0

test mail 0129
by Luan Shengde 29 Jan '22

29 Jan '22

test msg 0129

1 0

test mail for subscription
by luanshd0525 28 Jan '22

28 Jan '22

test msg for subscription

1 0

test mail for mailing list
by 暗夜 28 Jan '22

28 Jan '22

test msg for mailing list

1 0

test for mailing list
by luanshd 28 Jan '22

28 Jan '22

test msg for mailing list

1 0

[PATCH compass-ci] doc: add doc of rootfs
by Wu Fengguang 05 Dec '21

05 Dec '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/help/rootfs/README.md | 146 ++++++++++++++++++ .../compass-ci-use-rootfs/cifs-nfs-hw.md | 58 +++++++ .../compass-ci-use-rootfs/cifs-nfs-qemu.md | 108 +++++++++++++ .../rootfs/compass-ci-use-rootfs/cifs-nfs.md | 7 + .../rootfs/compass-ci-use-rootfs/container.md | 86 +++++++++++ .../rootfs/compass-ci-use-rootfs/initramfs.md | 80 ++++++++++ .../rootfs/compass-ci-use-rootfs/local.md | 80 ++++++++++ doc/help/rootfs/how-to-get-rootfs/README.md | 35 +++++ .../auto-generate-cifs-nfs.md | 97 ++++++++++++ .../manual-generate-cifs-nfs-local.md | 58 +++++++ .../manual-generate-initramfs.md | 39 +++++ 11 files changed, 794 insertions(+) create mode 100644 doc/help/rootfs/README.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/container.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/initramfs.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/local.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/README.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md diff --git a/doc/help/rootfs/README.md b/doc/help/rootfs/README.md new file mode 100644 index 000000000000..eac1ea01ee58 --- /dev/null +++ b/doc/help/rootfs/README.md @@ -0,0 +1,146 @@ +[TOC] + +1 概念解释 +========== + +1.1 Compass-CI的rootfs是什么？ +------------------------------ + +Compass-CI的rootfs是指存放在服务器端的，有版本的，可以为执行机在执行指定os,os_arch,os_version的job时提供相同的，干净的执行环境的docker镜像/文件/目录。 + +Compass-CI当前可供测试的执行机类型有：docker, 虚拟机, 物理机。 +- 对docker而言，rootfs就是docker registry里面的指定版本的docker image。 +- 对虚拟机和物理机而言，rootfs就是在指定版本的iso安装完毕后，将其所有文件（运行时文件除外）拷贝出来，归档到/srv/os或/srv/initrd下版本对应的目录/文件。 + +1.2 os_mount是什么？ +-------------------- + +Compass-CI的job执行是通过submit命令来触发的，os_mount是submit时候的一个参数，用来指定本次job执行所需os是通过何种方式提供的。 + +|----------------------------------|-------------------------------------------------------------------------------| +| Compass-CI目前支持的os_mount类型 | 简单说明 | +|----------------------------------|-------------------------------------------------------------------------------| +| container | 对应执行环境为docker | +| initramfs | 内存文件系统，系统所需文件一次性全部加载到内存中 | +| nfs | 内存文件系统，系统所需文件是通过nfs协议mount的，使用到多少文件，加载多少文件 | +| cifs | 内存文件系统，系统所需文件是通过cifs协议mount的，使用到多少文件，加载多少文件 | +| local | 硬盘文件系统，系统所需文件是位于一个指定名称格式的逻辑卷中 | +|----------------------------------|-------------------------------------------------------------------------------| + +1.3 rootfs与os_mount的关系是什么？ +---------------------------------- + +job通过submit命令来提交到compass-ci调度器； + +submit job时候需要指定os_mount，若不指定，默认值如下： +- 若指定本次执行机类型为docker，则os_mount默认为container； +- 若指定本次执行机类型为虚拟机/物理机，则os_mount默认为initramfs； + +rootfs需要在sbumit job之前制作好，否则submit job会得到错误返回。 + +|--------------|-----------------------------------------------------------------| +| os_mount类型 | 对应的rootfs存储位置 | +|--------------|-----------------------------------------------------------------| +| container | Compass-CI集群的docker registry。image格式：${os}:${os_version} | +| initramfs | /srv/initrd/osimage/${os}/${os_arch}/${os_version}/current | +| cifs/nfs | /srv/os/${os}/${os_arch}/${os_version} | +| local | /srv/os/${os}/${os_arch}/${os_version}-iso | +|--------------|-----------------------------------------------------------------| + +注： +local与cifs/nfs现阶段的区别如下： +- cifs/nfs所需的rootfs目前是通过安装iso到虚拟机得到的； +- local所需的rootfs目前是通过安装iso到物理机得到的； +- 理论上： + - 就与手动安装iso到物理机的系统差异而言，安装iso到物理机得到的rootfs，差异是小于安装iso到虚拟机得到的rootfs的； + - 所以，local能用的rootfs，是可以直接用于cifs/nfs的，而且以后如果安装iso到物理机得到rootfs的方式自动化完毕，那么在磁盘的/srv/os下，是只需要保留安装iso到物理机得到的rootfs的，现阶段local需要的-iso后缀可以去掉，cifs/nfs/local，都使用同一套rootfs。 + + +2 使用指南 +========== + +[提交job](../../job/submit/submit-job.zh.md) +[os_mount](../../job/fields/os_mount.md) +如何查看当前compass-ci集群支持的rootfs都有哪些？ # 待补充 + + +3 管理员手册 +============ + +3.1 Compass-CI的rootfs是怎么来的？（原理） +------------------------------------------ + +### 3.1.1 container + +基于指定版本的基础镜像基础，制作docker image，并上传到docker registry中 + +### 3.1.2 nfs/cifs + +- 安装iso到虚拟机的硬盘（qcow2）上； +- 安装完成后，使用硬盘系统再进行一次启动/关机操作； +- 将qcow2中的rootfs拷贝出来； # 可以使用compass-ci/container/qcow2rootfs容器进行拷贝 +- 对拷贝出来的rootfs进行一些post配置，即可用于compass-ci。 + +> 详细post配置见之后的rootfs如何制作部分 + +### 3.1.3 local + +- 安装iso到物理机的硬盘A上； +- 安装完成后，使用硬盘A上的系统再进行一次启动/关机操作； +- 使用非安装iso的硬盘A启动系统（可以使用内存文件系统，也可以使用其他硬盘上的系统），将硬盘A中的rootfs拷贝出来； +- 对拷贝出来的rootfs进行一些post配置，即可用于compass-ci。 + +> 详细post配置见之后的rootfs如何制作部分 + +### 3.1.4 initramfs + +iniramfs所需的rootfs是一个包含了所有系统必须文件的cgz文件。 +所以我们使用cpio将cifs/nfs/local制作出来的rootfs打包制作成cgz文件即可。 +假设此rootfs位于B目录，将B目录整体打包。 + +> 详细打包命令见之后的rootfs如何制作部分 + + +3.2 我已有Compass-CI集群，如何获得rootfs？ +------------------------------------------ + +[3.2 我已有Compass-CI集群，如何获得rootfs？](./how-to-get-rootfs/README.md) + + +3.3 我已有Compass-CI集群，已有rootfs，如何更新？ +------------------------------------------------ + +### 3.3.1 更新os_mount=container的rootfs + +根据docker registry里的基线版本启动一个docker container，进行需要的更新后，覆盖提交到docker registry。 + +### 3.3.2 更新os_mount=cifs/nfs/local的rootfs + +根据/srv/os/${os}/${os_arch}/${os_version}这个软链接指向的基线版本目录复制一个新的版本目录，在新版本目录中进行需要的更新后，将软链接指向新的版本。 + +### 3.3.3 更新os_mount=initramfs的rootfs + +将/srv/initrd/osimage/${os}/${os_arch}/${os_version}/current这个软链接指向的基线版本cgz解压到一个临时目录，在临时版本目录中进行需要的更新后，将临时目录重新打包为新的版本，并将软链接指向新的版本。 + +解压cgz命令参考： +``` +unzip_dir="$(date "+%Y%m%d%H%M")-unzip" + +mkdir -p $unzip_dir && cd $unzip_dir +zcat < $(realpath /srv/initrd/osimage/${os}/${os_arch}/${os_version}/current) | cpio -idmv +``` + +制作cgz命令参考： +- 参考之前的手动制作initramfs rootfs步骤 + + +3.4 Compass-CI内部是如何使用rootfs的？ +-------------------------------------- + +[3.4.1 os_mount=container的rootfs是如何使用起来的](./compass-ci-use-rootfs/container.md) + +[3.4.2 os_mount=cifs/nfs的rootfs是如何使用起来的](./compass-ci-use-rootfs/cifs-nfs.md) + +[3.4.3 os_mount=local的rootfs是如何使用起来的](./compass-ci-use-rootfs/local.md) + +[3.4.4 os_mount=initramfs的rootfs是如何使用起来的](./compass-ci-use-rootfs/initramfs.md) diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md new file mode 100644 index 000000000000..9812bf439363 --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md @@ -0,0 +1,58 @@ +#### 3.4.2.2 物理机使用cifs/nfs的rootfs流程 + +物理机类型的testbox需要设置物理机的首选启动项为PXE，它的运行是从上电启动开始的。 + +所以我们分析rootfs如何使用，从上电开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- 物理机上电启动，进入到pxe阶段，pxe阶段会向同一网络发送pxe请求； + +- 这时Compass-CI集群提供的dnsmasq服务，会接收到这个pxe请求，返回给物理机一个可用的ipxe驱动下载地址； # 如：/tftpboot/ipxe/bin-arm64-efi/snp.efi + +- 物理机拿到并加载ipxe驱动，加载完成后会再次向同一网络发出ipxe请求； + +- 这时Compass-CI集群提供的dnsmasq服务，会接收到这个ipxe请求，返回给物理机一个tftp ipxe文件下载地址； # 如：boot.ipxe （此处为相对路径，即文件真实路径：/tftpboot/boot.ipxe） + +- 物理机拿到并执行ipxe命令行所组成的boot.ipxe。 + - boot.ipxe内容如下： + ``` + #!ipxe + set scheduler 172.168.131.113 + set port 3000 + + chain http://${scheduler}:${port}/boot.ipxe/mac/${mac:hexhyp} + + exit + ``` + - 所以，到此步骤，物理机testbox已经与调度器建立了连接，等待job。 + + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回如下返回值: + ``` + chain http://#{ENV["SCHED_HOST"]}:#{ENV["SCHED_PORT"]}/boot.ipxe/mac/${mac:hexhyp}" + ``` + - 物理机接收到此返回值，会继续请求job + + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- 客户端解析并执行服务端（调度器）传回来的返回值，开始启动 + + - 服务端传回来的返回值，直接是由ipxe命令组成的字符串，所以，这个字符串会被ipxe直接解析执行 + - 如上面的例子，这个字符串的最后一行是“boot”，所以，物理机此时，会进入Compass-CI自定义化的Linux开机流程。 + +- Compass-CI的自定义化的linux开机流程 + 由于都是os_mount=cifs/nfs，所以，物理机的开机流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”部分的开机流程一致。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md new file mode 100644 index 000000000000..4195f5ab2a4a --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md @@ -0,0 +1,108 @@ +#### 3.4.2.1 虚拟机使用cifs/nfs的rootfs流程 + +虚拟机类型的testbox是通过调用/c/compass-ci/providers/qemu.sh脚本，同时给其传入各种运行时参数，运行起来的。 + +所以我们分析rootfs如何使用，从这个脚本开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- /c/compass-ci/providers/qemu.sh + - functions: + main() + - key code: + source "$CCI_SRC/providers/$provider/${template}.sh" + +- /c/compass-ci/providers/qemu/kvm.sh + - functions: + write_logfile() + - key code: + ```bash + url=ws://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/ws/boot.ipxe/mac/${mac} + ipxe_script_path="$(pwd)/${ipxe_script}" + command -v ruby && + ruby -r "${CCI_SRC}/providers/lib/common.rb" -e "ws_boot '$url','$hostname','$index','$ipxe_script_path'" + ``` + - 说明： + - ws_boot()是位于/c/compass-ci/providers/lib/common.rb中的，compass-ci封装的一个方法; + - ws_boot()运行的一侧，属于客户端，它会与服务端（调度器）建立websocket长链接，请求job； + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回包含“no job now”的返回值； + /c/compass-ci/providers/qemu/kvm.sh中会处理返回值： + - 如果返回值包含“no job now”，那么继续循环请求job； + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- /c/compass-ci/providers/qemu/kvm.sh + - functions: + write_logfile() > parse_ipxe_script() > run_qemu() + - key code: + ```bash + ################### + # func: parse_ipxe_script() + # - 解析调度器传回来的返回中的kernel那一行 + # - kernel那一行举例: kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + ################### + + kernel) + kernel=$(basename "$b") + wget --timestamping -nv -a ${log_file} $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + + ################### + # func: run_qemu() + # - 使用一些参数启动qemu + # - ${append}是从上面解析得来的,请注意上面例子中kernel后面的3个关键字段： + # - overlay + # - initrd=initramfs.lkp-5.3.18-57-default.img + # - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino + ################### + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" + ``` + - 说明： + - 到此步，我们已经知道了，qemu是通过命令调用起来的，本次job所需的kernel, initrd(s), rootfs均以参数的形式传给了这个命令。 + - 接下来，就到了Compass-CI的自定义化的linux开机启动流程。 + +- Compass-CI的自定义化的linux开机流程 + + - Linux开机流程简要说明： + - 开机启动内核，内核加载initrd(s)中的所有文件到内存； # initrd(s)所有文件组合起来，会是一个文件系统 + - 在内存中依次执行initrd(s)组成的系统中定义的启动项； + - 在执行这些启动项的时候，会根据传入的内核命令行参数（也就是${append}），来找到并启动本次要真正使用的文件系统。 # 这一步骤，依然还是在initrd(s)组成的文件系统中 + - 启动本次真正要使用的文件系统。 # 这就是所谓的“切根” —— 使用的根文件系统，从initrd(s)组成的文件系统，“切”到真正要使用的文件系统 + + - 对cifs/nfs类型的job： + - 它真正要使用的文件系统，就是root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino + + - 我们传入的initrd(s)组成的文件系统，包括了当前这台testbox执行分配给它的job所需要的文件； # 如lkp.cgz，job.cgz中的文件 + - 但是这些文件是在initrd(s)组成的文件系统中的，而不在本次要真正使用的文件系统中； + - 相当于在initrd(s)组成的文件系统（/目录）中，有我们job需要的文件的； + - 而本次要真正使用的文件系统（/sysroot目录）中，没有我们job需要的文件的； # root=cifs://xxxx会通过指定的协议（cifs/nfs）挂载到/sysroot目录 + + - 所以，我们需要把job需要的文件从/目录，拷贝到/sysroot目录。 + - 这个拷贝的动作，是由initrd(s)中的某一个开机启动项来触发的。 + + - 在cifs/nfs类型的job，由于我们传入的内核命令行参数中有overlay这个关键字，而overlay这个关键字，对应的有其开机启动项，拷贝的动作就是在这个开机启动项触发的。 + - overlay对应的开机启动项，在默认的initrd中是不支持的，所以我们每做出一个rootfs，都需要通过/c/compass-ci/container/dracut-initrd这个容器来生成一个initrd。 + - 在本例中，就是initrd=initramfs.lkp-5.3.18-57-default.img + - 而cifs/nfs类型的job，这个拷贝的动作对应的代码，位于：/c/compass-ci/container/dracut-initrd/bin/overlay-lkp.sh + + - 拷贝前：root=cifs://xxxx已经通过指定的协议（cifs/nfs）挂载到/sysroot目录了 + - 拷贝中：job所需要的文件会被拷贝到job真正要使用的文件系统（/sysroot）中 + - 拷贝后：/sysroot是本次要真正使用的文件系统，接下来就是来启动/sysroot中的文件系统。 + + - 在这个系统中，已经有我们执行lkp所定义的任务所需要的一系列文件。其中就包括一个关键的系统服务：lkp-bootstrap.service。 + - 所以，在/sysroot中的文件系统执行它自己的开机启动流程时，就能通过lkp-bootstrap.service，来执行我们定义好的job。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md new file mode 100644 index 000000000000..dbfe10f6e9ae --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md @@ -0,0 +1,7 @@ +### 3.4.2 os_mount=cifs/nfs的rootfs是如何使用起来的 + +os_mount=cifs/nfs的rootfs会被虚拟机/物理机类型的testbox使用，虚拟机与物理机类型的testbox的启动方式不同，所以分开来讲。 + +[3.4.2.1 虚拟机使用cifs/nfs的rootfs流程](./cifs-nfs-qemu.md) + +[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md) diff --git a/doc/help/rootfs/compass-ci-use-rootfs/container.md b/doc/help/rootfs/compass-ci-use-rootfs/container.md new file mode 100644 index 000000000000..31f961f9fdba --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/container.md @@ -0,0 +1,86 @@ +### 3.4.1 os_mount=container的rootfs是如何使用起来的 + +os_mount=container的rootfs会被docker类型的testbox使用，docker类型的testbox是通过调用/c/compass-ci/providers/docker.rb脚本，同时给其传入各种运行时参数，运行起来的。 + +所以我们分析rootfs如何使用，从这个脚本开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- /c/compass-ci/providers/docker.rb + - key code: + ```ruby + require_relative './docker/docker' + main(ENV['hostname'], ENV['queues'], ENV['uuid'], ENV['index']) + ``` + +- /c/compass-ci/providers/docker/docker.rb + - functions: + main() > parse_response() + - key code: + ```ruby + require_relative '../lib/common' + ...omit... + response = ws_boot(url, hostname, index) + ``` + - 说明： + - ws_boot()是位于/c/compass-ci/providers/lib/common.rb中的，compass-ci封装的一个方法; + - ws_boot()运行的一侧，属于客户端，它会与服务端（调度器）建立websocket长链接，请求job； + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回包含“no job now”的返回值； + /c/compass-ci/providers/docker/docker.rb中会处理返回值： + - 如果返回值包含“no job now”，那么继续循环请求job； + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + {"job_id"=>"crystal.3583794", "docker_image"=>"centos:7", "initrds"=>"[\"http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3583794/job.cgz\",\"http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz\",\"http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd…"]"} + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- /c/compass-ci/providers/docker/docker.rb + - functions: + parse_response() > start_container() + - key code: + ```ruby + ################### + # hash: 是服务端返回值解析而来的字典 + # - 如上例: "docker_image"=>"centos:7" + ################### + + docker_image = hash['docker_image'] + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" + ``` + +- /c/compass-ci/sbin/docker-pull + - functions: + main() > local_repository() + - key code: + ```bash + docker pull $DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/$image_name 2> /dev/null + ``` + - 说明： + - 到此步，Compass-CI集群的docker registry里面的os_mount=container的rootfs（docker iamge），就被pull到执行机本地了。 + +- /c/compass-ci/providers/docker/docker.rb + - functions: + start_container() + - key code: + ```ruby + system( + { 'job_id' => hash['job_id'], + 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" }, + ENV['CCI_SRC'] + '/providers/docker/run.sh' + ) + ``` + +- /c/compass-ci/providers/docker/run.sh + - key code: + ```bash + docker run + ...omit... + ${docker_image} + ``` + - 说明： + - 到此步，Compass-CI集群的docker registry里面的os_mount=container的rootfs（docker iamge），就被使用起来了。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md b/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md new file mode 100644 index 000000000000..6b917738565f --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md @@ -0,0 +1,80 @@ +### 3.4.4 os_mount=initramfs的rootfs是如何使用起来的 + +#### 3.4.4.1 虚拟机使用initramfs的rootfs流程 + +虚拟机使用initramfs的rootfs流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”大致相同，我们这里只说不同点。 + +- 不同点1：调度器的返回值不同 + - cifs/nfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - initramfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20210609… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8800/initrd/deps/initramfs/debian/aarch64/sid/run-ip… + initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/l… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3585813/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro rdinit=/sbin/init prompt_ramdisk=0 initrd=20210609.0.cgz initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=run-ipconfig_20201103.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - 分析： + 两种os_mount的不同，核心在于传递给kernel的命令行参数不同，而这个不同，会反映在不同点2：Compass-CI的自定义化的Linux启动流程不同 + +- 不同点2：内核命令行参数不同，导致Compass-CI的自定义化的Linux启动流程不同 + + - os_mount=initramfs的系统，会直接使用在内存中解开的initrd(s)启动的系统，没有“切根”步骤 + - initramfs的系统开机流程简要说明： + - 开机启动内核，内核加载initrd(s)中的所有文件到内存； # initrd(s)所有文件组合起来，会是一个文件系统 + - 在内存中依次执行initrd(s)组成的系统中定义的启动项； # 所以，os_mount=initramfs，它的job的运行环境，是在内存中的一个文件系统。 + + - cifs/nfs与local的内核命令行参数相同项： + - user=lkp + - job=/lkp/scheduled/job.yaml + - ip=dhcp + - initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz + - initrd=job.cgz + - initrd=v2021.09.23.cgz + - initrd=9f87e65401d649095bacdff019d378e6.cgz + - rootfs_disk=/dev/vdb + - crashkernel=auto + - rootovl + - ro + + - cifs/nfs的独立参数项： + - initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img + - 这个initrd文件是我们通过/c/compass-ci/container/dracut-initrd容器制作出来的，它包含一个最小化的文件系统，这个文件系统，是debian的。 + - 但是os_mount=initramfs，由于没有切根的动作，所以它不能有debian系统的文件，它需要的是我们本次job真正要使用的文件系统。 + + - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino # root设备使用cifsroot/nfsroot，它会调用95cifs/95nfs这个启动项。 + - 说明： + - 从数字上95cifs/95nfs，会先于90overlay-root执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95cifs/95nfs，挂载远程的cifs/nfs服务端的rootfs到/sysroot； + - 然后执行90overlay-root，会基于/sysroot，再覆盖一层可读写的overlayfs，并将job所需的文件拷贝到这层overlayfs中； + - 然后将这层可读写的overlayfs作为本次要使用的文件系统，并启动它，开始执行job。 + + - initramfs的独立参数项： + - rdinit=/sbin/init # 指定initrd(s)组成的文件系统中，要运行的1号进程 + - prompt_ramdisk=0 # 如上面我们讲的，os_mount=initramfs对应的文件系统是内存文件系统，所以需要这个参数，来让内核支持内存文件系统 + - initrd=run-ipconfig_20201103.cgz # 内存文件系统需要initramfs-tools来支持一些功能，比如支持rd.init，支持ipconfig配置ip等... （详情参考：https://manpages.ubuntu.com/manpages/xenial/man8/initramfs-tools.8.html） + - initrd=20210609.0.cgz # 这个文件是os_mount=initramfs所对应的rootfs，它里面是本次job所指定的${os},${os_arch},${os_version}的所有文件 + + - 说明： + - os_mount=initramfs没有切根动作，initrd(s)组成的系统直接就是job的执行环境。 + +#### 3.4.4.2 物理机使用initramfs的rootfs流程 + +物理机使用initramfs的rootfs流程与[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md)大致相同，不同点见上面的“3.4.4.1 虚拟机使用initramfs的rootfs流程”部分的两个不同点。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/local.md b/doc/help/rootfs/compass-ci-use-rootfs/local.md new file mode 100644 index 000000000000..3d974ea3c9af --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/local.md @@ -0,0 +1,80 @@ +### 3.4.3 os_mount=local的rootfs是如何使用起来的 + +os_mount=local的rootfs会被虚拟机/物理机类型的testbox使用，虚拟机与物理机类型的testbox的启动方式不同，所以分开来讲。 + +#### 3.4.3.1 虚拟机使用local的rootfs流程 + +虚拟机使用local的rootfs流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”大致相同，我们这里只说不同点。 + +- 不同点1：调度器的返回值不同 + - cifs/nfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - local返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584978/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp local use_root_partition= save_root_partition= os_version=20.03-iso os_lv_size=10G os_partition= rw root=172.168.131.113:/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07 initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - 分析： + 两种os_mount的不同，核心在于传递给kernel的命令行参数不同，而这个不同，会反映在不同点2：Compass-CI的自定义化的Linux启动流程不同 + +- 不同点2：内核命令行参数不同，导致Compass-CI的自定义化的Linux启动流程不同 + - cifs/nfs与local的内核命令行参数相同项： + - user=lkp + - job=/lkp/scheduled/job.yaml + - ip=dhcp + - initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img + - initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz + - initrd=job.cgz + - initrd=v2021.09.23.cgz + - initrd=9f87e65401d649095bacdff019d378e6.cgz + - rootfs_disk=/dev/vdb + - crashkernel=auto + + - cifs/nfs的独立参数项： + - rootovl # 这个参数项会让dracut的90overlay-root，这一开机启动项执行 + - ro # root设备以只读模式挂载，因为有了rootovl，所以我们job对于系统所有的操作，都是在overlay的upper层进行的，并不会影响远程挂载的nfs/cifs服务端的文件 + - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino # root设备使用cifsroot/nfsroot，它会调用95cifs/95nfs这个启动项。 + - 说明： + - 从数字上95cifs/95nfs，会先于90overlay-root执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95cifs/95nfs，挂载远程的cifs/nfs服务端的rootfs到/sysroot； + - 然后执行90overlay-root，会基于/sysroot，再覆盖一层可读写的overlayfs，并将job所需的文件拷贝到这层overlayfs中； + - 然后将这层可读写的overlayfs作为本次要使用的文件系统，并启动它，开始执行job。 + + - local的独立参数项： + - local # 这个参数项是我们自定义的，它会让我们给initrd(s)的系统中自定义的的开机启动项90lkp被调用 + - use_root_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - save_root_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_version=20.03-iso # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_lv_size=10G # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - rw # root设备以读写模式挂载，这个参数只会在高级用户需要自定义ipxe命令行，且指定的root设备直接是某一块需要可读写的块设备时用到 + - root=172.168.131.113:/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07 # 这个是固定的，一方面会执行95nfs，另一方面，在之后的90lkp也会用到它 + - 说明： + - 关于90lkp，请见：/c/compass-ci/container/dracut-initrd/modules.d/90lkp + - 从数字上95nfs，会先于90lkp执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95nfs，挂载远程的nfs服务端的rootfs到/sysroot（实际上并不会用到它）； + - 然后执行90lkp，会根据传入的内核命令行参数（包括root），来将root对应的rootfs拷贝到指定格式的lv中； # 详见：/c/compass-ci/container/dracut-initrd/bin/set-local-sysroot.sh + - 然后将这个lv作为本次要使用的文件系统，并启动它，开始执行job。 + + +#### 3.4.3.2 物理机使用local的rootfs流程 + +物理机使用local的rootfs流程与[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md)大致相同，不同点见上面的“3.4.3.1 虚拟机使用local的rootfs流程”部分的两个不同点。 + diff --git a/doc/help/rootfs/how-to-get-rootfs/README.md b/doc/help/rootfs/how-to-get-rootfs/README.md new file mode 100644 index 000000000000..74673df0a9b0 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/README.md @@ -0,0 +1,35 @@ +### 3.2.1 从compass-ci官方下载 + +待补充 + +### 3.2.2 通过自动化工具制作rootfs + +#### 3.2.2.1 自动化制作os_mount=container的rootfs + +待补充 + +#### 3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs + +[3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs](./auto-generate-cifs-nfs.md) + +#### 3.2.2.3 自动化制作os_mount=local的rootfs + +待补充 + +#### 3.2.2.4 自动化制作os_mount=initramfs的rootfs + +待补充 + +### 3.2.3 通过手动方式制作rootfs + +#### 3.2.3.1 手动制作os_mount=container的rootfs + +待补充 + +#### 3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs + +[3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs](./manual-generate-cifs-nfs-local.md) + +#### 3.2.3.3 手动制作os_mount=initramfs的rootfs + +[3.2.3.3 手动制作os_mount=initramfs的rootfs](./manual-generate-initramfs.md) diff --git a/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md b/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md new file mode 100644 index 000000000000..8a848f947cd0 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md @@ -0,0 +1,97 @@ +#### 3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs + +**说明：由于自动化天然有其有限性，故此种方式有其支持的[iso范围](https://gitee.com/ycvayne/iso2qcow2/tree/master/conf/ks)，如果需要制作的iso版本不在支持的iso范围中，请参考下面的手动安装iso制作rootfs部分** + +在本地compass-ci集群中，提交iso2rootfs.yaml即可 + +``` +zhangsan@localhost ~% cat iso2rootfs-21.09.yaml +suite: iso2rootfs +category: benchmark +iso2rootfs: + ################# + # iso related fields to be used to generate rootfs + ################# + + iso_os: openeuler + iso_arch: aarch64 + iso_version: 21.09 + + ################# + # place the result rootfs related fields + ################# + + # 1. Result rootfs will be placed in the following location on the + # remote file server: + # - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + # 2. Remote file server protocols current supported: + # - nfs + # - cifs + rootfs_protocol: cifs + rootfs_server: 1.1.1.1 + rootfs_path: os-rw + rootfs_mount_param: guest,vers=1.0,noacl,nouser_xattr + + initramfs_protocol: cifs + initramfs_server: 1.1.1.1 + initramfs_path: initrd/osimage + initramfs_mount_param: port=446,guest,vers=1.0,noacl,nouser_xattr + + ################# + # config rootfs related fields + ################# + + # you can config add some configurations of the result rootfs. + # supported fields: + # - dns: will config /etc/resolv.conf. + # - no_selinux: will disable selinux in /etc/selinux/config. + # - no_fstab: will comment all line in /etc/fstab. + # - enable_repos: will enable all repo file of result rootfs. + config_rootfs: no_selinux, no_fstab + + ## install pkgs for result rootfs + ## - example: vim, git, xxx + rootfs_install_pkgs: + + ################# + # iso srouce related fields + ################# + + # iso url which you want generate rootfs from + # - demo: + # iso_url: http://1.1.1.1/openEuler-20.03-LTS-aarch64-LTS/openEuler-20.03-LTS-aarch64-… + + iso_url: http://1.1.1.1:8000/os/install/iso/openeuler/aarch64/21.09/openEuler-21.09-… + + # dailybuild_iso_url_file: + # - The `dailybuild iso url file` content is the url of a iso. + # - The iso checksum file also exists on the network, and the checksum file path is "{dailybuild_iso_url}.check256sum" + # - if your have this field, the above `iso_url` field will be useless. + # - demo: + # dailybuild_iso_url_file: http://1.1.1.1/dailybuilds/openEuler-20.03-LTS-aarch64-LTS/release_iso + # root@localhost ~% curl http://1.1.1.1/dailybuilds/openEuler-20.03-LTS-aarch64-LTS/release_iso + # http://1.1.1.1//dailybuilds/openEuler-20.03-LTS-aarch64-LTS/1970-01-01-00-0… + dailybuild_iso_url_file: + + ################# + # submit test yaml related fields + ################# + + # submit target tests yaml + ## 1. You can add as many jobs as you like. + ## 2. The following three fields is required for every test job. + test1_yaml: iperf.yaml + test1_os_mount: initramfs + test1_testbox: vm-2p16g + ## the following three fields is required when your job.yaml and + ## script for this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: + +secrets: + MY_EMAIL: XXXXXXXXXXX(a)163.com + MY_NAME: zhangsan + MY_TOKEN: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +zhangsan@localhost ~% #submit -m ./iso2rootfs-21.09.yaml testbox=taishan200-2280-2s64p-128g--a108 +``` diff --git a/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md b/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md new file mode 100644 index 000000000000..97160f70eb9c --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md @@ -0,0 +1,58 @@ +#### 3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs + +**说明：从本质上来讲，cifs/nfs/local所需的rootfs是可以共用的，所以，如果你制作出了local类型的rootfs，那么cifs/nfs均可以使用此rootfs。** + +1. 手动安装iso到物理机A # 此步骤无说明，请自行百度 + - 硬盘可选任意一块硬盘A； + - 安装完毕，一般会提示“按Enter键重启系统”。 +2. firstboot + - 在1步骤的“按Enter键重启系统”，启动系统后，不要做任何操作，直接reboot再次重启系统 +3. 将物理机A上的硬盘A中的系统拷贝到指定目录。 # 如：/srv/os/${os}/${os_arch}/${os_version}-20211129000000 + - 使用物理机A上的非A硬盘中的系统启动物理机A + - 使用位于物理机A上的其他硬盘的系统启动物理机A + - 可以给物理机A提交一个内存文件系统的job（此job不限制os,os_arch,os_version） + - 在物理机A上，拷贝硬盘A中的系统到指定目录。 + - 通过nfs/cifs协议挂载你的Compass-CI集群主节点的/srv/os/${os}/${os_arch}/${os_version}-20211129000000至/mnt1； # rw读写模式 + - 挂在硬盘A至/mnt2 + - rsync -a /mnt2/. /mnt + - 释放物理机A +4. 对指定目录的rootfs进行一些处理 + - 解压内核并生成${rootfs}/boot/vmlinuz + - 解压内核(ipxe需要)：如果你的内核格式中有gzip字样，请参考下面的解决文档解决 + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: unzip_vmlinuz + - 生成vlinuz软链接（compass-ci代码逻辑需要）： + ``` + root@z9 /srv/os/openeuler/aarch64/20.03-iso# ll boot | grep vmlinuz + -rwxr-xr-x 1 root root 7.1M 2021-06-29 20:56 vmlinuz-0-rescue-8f22ff33e906472498de4a5b3fc087ef + -rw-rw-r-- 1 root root 20M 2021-06-29 21:10 vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 + lrwxrwxrwx 1 root root 43 2021-06-29 21:10 vmlinuz -> ./vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 + ``` + - 生成${rootfs}/boot/modules.cgz（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: create_get_modules + - 生成${rootfs}/initrd.lkp（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: create_get_initrd + - 注释掉${rootfs}/etc/fstab的启动项（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/lkp-tests/blob/master/tests/iso2rootfs + - function: disable_fstab + - 关闭${rootfs}的selinux + 参考： + - file: https://gitee.com/wu_fengguang/lkp-tests/blob/master/tests/iso2rootfs + - function: disable_selinux + - 创建真实版本的软链接 + ``` + root@z9 /srv/os/openeuler/aarch64# ll -d 20.03-iso* + dr-xr-xr-x 1 root root 190 2021-08-24 09:09 20.03-iso-2021-08-24-18-01-07 + dr-xr-xr-x 1 root root 190 2021-08-24 09:09 20.03-iso-2021-08-24-09-07-18 + lrwxrwxrwx 1 root root 29 2021-08-24 18:16 20.03-iso -> 20.03-iso-2021-08-24-18-01-07 + ``` +5. 使用制作出来的rootfs提交job + ``` + root@localhost ~% submit -m -c borrow-1d.yaml testbox=vm-2p16g os=openeuler os_version=20.03-iso os_mount=cifs + ``` diff --git a/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md b/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md new file mode 100644 index 000000000000..d7e5553dd165 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md @@ -0,0 +1,39 @@ +#### 3.2.3.3 手动制作os_mount=initramfs的rootfs + +1. 将制作出来的相同${os},${os_arch},${os_version}的os_mount=cifs/nfs/local的rootfs拷贝一份到临时目录。 + ```bash + mkdir /srv/os/openeuler/aarch64/20.03-iso-tmp + rsync -a /srv/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07/. /srv/os/openeuler/aarch64/20.03-iso-tmp/ + ``` +2. 进入临时目录，删掉一些不需要加入到initamfs系统的文件 + + **说明：也可以不删除这些文件，如果你知道你需要它们** + ```bash + cd /srv/os/openeuler/aarch64/20.03-iso-tmp/ + rm -rf lib/modules/* + rm -rf boot/vmlinuz* + rm -rf boot/initr* + ``` + + 3. 生成initramfs的rootfs + ```bash + ############ + # 为什么要去掉${os_version}的-iso后缀： + # - 目前，只有os_mount=local对应的rootfs需要-iso后缀，initramfs不需要 + ############ + + initramfs_cgz=/srv/initrd/osimage/$os/$os_arch/${os_version%-iso}/$(date +"%Y%m%d").0.cgz + mkdir -p $(dirname $initramfs_cgz) + cd /srv/os/openeuler/aarch64/20.03-iso-tmp/ + find . |cpio -o -Hnewc | gzip -9 > $initramfs_cgz + + cd $(dirname $initramfs_cgz) + ln -sf $(basename $initramfs_cgz) current + + ############ + # 如何获得../../../../deps/nfs/debian/aarch64/sid/run-ipconfig.cgz： + # - 从Compass-CI官网下载：https://api.compass-ci.openeuler.org:20008/initrd/deps/nfs/debian/aarch64/sid/ + ############ + + ln -sf ../../../../deps/nfs/debian/aarch64/sid/run-ipconfig.cgz run-ipconfig.cgz + ``` -- 2.23.0

1 0

test
by FengGuang Wu 05 Dec '21

05 Dec '21

1 0

test
by Wu Fengguang 05 Dec '21

05 Dec '21

1 0

Re: [PATCH lkp-tests] tests/rpmbuild: use HTTPS instead
by Liu Shaofei 09 Oct '21

09 Oct '21

On Sat, Oct 09, 2021 at 11:36:27AM +0800, Wang Lin wrote: >change to use https and domain name > >Signed-off-by: Wang Lin <wanglin199(a)huawei.com> >--- > tests/rpmbuild | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > >diff --git a/tests/rpmbuild b/tests/rpmbuild >index 0f5244a1b..9b13591d2 100755 >--- a/tests/rpmbuild >+++ b/tests/rpmbuild >@@ -206,7 +206,13 @@ localinstall_dependent_package() > yum install -y wget > mkdir /tmp/rpm > cd /tmp/rpm >- wget -c -r -np -nd "http://${LKP_SERVER}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" >+ local protocol >+ if [ ${#SRV_HTTP_PROTOCOL} -ne 0 ] ; then >+ protocol=$SRV_HTTP_PROTOCOL >+ else >+ protocol="https" >+ fi >+ wget -c -r -np -nd "$protocol://${SCHED_HOST}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" perhaps you can write like this: wget -c -r -np -nd "${SRV_HTTP_PROTOCOL:-https}://${SCHED_HOST}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" Thanks, Liushaofei > yum localinstall -y *.rpm > } > show_rpm_files() >-- >2.23.0 >

1 0

openEuler Summit 2021 诚挚的邀请您提交演讲议题、成为共建单位、贡献展示方案
by Marketing openEuler 29 Sep '21

29 Sep '21

openEuler Summit 是由欧拉开源社区举办的开发者峰会。全新升级后的首场峰会将于 11 月10日在北京举行。 openEuler 从服务器操作系统，升级为数字基础设施的操作系统，支持 IT、CT、OT 等数字基础设施全场景，覆盖服务器、云、边、嵌入式等各种形态的需求。伴随 openEuler 21.09 的发布，openEuler 已经包含了服务器、云原生、边缘计算和嵌入式的四大应用场景。 openEuler 通过开源开放，不断探索科技创新的边界。开发者、用户、社区贡献者、软件爱好者在 openEuler Summit 汇聚，共同解读 openEuler 的最新版，探讨未来的技术路线，让技术、生态、商业在这里产生奇妙的化学反应。 *本次峰会旨在推动 openEuler 在多样性计算、云计算、边缘计算、嵌入式等技术方向的持续探索和创新。* 开源是一种态度、分享是一种精神。Call for Speaker、Call for Sponsor、Call for Demo 现已全面开放报名。大会报名通道同步开启。 *我们诚挚的邀请您提交演讲议题、发表联合演讲、成为共建单位、贡献展示方案、参与社区建设。* - *大会官网：*https://www.openeuler.org/zh/interaction/summit-list/summit2021/ - *Call for Speaker：*https://shimowendang.com/forms/X6X9jj9KPcdQwVr8/fill - *Call for Sponsor：*https://shimowendang.com/forms/k76zTLKvumYwRdsP/fill - *Call for Demo：*https://shimowendang.com/forms/LjHs8JlsLSsW92kl/fill

1 0

Re: [PATCH compass-ci 2/2] modify url about srv-http
by Liu Shaofei 25 Aug '21

25 Aug '21

On Wed, Aug 25, 2021 at 04:55:29PM +0800, Lu Weitao wrote: >[Why] >compass-ci/container/srv-http may change the communication protocol: > http => https >it's depends on whether there are ssl certificates on the server >we need convert all 'http://compass-ci.api.openeuler.org.*' to >'https://compass-ci.api.openeuler.org.*' > >[How] >scope: gg http.*SRV_HTTP > >lib/mail_bisect_result.rb:44: prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >lib/mail_bisect_result.rb:45: prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" >sbin/download-rootfs:38: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" >sbin/download-rootfs:48: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" >sbin/download-rootfs:49: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" >sparrow/0-package/os/openEuler:53: wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >sparrow/0-package/os/openEuler:54: wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/constants.rb | 3 +++ > lib/mail_bisect_result.rb | 4 ++-- > sbin/download-rootfs | 6 +++--- > sparrow/0-package/os/openEuler | 4 ++-- > sparrow/local/compare-results.md | 2 +- > 5 files changed, 11 insertions(+), 8 deletions(-) > >diff --git a/lib/constants.rb b/lib/constants.rb >index d85ca5c3..38e1ec92 100644 >--- a/lib/constants.rb >+++ b/lib/constants.rb >@@ -70,6 +70,9 @@ SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || > > SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' > >+SRV_HTTP_PROTOCOL ||= >+ File.exist?('/etc/ssl/certs/web-backend.key') && File.exist?('/etc/ssl/certs/web-backend.crt') ? 'https' : 'http' >+ > ASSISTANT_HOST ||= config['ASSISTANT_HOST'] || ENV['ASSISTANT_HOST'] || '172.17.0.1' > ASSISTANT_PORT ||= config['ASSISTANT_PORT'] || ENV['ASSISTANT_PORT'] || 8101 > >diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb >index 855fcdc9..cff3137b 100644 >--- a/lib/mail_bisect_result.rb >+++ b/lib/mail_bisect_result.rb >@@ -41,8 +41,8 @@ class MailBisectResult > > def compose_mail > subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" >- prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >- prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" >+ prefix_srv_result = "#{SRV_HTTP_PROTOCOL}://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >+ prefix_srv_git = "#{SRV_HTTP_PROTOCOL}://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" > pkgbuild_repo_url = "we build project with this script:\n#{prefix_srv_git}/git/#{@pkgbuild_repo}/PKGBUILD\n" > first_bad_commit_job_url = "first bad commit result :\n#{prefix_srv_result}#{@first_bad_commit_result_root}/build-pkg\n" > >diff --git a/sbin/download-rootfs b/sbin/download-rootfs >index 0d0d1c7b..9f1d0070 100755 >--- a/sbin/download-rootfs >+++ b/sbin/download-rootfs >@@ -35,7 +35,7 @@ wget_download() > download_os_rootfs() > { > cd $rootfs_dir && { >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch…" > > gzip -dc ${os_version}.cgz | cpio -id > rm -rf ${os_version}.cgz >@@ -45,8 +45,8 @@ download_os_rootfs() > download_osimage() > { > cd $osimage_dir && { >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nf…" ^-----^-----^-------^--- need to change ${os_mount}/${os}/${os_version} ? Thanks, Liushaofei > } > } > >diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler >index 1c61e2c0..399e5e47 100755 >--- a/sparrow/0-package/os/openEuler >+++ b/sparrow/0-package/os/openEuler >@@ -50,8 +50,8 @@ public_network_ok && { > > # lib64lockfile1 from http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > # lockfile-progs from http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >- wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >- wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >+ wget_rpm https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >+ wget_rpm https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > > rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm > rm -rf *.rpm >diff --git a/sparrow/local/compare-results.md b/sparrow/local/compare-results.md >index e555f32e..d82b0165 100644 >--- a/sparrow/local/compare-results.md >+++ b/sparrow/local/compare-results.md >@@ -6,7 +6,7 @@ > submit -m cci-depends.yaml cci-depends.benchmark=iperf > ``` > >-2. 编辑os-matrix-dc.yaml，指定使用centos：7和 [openEuler：20.03-pre](http://api.compass-ci.openeuler.org:20008/initrd/dockerimage/openeuler-pre.tar)（由于dockerhub没有openeuler20.03的镜像，可以下载该链接并使用docker load -i openeuler-pre.tar加载到本地），如需要使用其他os，参考如下文件格式修改即可。 >+2. 编辑os-matrix-dc.yaml，指定使用centos：7和 [openEuler：20.03-pre](https://api.compass-ci.openeuler.org:20008/initrd/dockerimage/openeuler-pre.tar)（由于dockerhub没有openeuler20.03的镜像，可以下载该链接并使用docker load -i openeuler-pre.tar加载到本地），如需要使用其他os，参考如下文件格式修改即可。 > ``` > cat > /c/lkp-tests/jobs/os-matrix-dc.yaml << EOF > os_arch: aarch64 >-- >2.23.0 >

1 0

Re: [PATCH compass-ci 1/2] container/srv-http: support https protocol
by Liu Shaofei 25 Aug '21

25 Aug '21

On Wed, Aug 25, 2021 at 04:55:28PM +0800, Lu Weitao wrote: >[Why] >Some clients(chrome|firefox bowser) can't access srv-http like: > http://api.compass-ci.openeuler.org/pub >because the url will be change as:(it's default behavior of bowser) > https://api.compass-ci.openeuler.org/pub > >so srv-http need support https access > >[How] >with start container/srv-http: > > check current server exists?: > /etc/ssl/certs/web-backend.crt > /etc/ssl/certs/web-backend.key > | | > v v > yes no > config nginx.conf like: config /nginx.conf like: > ... ... > listen $port ssl: listen $port; > ... > | > v > start nginx in container > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > container/srv-http/docker_run.sh | 9 +++++++++ > container/srv-http/root/sbin/entrypoint.sh | 13 ++++++++++++- > container/srv-http/start-cci | 2 ++ > container/srv-http/start-git | 2 ++ > container/srv-http/start-initrd | 2 ++ > container/srv-http/start-os | 2 ++ > container/srv-http/start-pub | 2 ++ > container/srv-http/start-result | 2 ++ > container/srv-http/start-rpm | 2 ++ > 9 files changed, 35 insertions(+), 1 deletion(-) > create mode 100644 container/srv-http/docker_run.sh > >diff --git a/container/srv-http/docker_run.sh b/container/srv-http/docker_run.sh >new file mode 100644 >index 00000000..fcd9083d >--- /dev/null >+++ b/container/srv-http/docker_run.sh >@@ -0,0 +1,9 @@ >+#!/bin/bash >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+ >+mount_ssl(){ >+ if [ -f "/etc/ssl/certs/web-backend.key" ] && [ -f "/etc/ssl/certs/web-backend.crt" ]; then >+ echo "-v /etc/ssl/certs:/opt/cert" >+ fi defined var: ssl_path="/etc/ssl/certs/" , avoid long contents; >+} >diff --git a/container/srv-http/root/sbin/entrypoint.sh b/container/srv-http/root/sbin/entrypoint.sh >index ced6eda5..d0d354c2 100755 >--- a/container/srv-http/root/sbin/entrypoint.sh >+++ b/container/srv-http/root/sbin/entrypoint.sh >@@ -1,6 +1,17 @@ > #!/bin/sh > >-sed -i "s/listen 11300;/listen $LISTEN_PORT;/g" /etc/nginx/conf.d/default.conf >+ssl_conf="\ ssl_certificate /opt/cert/web-backend.crt;\n\ >+ ssl_certificate_key /opt/cert/web-backend.key;\n\ >+ ssl_session_timeout 5m;\n\ >+ ssl_ciphers BCDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;\n\ >+ ssl_prefer_server_ciphers on;\n" >+ >+if [ -f "/opt/cert/web-backend.key" ] && [ -f "/opt/cert/web-backend.crt" ]; then >+ sed -i "s/listen 11300;/listen $LISTEN_PORT ssl;/g" /etc/nginx/conf.d/default.conf >+ sed -i "/server_name/a $ssl_conf" /etc/nginx/conf.d/default.conf >+else >+ sed -i "s/listen 11300;/listen $LISTEN_PORT;/g" /etc/nginx/conf.d/default.conf >+fi perhaps modify like this: listen="$LISTEN_PORT" if [ -f "/opt/cert/web-backend.key" ] && [ -f "/opt/cert/web-backend.crt" ]; then listen="$listen ssl" sed -i "/server_name/a $ssl_conf" /etc/nginx/conf.d/default.conf fi sed -i "s/listen 11300;/listen $listen;/g" /etc/nginx/conf.d/default.conf Thanks, Liushaofei > nginx > /usr/sbin/php-fpm7 >diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci >index aa539d20..1f160f44 100755 >--- a/container/srv-http/start-cci >+++ b/container/srv-http/start-cci >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-cci > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20011:20011 > -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-git b/container/srv-http/start-git >index 0b0ff477..5851af6f 100755 >--- a/container/srv-http/start-git >+++ b/container/srv-http/start-git >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-git > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20010:20010 > -v /srv/git/archlinux:/srv/git/archlinux:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd >index 41958da2..c8e7eded 100755 >--- a/container/srv-http/start-initrd >+++ b/container/srv-http/start-initrd >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-initrd > > cmd=( >@@ -17,6 +18,7 @@ cmd=( > -v /srv/initrd/osimage:/srv/initrd/osimage:ro > -v /srv/initrd/deps:/srv/initrd/deps:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-os b/container/srv-http/start-os >index 5b63902e..c2012397 100755 >--- a/container/srv-http/start-os >+++ b/container/srv-http/start-os >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-os > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20009:20009 > -v /srv/os:/srv/os:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub >index 6dfd5a44..2224737e 100755 >--- a/container/srv-http/start-pub >+++ b/container/srv-http/start-pub >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-pub > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20006:20006 > -v /srv/pub:/srv/pub:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-result b/container/srv-http/start-result >index 32bdf7df..c9e0eac2 100755 >--- a/container/srv-http/start-result >+++ b/container/srv-http/start-result >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-result > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20007:20007 > -v /srv/result:/srv/result:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm >index 2d38298c..62fb1385 100755 >--- a/container/srv-http/start-rpm >+++ b/container/srv-http/start-rpm >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-rpm > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20012:20012 > -v /srv/rpm/pub:/srv/rpm/pub:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >-- >2.23.0 >

1 0

openEuler 任务打榜赛6月21日-6月28日周榜单来啦
by Marketing openEuler 29 Jun '21

29 Jun '21

openEuler 任务打榜赛6月21日-6月28日周榜单来啦，快来看看有没有你的名字吧。这里提醒已领取赛题但是还没有加群的小伙伴赶紧联系小助手加群呀。好啦~，下面让我们看一下有哪些开发者上榜吧！ gitee用户账号积分值解决issue/pr数量 yaozc701 20 4 dada5208 16 5 shuyao13 14 4 houlibao 12 3 liuerwuliu 12 3 hw999666 8 2 gc7500 8 2 weidongkl 6 1 shao1112 6 2 wss1235 2 1 恭喜以上开发者，请联系小助手领奖。没有上榜的小伙伴不要灰心，请再接再历。周榜、月榜前十名均有奖励，期待在榜单上早点见到你。添加小助手微信（SoftCorePro）回复“*任务打榜赛*”进入交流群 *了解更多* - 官网：https://openeuler.org/zh/interaction/hdc/2021-developer.html - 我要报名： https://competition.huaweicloud.com/information/1000041420/introduction - 我要做题： https://gitee.com/organizations/src-openeuler/issues?assignee_id=&author_id…

1 0

快来看你是否上榜~！openEuler 任务打榜赛5月榜单出炉
by Marketing openEuler 24 Jun '21

24 Jun '21

经过一个月的角逐，openEuler 任务打榜赛5月榜单来啦。在这一个月的时间里，开发者参加任务打榜赛的热情高涨：总计有*38*人领取*118*道赛题，赛道交流群成员*70*个，这里 *提醒领取赛题但是还没有加群的小伙伴赶紧联系小助手加群*呀。好啦~，下面让我们看一下5月份有哪些开发者上榜吧！ gitee用户账号 gitee邮箱积分值解决issue/pr数量 li-kaiyuan66666 likaiyuan7(a)huawei.com 6 1 yanxiaobing2020 yanxiaobing(a)huawei.com 2 1 zhu-yuncheng zhuyuncheng(a)huawei.com 6 1 polite2anyone zhangyao05(a)outlook.com 10 2 yaozc701 yaozc7(a)chinaunicom.cn 18 2 恭喜以上开发者，请中奖的开发者联系小助手*领奖*。没有上榜的小伙伴不要灰心请再接再历，周榜、月榜前十名均有奖励，期待在榜单上早点见到你。让我们共同期待6月的榜单。 [image: image.png] 添加小助手微信回复“*任务打榜赛*”进入交流群 *了解更多* - 官网：https://openeuler.org/zh/interaction/hdc/2021-developer.html - 我要报名： https://competition.huaweicloud.com/information/1000041420/introduction - 我要做题： https://gitee.com/organizations/src-openeuler/issues?assignee_id=&author_id…

1 1

完成任务，获得荣誉，openEuler 社区的这项比赛有点逆天！
by Marketing openEuler 27 May '21

27 May '21

先说比赛规则独立开发者参赛 *做题得分* *每周结算* 这是任务打榜赛不满足？来看一下*特性命题赛*？独立开发者组队参赛区域赛、半决赛、总决赛过五关斩六将谁可以独占鳌头？你将是*社区贡献者* *你的 ID* 会被铭刻在社区还有*“一”点奖金* [image: WechatIMG16.jpeg]

1 0

Fwd: openEuler Developer Day 启动大会招募环节，报名通道同步开启！
by Marketing openEuler 17 May '21

17 May '21

openEuler 社区的开发者：您好！ openEuler Developer Day 2021 将于*6**月10日*在北京举行。本次大会是由 openEuler 社区发起，主要面向签署了 CLA的核心开发者、南北向生态厂商以及核心用户的一场技术盛会，旨在推动 openEuler 社区在多样性计算、云原生全栈、全场景协同等技术方向的持续探索和创新。 *开放招募* openEuler Developer Day 正式开放招募环节：*Call for Speaker* <https://shimo.im/forms/KgXxt8JCYKYtVXHx/fill>、*Call for Sponsor* <https://shimo.im/forms/vwCC68q3pQvvR3Yc/fill>、*Call for SIG* <https://shimo.im/forms/x6vXyW8QHCGK9QJD/fill>、*Call for Demo* <https://shimo.im/forms/PQTvWKjHgg96DjD8/fill> 欢迎 openEuler 社区的开发者提交以上四个方面的材料。 [image: 截屏2021-05-13 下午8.12.43.png] *大会亮点* 这是一场大型的面向 openEuler 社区开发者的技术交流会议。在现场共同探讨*热点技术方向*和*如何参与社区贡献*，通过大会*公开接收用户需求*，*完成版本规划*。 *技术委员会*、*品牌委员会*、*用户委员会*、*20个 SIG 组*会同期举办*开放工作会议**（仅限签署社区 CLA 的开发者参加）*。技术委员会主席给大家解读 *openEuler 21.03*，发布下个社区版的*技术定位*，分享社区创新思路。来自互联网、金融、运营商的客户分享成功的应用实践经验。开源圆桌中，客户、伙伴、厂商、研究机构一同探究开源软件供应链的意义和风险，开源模式如何推动独立演进操作系统技术的发展。更有开发者心心念念的 *Party* 心动不如行动，赶紧来报名参会 <https://e-campaign.huawei.com/m/mIzQFr>吧！！！

1 0

[PATCH v4 lkp-tests 7/7] distro/depends: add install depend for benchmarktool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev index 23c6ff661..05583bef9 100644 --- a/distro/depends/mysql-8-0-20-dev +++ b/distro/depends/mysql-8-0-20-dev @@ -14,3 +14,4 @@ libmpc libmpc-devel mpfr mpfr-devel +java -- 2.23.0

1 0

[PATCH v4 lkp-tests 6/7] pkg/mysql-8-0-20: add build install benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 51 ++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD index 66df6b33f..26247c6f0 100644 --- a/pkg/mysql-8-0-20/PKGBUILD +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -7,13 +7,14 @@ url="http://www.mysql.com" license=('GPLv2') source=("git://$GIT_SERVER/upstream/m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch.git" "git://$GIT_SERVER/upstream/g/gcc/gcc.git" + "https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/database/p…" "my.cnf") -md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') +md5sums=('SKIP' 'SKIP' 'f33c03fd48ff5993dca6d00b53a46da4' '99c4965aa870c475000993f52a0dee66') build() { - build_gcc build_mysql + build_benchmarksql } build_gcc() @@ -47,9 +48,8 @@ build_gcc() --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ --with-boot-ldflags='-Wl,-z,relro,-z,now' \ --with-multilib-list=lp64 - make -j96 - make install & - wait + make -j96 > /dev/null + make install > /dev/null } } @@ -58,6 +58,11 @@ build_mysql() unset CPPFLAGS unset CFLAGS unset CXXFLAGS + + cd $srcdir/my-sql_with_boostkit_patch && { + grep 3f3d2692fd05a18011d93dcadeef661f386595ac compass-ci.cfg || build_gcc + } + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver CMAKE=cmake @@ -65,8 +70,36 @@ build_mysql() mkdir -p build cd build && { - $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ - make -j96 + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ > /dev/null + make -j96 > /dev/null + } +} + +build_benchmarksql() +{ + cd $srcdir && { + cp -r benchmarksql5.0-for-mysql "$HOME/" + chmod -R 777 "$HOME/benchmarksql5.0-for-mysql" + } + + cd "$HOME/benchmarksql5.0-for-mysql/run" && { + chmod 777 *.sh + sed -i \ + -e '/^conn=/ s|mysql://.*:3306|mysql://localhost:3306|' \ + -e '/^user=/cuser=root' \ + -e '/^password=/cpassword=mysql_password' \ + -e '/^warehouses=/cwarehouses=1000' \ + -e '/^loadWorkers=/cloadWorkers=100' \ + -e '/^terminals=/cterminals=200' \ + -e '/^runTxnsPerTerminal=/crunTxnsPerTerminal=0' \ + -e '/^runMins=/crunMins=30' \ + -e '/^limitTxnsPerMin=/climitTxnsPerMin=1000' \ + -e '/^terminalWarehouseFixed=/cterminalWarehouseFixed=true' \ + -e '/^newOrderWeight=/cnewOrderWeight=45' \ + -e '/^paymentWeight=/cpaymentWeight=43' \ + -e '/^orderStatusWeight=/corderStatusWeight=4' \ + -e '/^deliveryWeight=/cdeliveryWeight=4' \ + props.conf } } @@ -75,6 +108,6 @@ package() cp my.cnf /etc/ cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { - make install - } + make install > /dev/null + } } -- 2.23.0

1 0

[PATCH v4 lkp-tests 5/7] tests: add init database for benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 66 +++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 index 990a053d7..4a49e0233 100755 --- a/tests/mysql-8-0-20 +++ b/tests/mysql-8-0-20 @@ -4,69 +4,75 @@ # mysql_host # mysql_port # mysql_db +# benchmark_db # mysql_password + . $LKP_SRC/lib/common.sh : "${mysql_user:=root}" : "${mysql_host:=127.0.0.1}" : "${mysql_port:=3306}" : "${mysql_db:=sysbench_test}" +: "${benchmark_db:=tpcc}" : "${mysql_password:=mysql_password}" create_basedir() { - mkdir -p /data - mkfs.xfs -f /dev/"$data_dev" - mount /dev/"$data_dev" /data + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data - mkdir -p /data/mysql/{data,share,tmp,run,log} + mkdir -p /data/mysql/{data,share,tmp,run,log} - useradd mysql - touch /data/mysql/log/mysql.log - chown -R mysql:mysql /data - chown -R mysql:mysql /data/mysql/log/mysql.log + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log } configure_mysql() { - if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then - echo "mysql is not installed!" && exit 1 - fi + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi - "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" - sleep 10 + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 - "$mysql_server" start - sleep 10 + "$mysql_server" start + sleep 10 - "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" - "$mysql_server" restart - sleep 10 + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + sed -i '/^skip-name-resolve/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 } create_database() { - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" - "$mysql_server" stop - sleep 10 + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $benchmark_db;" + "$mysql_server" stop + sleep 10 } start_mysql() { - "$mysql_server" start & + "$mysql_server" start & wait } main() { - local mysql="/usr/local/mysql/bin/mysql" - local mysqld="/usr/local/mysql/bin/mysqld" - local mysql_server="/usr/local/mysql/support-files/mysql.server" + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" - create_basedir - configure_mysql - create_database - start_mysql + create_basedir + configure_mysql + create_database + start_mysql } main -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/7] tests: add script for benchmarksql test tool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/benchmark-sql | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 tests/benchmark-sql diff --git a/tests/benchmark-sql b/tests/benchmark-sql new file mode 100755 index 000000000..6acac2847 --- /dev/null +++ b/tests/benchmark-sql @@ -0,0 +1,15 @@ +#!/bin/sh +# benchmark_pkg_name + +: "${benchmark_pkg_name:=benchmarksql5.0-for-mysql}" + +run_benchmark_mysql() +{ + cd "$HOME/$benchmark_pkg_name/run/" && { + ./runDatabaseBuild.sh props.conf + ./runBenchmark.sh props.conf + ./runDatabaseDestroy.sh props.conf + } +} + +run_benchmark_mysql -- 2.23.0

1 0

[PATCH v4 lkp-tests 3/7] stats: add extract scritp
by Luan Shengde 30 Apr '21

30 Apr '21

add stats script for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- stats/benchmark-sql | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/benchmark-sql diff --git a/stats/benchmark-sql b/stats/benchmark-sql new file mode 100755 index 000000000..d4794b4f4 --- /dev/null +++ b/stats/benchmark-sql @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 450.31 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmTOTAL = 999.71 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session Start = 2021-04-29 10:58:15 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session End = 2021-04-29 11:28:16 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Transaction Count = 30003 + +while (line = STDIN.gets) + case line.chomp! + when /Measured tpmC $NewOrders$ = (\d*\.\d*)/ + puts "Measured_tpmC$NewOrders$: #{$1}" + when /Measured tpmTOTAL = (\d*\.\d*)/ + puts "Measured_tpmTOTAL: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/benchmark-sql.yaml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 jobs/benchmark-sql.yaml diff --git a/jobs/benchmark-sql.yaml b/jobs/benchmark-sql.yaml new file mode 100644 index 000000000..4563b666c --- /dev/null +++ b/jobs/benchmark-sql.yaml @@ -0,0 +1,16 @@ +suite: benchmark-sql + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +benchmark_db: tpcc +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +benchmark-sql: -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for sysbench-mysql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/sysbench-mysql.yaml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 000000000..af017df4b --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,28 @@ +suite: sysbench-mysql + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 60 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 20000 + mysql_socket: /data/mysql/run/mysql.sock -- 2.23.0

1 0

[PATCH v2 lkp-tests] resize virtual rootfs from 50% to 90% memory, for docker no sapce error
by Zhang Dewan 30 Apr '21

30 Apr '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- lib/bootstrap.sh | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 417c8662a..82a1c17d7 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -57,6 +57,35 @@ mount_tmpfs() mount -t tmpfs -o mode=1777 tmp /tmp } +resize_virtual_rootfs() +{ + # tmpfs and rootfs default use 50% memory: + # + # root@taishan200-2280-2s48p-256g--a13 ~# mount | head -n 1 + # rootfs on / type rootfs (rw,size=133395136k,nr_inodes=2084299) + # + # root@taishan200-2280-2s48p-256g--a13 ~# df -h / + # Filesystem Size Used Avail Use% Mounted on + # rootfs 128G 114G 14G 90% / + # + # this maybe cause the docker no space error. + + if mount | head -n 1 | grep "type rootfs" > /dev/null + then + echo "Set the rootfs to 90% memory" + mount -o remount,rw,size=90% rootfs / + fi + # Test Result: + # root@taishan200-2280-2s64p-256g--a125 ~# free -g + # total used free shared buff/cache available + # Mem: 255Gi 8.5Gi 243Gi 2.7Gi 2.9Gi 224Gi + # Swap: 0B 0B 0B + # root@taishan200-2280-2s64p-256g--a125 ~# df -h / + # Filesystem Size Used Avail Use% Mounted on + # rootfs 230G 2.7G 228G 2% / + +} + get_net_devices() { local net_devices @@ -737,6 +766,7 @@ boot_init() mount_kernel_fs mount_tmpfs + resize_virtual_rootfs redirect_stdout_stderr echo 'Kernel tests: Boot OK!' -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: specify the correct filename in source cmd
by Xu Xijian 30 Apr '21

30 Apr '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/README.md b/sparrow/README.md index 532b7f9..a519497 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -61,7 +61,7 @@ 本文以/c/lkp-tests/jobs/目录下已有的通用测试用例host-info.yaml为例 - 使环境变量生效 ```bash - source /etc/profile.d/compass-ci + source /etc/profile.d/compass.sh ``` - 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… -- 2.23.0

1 0

[PATCH v3 lkp-tests 7/7] distro/depends: add depend for benchmarktool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev index 23c6ff661..05583bef9 100644 --- a/distro/depends/mysql-8-0-20-dev +++ b/distro/depends/mysql-8-0-20-dev @@ -14,3 +14,4 @@ libmpc libmpc-devel mpfr mpfr-devel +java -- 2.23.0

1 0

[PATCH v3 lkp-tests 6/7] pkg/mysql-8-0-20: add build install benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 51 ++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD index 66df6b33f..26247c6f0 100644 --- a/pkg/mysql-8-0-20/PKGBUILD +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -7,13 +7,14 @@ url="http://www.mysql.com" license=('GPLv2') source=("git://$GIT_SERVER/upstream/m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch.git" "git://$GIT_SERVER/upstream/g/gcc/gcc.git" + "https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/database/p…" "my.cnf") -md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') +md5sums=('SKIP' 'SKIP' 'f33c03fd48ff5993dca6d00b53a46da4' '99c4965aa870c475000993f52a0dee66') build() { - build_gcc build_mysql + build_benchmarksql } build_gcc() @@ -47,9 +48,8 @@ build_gcc() --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ --with-boot-ldflags='-Wl,-z,relro,-z,now' \ --with-multilib-list=lp64 - make -j96 - make install & - wait + make -j96 > /dev/null + make install > /dev/null } } @@ -58,6 +58,11 @@ build_mysql() unset CPPFLAGS unset CFLAGS unset CXXFLAGS + + cd $srcdir/my-sql_with_boostkit_patch && { + grep 3f3d2692fd05a18011d93dcadeef661f386595ac compass-ci.cfg || build_gcc + } + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver CMAKE=cmake @@ -65,8 +70,36 @@ build_mysql() mkdir -p build cd build && { - $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ - make -j96 + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ > /dev/null + make -j96 > /dev/null + } +} + +build_benchmarksql() +{ + cd $srcdir && { + cp -r benchmarksql5.0-for-mysql "$HOME/" + chmod -R 777 "$HOME/benchmarksql5.0-for-mysql" + } + + cd "$HOME/benchmarksql5.0-for-mysql/run" && { + chmod 777 *.sh + sed -i \ + -e '/^conn=/ s|mysql://.*:3306|mysql://localhost:3306|' \ + -e '/^user=/cuser=root' \ + -e '/^password=/cpassword=mysql_password' \ + -e '/^warehouses=/cwarehouses=1000' \ + -e '/^loadWorkers=/cloadWorkers=100' \ + -e '/^terminals=/cterminals=200' \ + -e '/^runTxnsPerTerminal=/crunTxnsPerTerminal=0' \ + -e '/^runMins=/crunMins=30' \ + -e '/^limitTxnsPerMin=/climitTxnsPerMin=1000' \ + -e '/^terminalWarehouseFixed=/cterminalWarehouseFixed=true' \ + -e '/^newOrderWeight=/cnewOrderWeight=45' \ + -e '/^paymentWeight=/cpaymentWeight=43' \ + -e '/^orderStatusWeight=/corderStatusWeight=4' \ + -e '/^deliveryWeight=/cdeliveryWeight=4' \ + props.conf } } @@ -75,6 +108,6 @@ package() cp my.cnf /etc/ cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { - make install - } + make install > /dev/null + } } -- 2.23.0

1 0

[PATCH v3 lkp-tests 5/7] tests: add init database for benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 66 +++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 index 990a053d7..4a49e0233 100755 --- a/tests/mysql-8-0-20 +++ b/tests/mysql-8-0-20 @@ -4,69 +4,75 @@ # mysql_host # mysql_port # mysql_db +# benchmark_db # mysql_password + . $LKP_SRC/lib/common.sh : "${mysql_user:=root}" : "${mysql_host:=127.0.0.1}" : "${mysql_port:=3306}" : "${mysql_db:=sysbench_test}" +: "${benchmark_db:=tpcc}" : "${mysql_password:=mysql_password}" create_basedir() { - mkdir -p /data - mkfs.xfs -f /dev/"$data_dev" - mount /dev/"$data_dev" /data + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data - mkdir -p /data/mysql/{data,share,tmp,run,log} + mkdir -p /data/mysql/{data,share,tmp,run,log} - useradd mysql - touch /data/mysql/log/mysql.log - chown -R mysql:mysql /data - chown -R mysql:mysql /data/mysql/log/mysql.log + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log } configure_mysql() { - if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then - echo "mysql is not installed!" && exit 1 - fi + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi - "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" - sleep 10 + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 - "$mysql_server" start - sleep 10 + "$mysql_server" start + sleep 10 - "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" - "$mysql_server" restart - sleep 10 + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + sed -i '/^skip-name-resolve/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 } create_database() { - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" - "$mysql_server" stop - sleep 10 + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $benchmark_db;" + "$mysql_server" stop + sleep 10 } start_mysql() { - "$mysql_server" start & + "$mysql_server" start & wait } main() { - local mysql="/usr/local/mysql/bin/mysql" - local mysqld="/usr/local/mysql/bin/mysqld" - local mysql_server="/usr/local/mysql/support-files/mysql.server" + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" - create_basedir - configure_mysql - create_database - start_mysql + create_basedir + configure_mysql + create_database + start_mysql } main -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/7] tests: add script for benchmarksql test tool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/benchmark-sql | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/benchmark-sql diff --git a/tests/benchmark-sql b/tests/benchmark-sql new file mode 100755 index 000000000..2ec6a7317 --- /dev/null +++ b/tests/benchmark-sql @@ -0,0 +1,17 @@ +#!/bin/sh +# benchmark_pkg_name + +: "${benchmark_pkg_name:=benchmarksql5.0-for-mysql}" + +run_benchmark_mysql() +{ + scripts=(runDatabaseBuild.sh runBenchmark.sh runDatabaseDestroy.sh) + + cd "$HOME/$benchmark_pkg_name/run/" && { + ./runDatabaseBuild.sh props.conf + ./runBenchmark.sh props.conf + ./runDatabaseDestroy.sh props.conf + } +} + +run_benchmark_mysql -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/7] stats: add extract scritp
by Luan Shengde 30 Apr '21

30 Apr '21

add stats script for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- stats/benchmark-sql | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/benchmark-sql diff --git a/stats/benchmark-sql b/stats/benchmark-sql new file mode 100755 index 000000000..d4794b4f4 --- /dev/null +++ b/stats/benchmark-sql @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 450.31 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmTOTAL = 999.71 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session Start = 2021-04-29 10:58:15 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session End = 2021-04-29 11:28:16 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Transaction Count = 30003 + +while (line = STDIN.gets) + case line.chomp! + when /Measured tpmC $NewOrders$ = (\d*\.\d*)/ + puts "Measured_tpmC$NewOrders$: #{$1}" + when /Measured tpmTOTAL = (\d*\.\d*)/ + puts "Measured_tpmTOTAL: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/benchmark-sql.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 jobs/benchmark-sql.yaml diff --git a/jobs/benchmark-sql.yaml b/jobs/benchmark-sql.yaml new file mode 100644 index 000000000..152253c4e --- /dev/null +++ b/jobs/benchmark-sql.yaml @@ -0,0 +1,20 @@ +suite: benchmark-sql + +os: openeuler +os_version: 20.03 +os_mount: initramfs + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +benchmark_db: tpcc +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +benchmark-sql: -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for sysbench-mysql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/sysbench-mysql.yaml | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 000000000..6c6606ea2 --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,33 @@ +suite: sysbench-mysql + +os: openeuler +os_version: 20.03 +os_mount: initramfs + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +benchmark_db: tpcc +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 60 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 20000 + mysql_socket: /data/mysql/run/mysql.sock -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/mail_job_result: optimize mail content
by Lu Weitao 30 Apr '21

30 Apr '21

optimize mail content limit max lines(100) of compare result Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 744b7d5c..7937de66 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -24,10 +24,10 @@ class MailJobResult def compose_mail set_submitter_info - context = get_compare_result(@job) + context = keep_100_lines(get_compare_result(@job)) return nil unless context - subject = "[Compass-CI] job: #{@job_id} compare result" + subject = "[Compass-CI] #{@job['commit_title'] || @job_id} comparsion" signature = "Regards\nCompass-CI\nhttps://gitee.com/openeuler/compass-ci" data = <<~BODY @@ -38,10 +38,17 @@ class MailJobResult Hi, Thanks for your participation in Kunpeng and software ecosystem! - Your Job: #{@job_id} had finished. - Bellow are comparsion result of the base_commit and current_commit: - \tbase_commit: #{@job['base_commit']} - \tcurrent_commit: #{@job['upstream_commit']} + Your job id: #{@job_id} + Bellow are comparsion result of the base commit and your commit: + \tbase commit: #{@job['base_commit']} + \tcommit_link: #{@job['upstream_url']}/commit/#{@job['base_commit']} + + \tyour commit: #{@job['upstream_commit']} + \tbranch: #{@job['upstream_branch']} + \tcommit_link: #{@job['upstream_url']}/commit/#{@job['upstream_commit']} + + compare command: + \tcompare upstream_commit=#{@job['base_commit']} upstream_commit=#{@job['upstream_commit']} --min_samples #{@job['nr_run']} #{context.to_s} \n\n#{signature}" BODY @@ -86,3 +93,7 @@ def get_compare_result(job) m_titles = [base_commit.slice(0, 16), commit_id.slice(0, 24)] compare_matrixes(matrices_list, suite_list, nil, m_titles, options: options) end + +def keep_100_lines(context) + context.split("\n")[0,99].join("\n") +end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_job_result: optimize mail content
by Lu Weitao 30 Apr '21

30 Apr '21

optimize mail content limit max lines(100) of compare result Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 744b7d5c..0bf52b6e 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -24,10 +24,10 @@ class MailJobResult def compose_mail set_submitter_info - context = get_compare_result(@job) + context = keep_100_lines(get_compare_result(@job)) return nil unless context - subject = "[Compass-CI] job: #{@job_id} compare result" + subject = "[Compass-CI] commit: #{@job['upstream_commit']} comparsion result" signature = "Regards\nCompass-CI\nhttps://gitee.com/openeuler/compass-ci" data = <<~BODY @@ -38,10 +38,15 @@ class MailJobResult Hi, Thanks for your participation in Kunpeng and software ecosystem! - Your Job: #{@job_id} had finished. - Bellow are comparsion result of the base_commit and current_commit: - \tbase_commit: #{@job['base_commit']} - \tcurrent_commit: #{@job['upstream_commit']} + Bellow are comparsion result of the base commit and your commit: + \tbase commit: #{@job['base_commit']} + \tcommit_link: #{@job['upstream_url']}/commit/#{@job['base_commit']} + + \tyour commit: #{@job['upstream_commit']} + \tbranch: #{@job['upstream_branch']} + \tcommit_link: #{@job['upstream_url']}/commit/#{@job['upstream_commit']} + + compare upstream_commit=#{@job['base_commit']} upstream_commit=#{@job['upstream_commit']} --min_samples #{@job['nr_run']} #{context.to_s} \n\n#{signature}" BODY @@ -86,3 +91,15 @@ def get_compare_result(job) m_titles = [base_commit.slice(0, 16), commit_id.slice(0, 24)] compare_matrixes(matrices_list, suite_list, nil, m_titles, options: options) end + +def keep_100_lines(context) + num = 0 + new_context = '' + context.each_line do |line| + new_context += line + num += 1 + break if num > 99 + end + + new_context +end -- 2.23.0

2 3

Re: [PATCH lkp-tests 1/6] add job yaml for sysbenc
by Li Ping 30 Apr '21

30 Apr '21

sysbench-mysql Reply-To: In-Reply-To: <20210429133100.902627-1-shdluan(a)163.com> On Thu, Apr 29, 2021 at 09:30:55PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > jobs/sysbench-mysql.yaml | 33 +++++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) > create mode 100644 jobs/sysbench-mysql.yaml > >diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml >new file mode 100644 >index 000000000..6c6606ea2 >--- /dev/null >+++ b/jobs/sysbench-mysql.yaml >@@ -0,0 +1,33 @@ >+suite: sysbench-mysql >+ >+os: openeuler >+os_version: 20.03 >+os_mount: initramfs >+ >+mysql_user: >+mysql_host: localhost >+mysql_port: >+mysql_db: >+benchmark_db: tpcc >+mysql_password: >+data_dev: sdb >+ >+cci-makepkg: >+ benchmark: mysql-8-0-20 >+ >+mysql-8-0-20: >+ >+sysbench-mysql: >+ oltp_test_mode: complex >+ max_requests: 0 >+ mysql_table_engine: innodb >+ oltp_tables_count: 1000 >+ oltp_table_size: 100000 >+ nr_threads: 256 >+ runtime: 60 >+ rand_type: special >+ rand_spec_pct: 100 >+ oltp_read_only: false >+ report_interval: 1 >+ events: 20000 >+ mysql_socket: /data/mysql/run/mysql.sock >-- >2.23.0 >

2 1

Re: [PATCH lkp-tests 2/6] add job yaml for benmarksql
by Li Ping 30 Apr '21

30 Apr '21

benchmark-sql Reply-To: In-Reply-To: <20210429133100.902627-2-shdluan(a)163.com> On Thu, Apr 29, 2021 at 09:30:56PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > jobs/benchmark-sql.yaml | 20 ++++++++++++++++++++ > 1 file changed, 20 insertions(+) > create mode 100644 jobs/benchmark-sql.yaml > >diff --git a/jobs/benchmark-sql.yaml b/jobs/benchmark-sql.yaml >new file mode 100644 >index 000000000..152253c4e >--- /dev/null >+++ b/jobs/benchmark-sql.yaml >@@ -0,0 +1,20 @@ >+suite: benchmark-sql >+ >+os: openeuler >+os_version: 20.03 >+os_mount: initramfs >+ >+mysql_user: >+mysql_host: localhost >+mysql_port: >+mysql_db: >+benchmark_db: tpcc >+mysql_password: >+data_dev: sdb >+ >+cci-makepkg: >+ benchmark: mysql-8-0-20 >+ >+mysql-8-0-20: >+ >+benchmark-sql: >-- >2.23.0 >

2 1

[PATCH lkp-tests] install-iso-sync-rootfs: delete cgz content for firstboot reboot
by Yu Chuan 30 Apr '21

30 Apr '21

[Why] In install-iso-firstboot step, we put the reboot-after-systemd-logind.cgz to rootfs, so we need delete all file in that cgz before we archive the result rootfs. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-sync-rootfs | 1 + tests/install-iso-sync-rootfs | 28 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/distro/depends/install-iso-sync-rootfs b/distro/depends/install-iso-sync-rootfs index 2abb6b833291..41c472eda77e 100644 --- a/distro/depends/install-iso-sync-rootfs +++ b/distro/depends/install-iso-sync-rootfs @@ -1,2 +1,3 @@ nfs-common lvm2 +wget diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index b74c7bed2044..c2b64a2b44d8 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -53,6 +53,31 @@ prepare_exclude_file() done } +delete_firstboot_cgz() +{ + local cgz_url="http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-systemd-logind.cgz" + + local tmp_dir="/tmp/$(basename $cgz_url)_${ROOTFS_SUFFIX}" + rm -rf $tmp_dir && mkdir -p $tmp_dir && cd $tmp_dir + + wget $cgz_url || { + log_info "wget cgz failed, skip delete it: $cgz_url" + return 0 + } + + gzip -dc reboot-after-systemd-logind.cgz | cpio -id + + [ -d "custom_bootstrap" ] || { + log_info "cannot find custom_bootstrap dir after unzip" + return 0 + } + + for i in $(find custom_bootstrap/* -not -type d) + do + rm -f ${lv_rootfs_mount_point}/${i#custom_bootstrap/} + done +} + sync_rootfs_to_nfs_server() { # mount nfs rootfs @@ -70,6 +95,9 @@ sync_rootfs_to_nfs_server() mkdir -p ${lv_rootfs_mount_point} mount /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} ${lv_rootfs_mount_point} || die "mount lv rootfs failed" + # delete cgz for firstboot reboot + delete_firstboot_cgz + # sync rootfs local exclude_file=/tmp/exclude_file_${ROOTFS_SUFFIX} -- 2.23.0

1 0

[PATCH v4 lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 30 Apr '21

30 Apr '21

tests/rpmbuild-pkg: add a new input parameter whether_email [usage]: rpmbuild-pkg: whether_email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 35 +++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 24 ++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..a6e0bc3ba 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,41 @@ Compass-CI " } +rpmbuild_report() +{ + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + $rpmbuild_result + +Regards +Compass-CI +" +} + +rpmbuild_success_content() +{ + rpmbuild_result="Your RPM Package is successfully built. + +You can click the follow link to obtain your RPM Package: +http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${compat_os}/${os_arch}/Packages + +And you can click the following link to view RPM Package build details: +http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_root" + + rpmbuild_report +} + +rpmbuild_failed_content() +{ + rpmbuild_result="Sorry to inform you that RPM Package built failed, you can click the following link to view details. +http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_root" + + rpmbuild_report +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..5ad5caf56 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - whether_email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + rpmbuild_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -29,6 +38,20 @@ from_git() download_upstream_repo } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + +rpmbuild_email() +{ + [ -n "$whether_email" ] && { + author_email=$(git_pretty_format "%ae") + author_name=$(git_pretty_format "%an") + send_email $1 + } +} + from_srpm() { [ -n "$repo_name" ] || die "repo_name is empty" @@ -143,3 +166,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +rpmbuild_email "rpmbuild_success" -- 2.23.0

1 0