February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/rpmbuild-pkg: various cleanup
by Wu Fengguang 20 Feb '21

20 Feb '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- tests/rpmbuild-pkg | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 781d04a0d..9f455f0d0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,8 +5,8 @@ . $LKP_SRC/lib/upload.sh [ -n "$rpm_repo" ] || die "rpm_repo is empty" -benchmark=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" +package_name=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { @@ -16,15 +16,15 @@ init_workspace() get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" } download_rpm_repo() { - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in ${filelist[*]} + for pkgfile in "${filelist[@]}" do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" @@ -34,20 +34,29 @@ download_rpm_repo() build_rpm() { - spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` + local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec + + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + # Install build depends - yum-builddep -y $spec_dir + yum-builddep -y $spec_file || exit # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_dir + spectool -g -R $spec_file || exit # Building rpm or srpm packages - rpmbuild -ba $spec_dir + rpmbuild -ba $spec_file || exit +} + +show_rpm_files() +{ + find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" + find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" - for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + local file + for file in $(show_rpm_files) do upload_one_curl ${file} ${sync_dest} done -- 2.23.0

1 0

[PATCH v2 compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..5a9bd2ddadec 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/install/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..334f5c2c4a69 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: add a test ping condition to network_ok
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] In physical machine, there will be a delay between the start of network card and the availability of the network. So we add a test ping condition to network_ok. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 06aa09bf65b2..38965ff60755 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -85,14 +85,19 @@ net_devices_link() done } +test_ping() +{ + ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 +} + network_ok() { local i for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && test_ping && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done is_clearlinux && { -- 2.23.0

2 7

[PATCH compass-ci] lib/git_mirror.rb: change the way to load DEFAULTS file
by Li Yuanchao 19 Feb '21

19 Feb '21

Before: DEFAULTS files are individual. After: Load DEFAULTS file and merge it with the parent directory DEFAULTS files. So the outer DEFAULTS files would keep common things for the inner ones. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 39b1619..346352b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -130,6 +130,7 @@ class MirrorMain defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) end def traverse_repodir(repodir) @@ -224,7 +225,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo) + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -449,15 +450,17 @@ end # main thread class MirrorMain - def merge_defaults(git_repo) - defaults_key = File.dirname(git_repo) + def merge_defaults(object_key, object) + return object if object_key == 'default' + + defaults_key = File.dirname(object_key) while defaults_key != '.' - return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(object) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + return @defaults['default'].merge(object) if @defaults['default'] - return @git_info[git_repo] + return object end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] lkp-bootstrap: delete unnecessary exit
by Xu Xijian 19 Feb '21

19 Feb '21

The exit cmd after reboot is unnecessary, delete it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- rootfs/addon/etc/init.d/lkp-bootstrap | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/addon/etc/init.d/lkp-bootstrap b/rootfs/addon/etc/init.d/lkp-bootstrap index f66ec4eb5..091b29890 100755 --- a/rootfs/addon/etc/init.d/lkp-bootstrap +++ b/rootfs/addon/etc/init.d/lkp-bootstrap @@ -24,7 +24,6 @@ if [ -s /lkp/run/lkp-bootstrap.pid ]; then # the previous lkp-bootstrap init.d script failed to restart the machine echo "LKP bootstrap: first run may failed, retry rebooting" reboot - exit fi read_kernel_cmdline_vars() -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 19 Feb '21

19 Feb '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] src/scheduler/kernel_params.cr: add noserverino option to cifs mount
by Xu Xijian 19 Feb '21

19 Feb '21

[Log] [ 7.507105] CIFS VFS: Autodisabling the use of server inode numbers on \\172.168.131.113\os. This server doesn't seem to support them properly. Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 6fbe405..bb5983f 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,7 +18,7 @@ class Job fs2root = { "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] doc/support-os-os_arch-os_version.md: add 20.03-SP1
by Xu Xijian 19 Feb '21

19 Feb '21

1. add 20.03-SP1 version to this doc, it's already supported 2. adjust some lines to be aligned Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..f91e91b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,24 @@ ``` os os_arch os_version supported-os_mount openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, ,initramfs - 20.09 cifs/nfs, ,initramfs + 20.03 cifs/nfs, ,initramfs + 20.03-SP1 cifs/nfs, + 20.09 cifs/nfs, ,initramfs mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, + 20.03-SP1 cifs/nfs, 20.09 cifs/nfs, centos aarch64 7 container, - 7.6.1810 cifs/nfs, ,initramfs - 7.8.2003 cifs/nfs, ,initramfs + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, ,initramfs +debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: change the rpm_repo field to level-1 parameter
by Li Ping 19 Feb '21

19 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 19fd5ec28..53e145b3b 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional +rpm_repo: rpmbuild-pkg: - rpm_repo: -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: fix no such file error
by Li Ping 19 Feb '21

19 Feb '21

[why]: grep: 2ping.spec: No such file or directory sed: no input files --- tests/rpmbuild-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 156ca0f63..781d04a0d 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -35,7 +35,7 @@ download_rpm_repo() build_rpm() { spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` # Install build depends yum-builddep -y $spec_dir # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 94 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..226251a8bace --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,94 @@ +# Summary +========= + +`os_mount` defines the mount method of testbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + - optional {kernel_append_root}: + - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. + and you can use the root partition data of this job in the futher job by {use_root_partition}. + - use_root_partition : give an iconic string, then the root partition data generated by the + pervious job which has the {save_root_partition} field will be used. + - root_partition_size: specify the size of root partition, *default is 10G*. + + - tips: + - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. + - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + +2. scheduler return the ipxe_str to testbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. dracut step of boot + ``` + if have {use_root_partition}; then + base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 + + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + + if {save_root_partition} == {use_root_partition}; then + boot from /dev/mappaer/os-{boot_lv_name} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + else + boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + boot from /dev/mappaer/os-{boot_lv_name} + fi + + else: + base_lv_name=openeuler_aarch64_20.03_{timestamp} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + # create logical volume + lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + + # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly + lvchange -p r /dev/mapper/os-{base_lv_name} + } + + boot_lv_name=openeuler_aarch64_20.03 + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + fi + ``` + +4. execute the job -- 2.23.0

2 2

[PATCH compass-ci] sched: add rpmbuild-pkg upload dir
by Cao Xueliang 19 Feb '21

19 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b724b07..1afc22f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -154,9 +154,9 @@ class Job set_submit_date() set_pp_params() set_rootfs() + set_os_mount() set_result_root() set_result_service() - set_os_mount() set_depends_initrd() set_kernel() set_initrds_uri() @@ -247,7 +247,7 @@ class Job def get_pkg_common_dir pkg_style = nil - ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| pkg_style = @hash[item]? break if pkg_style end @@ -276,6 +276,8 @@ class Job package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" + elsif @hash["rpmbuild-pkg"]? + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: optimize the test_ping logic
by Yu Chuan 19 Feb '21

19 Feb '21

[Why] 1. '-W' => '-w' -w <deadline> reply wait <deadline> in seconds -W <timeout> time to wait for response deadline(-w) is more appropriate than timeout(-W). 2. delete 'return 1' If ping does not receive any reply packets at all it will exit with code 1. If a packet count and deadline are both specified, and fewer than count packets are received by the time the deadline has arrived, it will also exit with code 1. On other error it exits with code 2. Otherwise it exits with code 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 38965ff60755..76f07e6a7669 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,7 +87,7 @@ net_devices_link() test_ping() { - ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 + ping -c 1 -w 1 $LKP_SERVER > /dev/null } network_ok() -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/rpmbuild-pkg: add depend pkg
by Wei Jihui 19 Feb '21

19 Feb '21

[Error] yum-builddep: command not found [How] %yum provides yum-builddep dnf-plugins-core-4.0.11-5.oe1.noarch : Core Plugins for DNF Repo : @System Matched from: Filename : /usr/bin/yum-builddep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/rpmbuild-pkg b/distro/depends/rpmbuild-pkg index 6f00a6a2..50fd9b75 100644 --- a/distro/depends/rpmbuild-pkg +++ b/distro/depends/rpmbuild-pkg @@ -1,2 +1,3 @@ build-essential rpmdevtools +dnf-plugins-core -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..18d165ca1e08 --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a list of disks, these disks will be combined into one volume group(os), +this volume group will have multiple logical volume, storing multiple rootfs, each of these +logical volume corresponds to a rootfs, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume default size: 10G + +4. more info can refer os_mount.md -- 2.23.0

2 2

[PATCH compass-ci 2/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 19 Feb '21

19 Feb '21

[why] for forwarded email from security admins, the email content will contain my_name/my_email(s) waiting to be assigned an account. a forwarded email may contain multi my_name/my_email(s), just loop them. add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 49 ++++++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..bc340ad 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -54,6 +54,8 @@ class ApplyAccount @mail_content = mail_content @es_host = @send_mail_host @es_port = ES_PORT + # email address file for security admins. + @sec_admins = YAML.load_file '/etc/compass-ci/conf/sec_admins.yaml' @my_info = {} end @@ -63,11 +65,31 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, we will get the email/name from the mail_content + # all emails from security admins will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name_1: my_email_1 + # my_name_2: my_email_2 + # --- + # the forwarded email allowed to contain multi emails/names + # in this case, we will loop them + if @sec_admins.include? @my_info['my_email'] + users = forward_users + users.each do |user| + @my_info.clear + next if user.match?(/---/) + + @my_info['my_name'], @my_info['my_email'] = user.split(/:\s+/) + @my_info['my_ssh_pubkey'] = [] + applying_account + sleep 10 + end + else + parse_mail_content + applying_account + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +97,17 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + forward_email_content.extract_users + end + + def applying_account + apply_my_account + store_account_info + send_mail('') + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +126,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +137,9 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + + # lab will also be stored to ES. + @my_info['lab'] = ENV['lab'] if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +147,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end -- 2.23.0

2 7

[PATCH compass-ci 1/4] container/mail-robot: optimize regular express
by Luan Shengde 19 Feb '21

19 Feb '21

[why] the mail robot will enable forwarded emails from security admins. the mail subject may 'apply account', 'Fw: apply account', .etc. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb index a809eae..c3bc943 100755 --- a/container/mail-robot/mail-robot.rb +++ b/container/mail-robot/mail-robot.rb @@ -32,7 +32,7 @@ def handle_new_email(mail_file, mail_drafts) end def apply_account(mail_content) - return unless mail_content.subject == 'apply account' + return unless mail_content.subject.match?(/apply account/i) assign_uuid = ApplyAccount.new(mail_content) assign_uuid.check_to_send_account -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 95 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 85 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bfb0c2f11 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,90 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' | grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + case a['msg'].chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + + # No such command + when /No such file or directory: b (.+) :/ + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + + # The URL of the RPM is invalid + when /Failure downloading (.+), / + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + + # Failed to download the yum repodata + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + + # The command runing error + when /non-zero return code/ + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{ansible_stats_error}" + + # Some role in not sipported this system + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + case items.chomp + # Failed to install some of the specified packages. + when /No package (.+) available/ + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + end + + next if a['message'].nil? + + case a['message'].chomp + + # Could not find or access ansible yaml + when /Could not find or access (.+) / + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

2 1

[PATCH v2 compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority one would be pop first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 18 Feb '21

18 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..ef00acb1bc0c 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/kickstart/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..ecd783a22dd2 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/kickstart/ks:/srv/os/kickstart/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

2 6

[PATCH v2 lkp-tests] lib/job.rb: delete read "/etc/compass-ci/defaults/*.yaml"
by Cao Xueliang 18 Feb '21

18 Feb '21

The compass-ci/scheduler will read the yamls, so, delete it in lkp-tests Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..37c538753 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -418,8 +418,7 @@ class Job def load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" - Dir.glob(['/etc/compass-ci/defaults/*.yaml', - "#{self_config_path}/defaults/*.yaml"]).each do |file| + Dir.glob(["#{self_config_path}/defaults/*.yaml"]).each do |file| load_one_defaults(file, @job) end -- 2.23.0

2 2

[PATCH lkp-tests] daemon: add nginx-sever
by Wei Jihui 18 Feb '21

18 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/nginx-server | 173 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100755 daemon/nginx-server diff --git a/daemon/nginx-server b/daemon/nginx-server new file mode 100755 index 00000000..a7dd1033 --- /dev/null +++ b/daemon/nginx-server @@ -0,0 +1,173 @@ +#!/bin/sh +# - nics +# - nics_ips +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/common.sh + +# To manually check CPU usage when clients sent http/https requests, please use htop tool. +# htop tool download url: http://rpmfind.net/linux/epel/8/Everything/aarch64/Packages/h/htop-2.2.0-6.… +# For send http request, can use httpress tool. +# httpress tool download url: http://rpmfind.net/linux/epel/7/aarch64/Packages/h/httpress-1.1.0-2.el7.aar… + +: "${protocol:=https}" # http or https +: "${alive_type:=short}" # short or long + +nics=(${nics//,/ }) +nics_ips=(${nics_ips//,/ }) +[ -n "$BENCHMARK_ROOT" ] || BENCHMARK_ROOT=/lkp/benchmarks + +check_nginx_install() +{ + if [ -x "$nginx_path" ]; then + pkill -f "$nginx_path" + else + echo "nginx is not installed" >&2 && exit 1 + fi + + if [ ! -f "$conf_dir/$protocol/basic.conf" ]; then + echo "Could not find nginx conf" >&2 && exit 1 + fi +} + +prepare() +{ + setenforce 0 + ulimit -SHn 409600 + export LD_PRELOAD=/usr/lib64/libtcmalloc.so.4 + + # require by $LKP_SRC/bin/set_nic_irq_affinity + pkill -9 irqbalance +} + +bind_core_net_irp() +{ + # binding cores are used to process NIC interrupts, each corresponds to a NIC. + # the interrupt cores in each CPU are bound only to the NIC that belong to the CPU. + # the CPU cores are bound by node. + + ip addr add "$nic_ip"/24 dev "$nic" + ip link set "$nic" up + + ethtool -L $nic combined $bind_core_nums + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic +} + +update_nginx_conf() +{ + update_key=$1 + update_value="$(eval echo '$'$update_key)" + sed -i "s|{{$update_key}}|$update_value|" "$conf_path" +} + +start_nginx() +{ + local ssl_certificate_path="$conf_dir/$protocol/server_2048.crt" + local ssl_certificate_key_path="$conf_dir/$protocol/server_2048.key" + local conf_path="$conf_dir/$protocol.$alive_type.$core_range.conf" + cp -f "$conf_dir/$protocol/basic.conf" "$conf_path" + source "$conf_dir/$protocol/$alive_type/$core_range" + + update_nginx_conf html_index + update_nginx_conf listen_port + update_nginx_conf worker_processes + update_nginx_conf worker_cpu_affinity + [ "$protocol" == 'https' ] && { + update_nginx_conf ssl_certificate + update_nginx_conf ssl_certificate_key + } + + "$nginx_path" -c "$conf_path" -p "$BENCHMARK_ROOT/nginx-server/" +} + +assign_core_ranges() +{ + if [ "$protocol" == 'http' ]; then + case "$core_nums" in + "4") + if [ "$alive_type" == 'long' ]; then + core_ranges='3' + else + core_ranges='2-3' + fi + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='2-3 6-7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47' + else + core_ranges='12-17 18-23 36-41 42-47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47 66-71 90-95' + else + core_ranges='0-15 16-31 48-63 64-79' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + else + case "$core_nums" in + "4") core_ranges='3' + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47' + else + core_ranges='23 47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47 68-71 92-95' + else + core_ranges='46-47 94-95' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + fi +} + +main() +{ + local core_ranges + local nginx_path="$BENCHMARK_ROOT/nginx-server/sbin/nginx" + local conf_dir="$BENCHMARK_ROOT/nginx-server/nginx_conf" + + check_nginx_install + prepare + assign_core_ranges + + index=0 + for core_range in $core_ranges + do + local nic=${nics[$index]} + local nic_ip=${nics_ips[$index]} + local bind_core_nums=$(cpu_list_num $core_range) + bind_core_net_irp + start_nginx + index=$((index+1)) + done +} + +main -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare: fix No Method Error
by Lu Weitao 18 Feb '21

18 Feb '21

[Why] use: compare group_id=fio-basic-openeuler-acr-1301864-1,fio-basic-openeuler-acr-1301864-2,fio-basic-openeuler-acr-1301864-3 -d pp.acr3.acr error message: Traceback (most recent call last): 2: from /home/wangyong/compass-ci/sbin/compare:72:in `<main>' 1: from /home/wangyong/compass-ci/lib/compare.rb:66:in `compare_group' /home/lwt/compass-ci/lib/compare_matrixes.rb:400:in `compare_group_matrices': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++++ lib/matrix2.rb | 2 ++ 2 files changed, 6 insertions(+) diff --git a/lib/compare.rb b/lib/compare.rb index 21e24df..7c182c4 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -63,6 +63,10 @@ def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + unless groups_matrices + warn 'Empty group matrices!' + exit + end compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ab92986..5e0ad21 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -128,6 +128,8 @@ def combine_group_query_data(job_list, dims) end get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end + return nil if groups.empty? + return groups, suites_hash, latest_jobs_hash end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/dev-env: add the local-testbox-env.yaml
by Cao Xueliang 18 Feb '21

18 Feb '21

The scheduler will read local-testbox-env.yaml, then add the content to the job yaml. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..31a12db 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -59,6 +59,23 @@ ES_PORT: $ES_PORT SRV_HTTP_PORT: $SRV_HTTP_PORT EOF +cat > /etc/compass-ci/defaults/local-testbox-env.yaml <<-EOF +LKP_SERVER: $server_ip +GIT_SERVER: $GIT_SERVER +INITRD_HTTP_HOST: $initrd_http_host +INITRD_HTTP_PORT: $initrd_http_port +GITCACHE_HOST: $gitcache_host +GITCACHE_PORT: $gitcache_port +SEND_MAIL_HOST: $server_ip +SEND_MAIL_PORT: $SEND_MAIL_PORT +ASSIST_RESULT_HOST: $server_ip +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $server_ip +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT +SRV_HTTP_HOST: $server_ip +SRV_HTTP_PORT: $SRV_HTTP_PORT +EOF + cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0