March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 4/5] src/scheduler: add local mount of kernel_common_params
by Xu Xijian 10 Mar '21

10 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index bb5983f..cbafb9f 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -4,8 +4,11 @@ class Job private def kernel_common_params common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" - return "#{common_params} rw" if "#{self.os_mount}" == "local" - return "#{common_params} rootovl ro" + return "#{common_params} rootovl ro" unless "#{self.os_mount}" == "local" + + use_root_partition = "/dev/mapper/os-#{os}_#{os_arch}_#{src_lv_suffix}" if @hash["src_lv_suffix"]? != nil + save_root_partition = "/dev/mapper/os-#{os}_#{os_arch}_#{boot_lv_suffix}" if @hash["boot_lv_suffix"]? != nil + return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw" end private def kernel_custom_params @@ -20,7 +23,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", - "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", + "local" => "root=#{OS_HTTP_HOST}:#{os_real_path}", # root is just used to temporarily mount a root in initqueue stage when lvm is not ready "container" => "", } -- 2.23.0

2 2

[PATCH compass-ci 3/5] container/dracut-initrd: set sysroot in local mount situation
by Xu Xijian 10 Mar '21

10 Mar '21

When mount local filesystem in os starting, we will create or use a local existed logical volume to replace remote filesystem. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 container/dracut-initrd/bin/set-local-sysroot.sh diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh new file mode 100644 index 0000000..65e6d3e --- /dev/null +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -0,0 +1,78 @@ +#!/bin/sh + +analyse_kernel_cmdline_params() { + rootfs="$(getarg root=)" + rootfs_src=$(echo $"$rootfs" | sed 's/\///') + timestamp="$(basename $root)" + export rootfs_src timestamp +} + +rsync_src_lv() { + local src_lv="$1" + + [ -e "$src_lv" ] && return # /dev/mapper/os-openeuler-aarch64-20210226104900 + + # create logical volume + src_lv_devname="$(basename $src_lv)" # os-openeuler_aarch64_20210226104900 + lvm lvcreate -L 10G --name "${src_lv_devname#os-}" os # openeuler_aarch64_20210226104900 + mke2fs -t ext4 -F "$src_lv" + + # sync nfsroot to $src_lv + mkdir -p /mnt1 && mount -t nfs "$rootfs_src" /mnt1 + mkdir -p /mnt2 && mount "$src_lv" /mnt2 + cp -a /mnt1/. /mnt2/ + umount /mnt1 /mnt2 + + # change permission of "$src_lv" to readonly + lvm lvchange --permission r "$src_lv" +} + +snapshot_boot_lv() { + local src_lv="$1" + local boot_lv="$2" + + [ "$src_lv" == "$boot_lv" ] && return + + lvm lvremove --force "$boot_lv" + boot_lv_devname="$(basename $boot_lv)" + lvm lvcreate --size 10G --name ${boot_lv_devname#os-} --snapshot "$src_lv" +} + +set_sysroot() { + boot_lv="$1" + umount "$NEWROOT" + mount "$boot_lv" "$NEWROOT" +} + +if ! getargbool 0 local; then + exit 0 +fi + +analyse_kernel_cmdline_params + +sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf + +use_root_partition="$(getarg use_root_partition=)" +if [ -n "$use_root_partition" ]; then + src_lv="$use_root_partition" # "$use_root_partition" "$root_$timestamp", root from getarg root= + rsync_src_lv "$src_lv" +else + src_lv="/dev/mapper/os-openeuler_aarch64_$timestamp" # "os-openeuler_aarch64_$timestamp" + [ -e "src_lv" ] || { + echo "warn dracut: FATAL: no src_lv with local mount, reboot" + reboot + } +fi + +save_root_partition="$(getarg save_root_partition=)" +if [ -n "$save_root_partition" ]; then + boot_lv="$save_root_partition" # "os-openeuler_aarch64_$save_partition" +else + boot_lv="/dev/mapper/os-openeuler_aarch64_default" +fi + +snapshot_boot_lv "$src_lv" "$boot_lv" + +set_sysroot "$boot_lv" + +exit 0 -- 2.23.0

2 2

[PATCH compass-ci 2/5] container/dracut-initrd: add handling of pre-mount stage into setup-dracut.sh
by Xu Xijian 10 Mar '21

10 Mar '21

Add the content of set-local-sysroot.sh into the tail of dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 849480d..d447251 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -18,3 +18,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" +pre_mount_last_line="$(eval 'sed -n \$p $pre_mount_file')" +[ "$pre_mount_last_line" = "exit 0" ] && sed -i '$d' "$pre_mount_file" +cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

2 2

[PATCH compass-ci 5/5] src/lib/job.cr: add keywords src_lv_suffix and boot_lv_suffix
by Xu Xijian 10 Mar '21

10 Mar '21

1. src_lv_suffix: to combine into use_root_partition, which specifies a logical volume to be a local source rootfs. 2. boot_lv_suffix: to combine into save_root_partition, whose rootfs is generated from use_root_partition. This rootfs will be mounted when start an os and saved locally after shutdown. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..424dd31 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -89,6 +89,8 @@ class Job docker_image kernel_version linux_vmlinuz_path + src_lv_suffix + boot_lv_suffix ) macro method_missing(call) -- 2.23.0

2 1

[PATCH compass-ci 2/2] container/srv-http: add /srv/initrd/rpmbuild-pkg
by Li Ping 10 Mar '21

10 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start-initrd | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 7c73ea1..495cc25 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -12,6 +12,7 @@ cmd=( -p 11303:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro + -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro -v /etc/localtime:/etc/localtime:ro -d srv-http -- 2.23.0

1 0

[PATCH compass-ci 1/2] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 10 Mar '21

10 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..a7674d4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + tmp_style = nil ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style + tmp_style = @hash[item]? + break if tmp_style end - return nil unless pkg_style + return nil unless tmp_style + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} + pkg_style.merge!(tmp_style.as_h? || Hash(String, JSON::Any).new) - pkg_style = JSON.parse("{}") if pkg_style == nil + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup - - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] providers/multi-qemu: add the function of restarting VMs
by Wu Zhende 10 Mar '21

10 Mar '21

Obtain the VM to be restarted from the mq and restart it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/providers/multi-qemu b/providers/multi-qemu index c729c8e..fb4167a 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -5,6 +5,9 @@ require 'fileutils' require 'optparse' +require 'json' + +require '../lib/mq_client' opt = {} options = OptionParser.new do |opts| @@ -42,6 +45,8 @@ HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['HOSTNAME']}" NR_VM = opt['nr_vm'] || 1 QUEUES = opt['queues'] || "#{ENV['HOSTNAME']}.#{RUBY_PLATFORM.split('-')[0]}" LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def main(hostname) start_time = record_runtime_log(hostname) @@ -99,7 +104,41 @@ def save_pid(arr) f.close end +def reboot_deadline_machine(hostname) + mq_queue = get_mq_queue(hostname) + mq = MQClient.new + queue = mq.queue(mq_queue, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "pkill #{job_id};echo $?" + mq.ack(info) + end +end + +def get_pid(testbox) + testbox =~ /.*-(\d+$)/ + $1 +end + +def get_mq_queue(hostname) + if hostname.include?('.') + hostname =~ /.*\.(.*)/ + else + hostname =~ /(.*)--.*/ + end + $1 || hostname +end + def multiqemu + Process.fork do + reboot_deadline_machine(HOSTNAME) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + pids = [] NR_VM.to_i.times do |i| pid = Process.fork do -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/mq_client.rb: add new class for rabbitmq
by Wu Zhende 10 Mar '21

10 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/mq_client.rb | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 lib/mq_client.rb diff --git a/lib/mq_client.rb b/lib/mq_client.rb new file mode 100755 index 0000000..63618f9 --- /dev/null +++ b/lib/mq_client.rb @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +# client for rabbitmq +class MQClient + def initialize(hostname = 'localhost', port = '5672') + @conn = Bunny.new(hostname: hostname, port: port) + @conn.start + @channel = @conn.create_channel + end + + def queue(queue_name, opts = {}) + @channel.queue(queue_name, opts) + end + + def ack(delivery_info) + @channel.ack(delivery_info.delivery_tag) + end +end -- 2.23.0

1 0

[PATCH compass-ci] lib/mq_client.rb: add new class for rabbitmq
by Wu Zhende 10 Mar '21

10 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/mq_client.rb | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/mq_client.rb diff --git a/lib/mq_client.rb b/lib/mq_client.rb new file mode 100755 index 0000000..412d7e1 --- /dev/null +++ b/lib/mq_client.rb @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "bunny" + +class MQClient + def initialize(hostname = "localhost", port = "5672") + @conn = Bunny.new(hostname: hostname, port: port) + @conn.start + @channel = @conn.create_channel + end + + def queue(queue_name, opts = {}) + @channel.queue(queue_name, opts) + end + + def ack(delivery_info) + @channel.ack(delivery_info.delivery_tag) + end +end -- 2.23.0

1 0

[PATCH compass-ci] container: fix cloning into 'deep-merge.cr' read error
by Cao Xueliang 10 Mar '21

10 Mar '21

The deep-merge repo has deleted in gitlab. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-shards/shard.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/shard.yml b/container/crystal-shards/shard.yml index 44d175d..ca8f72e 100644 --- a/container/crystal-shards/shard.yml +++ b/container/crystal-shards/shard.yml @@ -25,7 +25,7 @@ dependencies: github: icyleaf/any_merge branch: master deep-merge: - gitlab: peterhoeg/deep-merge.cr + github: peterhoeg/deep-merge.cr json_on_steroids: github: anykeyh/json_on_steroids rate_limiter: -- 2.23.0

1 0