- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/srv-http: mount /srv/cci/libvirt-xml to srv-http
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt support user define domain.xml file to start kvm the /srv/cci/libvirt-xml will save user's xml file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index d9f461b..945cde0 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -16,6 +16,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -d srv-http ) -- 2.23.0

1 0

[PATCH compass-ci 1/3] qemu/kvm.sh: fix script will hang when there is no job
by Yu Chuan 18 Jan '21

18 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd When there is no job corresponding to current vm in the remote redis queue, var $initrds will be equal to "". And from `cat --help`, we can know the follow: cat Copy standard input to standard output. So the process will get stuck waiting for input. [How] Function check_option_value() is used to check whether $initrds or $kernel is empty. And $initrds and $kernel are assigned in the function parse_ipxe_script(). So move check_option_value() before set_initrd(). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e9138a7c0..35678fbd44ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -197,8 +197,8 @@ write_logfile parse_ipxe_script -set_options check_option_value +set_options print_message -- 2.23.0

2 6

[PATCH v4 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 18 Jan '21

18 Jan '21

[Why] Unpacked vmlinux into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 33 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..e8c56ba 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -5,6 +5,8 @@ # This file is called by script run running on host # define functions and set host environment variables +. $CCI_SRC/container/qcow2rootfs/bin/common + get_host_aarch() { aarch=$(arch) @@ -133,7 +135,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +157,32 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + cd "$ROOTFS_DIR/boot" + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + + if [ $? -eq 0 ]; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 18 Jan '21

18 Jan '21

before: if execute ./buildall and docker images exist, then ./build after: if execute ./buildall or docker images not exist, then ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..77fe43a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

5 7

[PATCH v2 compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

2 1

[PATCH lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/libvirt-2p16g | 5 +++++ hosts/libvirt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/libvirt-2p16g create mode 100644 hosts/libvirt-2p32g diff --git a/hosts/libvirt-2p16g b/hosts/libvirt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/libvirt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/libvirt-2p32g b/hosts/libvirt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/libvirt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 1

[PATCH v2 compass-ci] sbin/es-find support range query
by Lu Weitao 18 Jan '21

18 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..c144a8b 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,43 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + range_hash[key] = assign_range(value) + end + + range_hash +end + +def assign_range(value) + range_value = {} + value_list = value.split(',') + if value_list.length > 1 + range_value[:gte] = value_list[0] + range_value[:lte] = value_list[1] + else + range_value[:gte] = value + end + + range_value +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +86,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +98,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] docker/run.sh: collect docker serial logs
by Wu Zhende 18 Jan '21

18 Jan '21

Save docker logs to /srv/cci/serial/logs/$hostname. So fluentd can tail this file and send logs to monitoring service. In this way can generate docker's dmsg file Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 4d53db7..2810641 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -7,6 +7,7 @@ : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} +: ${log_dir:="/srv/cci/serial/logs/$hostname"} if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then tbox_group=${BASH_REMATCH[1]} @@ -38,4 +39,4 @@ cmd=( /root/sbin/entrypoint.sh ) -"${cmd[@]}" +"${cmd[@]}" 2>&1 | tee -a "$log_dir" -- 2.23.0

2 1

[PATCH compass-ci] lib/job.cr: enable the account check function
by Wu Zhende 18 Jan '21

18 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..af7cb71 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -390,15 +390,15 @@ class Job private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + account_info = @es.get_account(self["my_email"]) - # raise account_info unless account_info.is_a?(JSON::Any) + raise error_msg unless account_info.is_a?(JSON::Any) - # @account_info = account_info.as_h - @account_info = account_info.as_h if account_info.is_a?(JSON::Any) + @account_info = account_info.as_h - # raise error_msg if @account_info["found"]? == false - # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - # raise error_msg unless self["my_token"] == @account_info["my_token"] + raise error_msg if @account_info["found"]? == false + raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

2 2

[PATCH v3 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 18 Jan '21

18 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 33 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..e8c56ba 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -5,6 +5,8 @@ # This file is called by script run running on host # define functions and set host environment variables +. $CCI_SRC/container/qcow2rootfs/bin/common + get_host_aarch() { aarch=$(arch) @@ -133,7 +135,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +157,32 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + cd "$ROOTFS_DIR/boot" + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + + if [ $? -eq 0 ]; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

3 4

[PATCH compass-ci] lib/job.cr: add comment for function set_sshr_info
by Wu Zhende 16 Jan '21

16 Jan '21

Explain why the key with "ssh_pub_key" is used as the entry of the function Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..03ca5af 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -205,6 +205,8 @@ class Job end private def set_sshr_info + # ssh_pub_key will always be set (maybe empty) by submit, + # if sshd is defined anywhere in the job return unless @hash.has_key?("ssh_pub_key") self["sshr_port"] = ENV["SSHR_PORT"] -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] docker/docker.rb: add docker serial log
by Wu Zhende 16 Jan '21

16 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..e1ab057 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_runtime_log(hostname, start_time: Time.new, is_start: true, hash: {}) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time + end + + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,14 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + start_time = record_runtime_log(hostname, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon/sshd: refactor sshd
by Wei Jihui 16 Jan '21

16 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/sshd | 61 +++++++----------------------------------------- lib/email.sh | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 53 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index e9f38006..7ba471a8 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -10,6 +10,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/http.sh . $PROJECT_DIR/lib/env.sh +. $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 @@ -36,56 +37,6 @@ run_ssh() fi } -data_success() -{ - data="To: $my_email -Subject: [NOTIFY Compass-CI] Machine application successful - -Dear $my_username, - - Your test machine has been provisioned. - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - -Regards -Compass-CI -" -} - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application failed - -Dear $my_username: - - Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. - -Regards -Compass-Ci -" -} - -compose_email() -{ - deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n "$port" ]; then - data_success - else - data_failure - fi -} - set_port() { [ -n "$my_ssh_pubkey" ] || return @@ -103,9 +54,13 @@ set_port() run_email() { [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" + + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + send_email "borrow_success" + else + send_email "borrow_failed" + fi } run_ssh diff --git a/lib/email.sh b/lib/email.sh index d7c2c44a..a0c99c24 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -15,6 +15,71 @@ Compass-CI " } +job_debug_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job debug: $id + +Dear $my_name, + + Your test job is ready to debug and test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_success_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application successful: $id + +Dear $my_name, + + Your test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application failed: $id + +Dear $my_name: + + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-CI +" +} + send_email() { local subject=$1 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: replace IP with domain name
by Wu Fengguang 16 Jan '21

16 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index eca70f196..4785477ec 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,7 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' srv_http_port = job['SRV_HTTP_PORT'] || '11300' url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

3 4

[PATCH compass-ci] sbin/auto_submit: fix invalid lab value
by Liu Yinsi 15 Jan '21

15 Jan '21

lab should set by dev env $lab Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index c121ae0..ecc4d70 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,7 +86,7 @@ end option = { my_name: 'Auto Submit Robot', my_email: 'autosubmit@localhost', - lab: 'autosubmit' + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } # fix verification problems when submit jobs by auto-submit service -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. use qemu-system-ARCH packages to replace qemue in debian, refer: https://packages.debian.org/sid/qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 106 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..07533ce 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,107 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('"---', '').gsub('---"', '') end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] container - libvirt-helper: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/libvirt_app.rb | 37 +++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/libvirt-helper/libvirt_app.rb diff --git a/container/libvirt-helper/libvirt_app.rb b/container/libvirt-helper/libvirt_app.rb new file mode 100755 index 0000000..551dfc4 --- /dev/null +++ b/container/libvirt-helper/libvirt_app.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['LIBVIRT_HELPER_PORT'] +end + +LIBVIRT_PATH = ENV['LIBVIRT_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] container/assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 +++++ container/assist-cci/start | 43 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..9ac31c1 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) + +ASSIST_CCI_PORT = defaults["ASSIST_CCI_PORT"] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +DEFAULT_CCI = '/c/compass-ci' + +defaults['ASSIST_CCI_PORT'] = ASSIST_CCI_PORT +defaults['LIBVIRT_XML_PATH'] = LIBVIRT_XML_PATH +env = docker_env(defaults) + +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/assist-cci: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-cci/Dockerfile diff --git a/container/assist-cci/Dockerfile b/container/assist-cci/Dockerfile new file mode 100644 index 0000000..49007f7 --- /dev/null +++ b/container/assist-cci/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER Ren Wen <15991987063(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install activesupport json yaml sinatra puma -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add package
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. current debian 'qemu' package makes no sense anymore, and it is very unlikely one will need all of them together, so delete it, install qemu-system-ARCH packages, can see:https://packages.debian.org/sid/qemu Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 15 Jan '21

15 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..81e0034 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,16 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + if return_error_line + error_messages_by_error_id << k + else + error_messages_by_error_id += v.to_a + end end end error_messages_by_error_id -- 2.23.0

2 1