December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..a1dbe77 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load_file(File.join('/srv', "#{result_root}", 'job.yaml')) +end -- 2.23.0

1 0

[PATCH compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 28 Dec '20

28 Dec '20

in x86_64 debian machine, UEFI support for qemu is ovmf/OVMF.fd, different arch use different UEFI support. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/nginx-server: spelling error
by Wei Jihui 28 Dec '20

28 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/depends/nginx-server b/distro/depends/nginx-server index da35d5f6..85e9cc2e 100644 --- a/distro/depends/nginx-server +++ b/distro/depends/nginx-server @@ -1,6 +1,6 @@ gcc glibc -ethool +ethtool libssl-dev zlib1g-dev libpcre3-dev -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: adapt kvm.sh for x86_64 and riscv64
by Liu Yinsi 28 Dec '20

28 Dec '20

put same options in public_options, different options in individual_options, merge options according different arch. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272042 /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-x86_64/z9.272046 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-riscv64/z9.272055 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-x86_64/z9.272057 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 78 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 48bacab..83b9606 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -20,8 +20,19 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-system-x86_64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [ -n "$(echo $kernel | grep "riscv64")" ] && qemu=qemu-system-riscv64 } write_logfile() @@ -31,6 +42,17 @@ write_logfile() cat $ipxe_script >> ${log_file} } +options_value() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + + helper=/usr/libexec/qemu-bridge-helper + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper + + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + get_rootfs() { append= @@ -78,18 +100,14 @@ print_message() sleep 5 } -kvm_command() +public_options() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -98,6 +116,46 @@ kvm_command() ) } +individual_options() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) +} + +merge_options() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") + ;; + esac +} + run_qemu() { "${kvm[@]}" --append "${append}" @@ -110,8 +168,12 @@ get_rootfs check_rootfs qemu_command +options_value print_message -kvm_command +public_options +individual_options +merge_options + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 28 Dec '20

28 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 28 Dec '20

28 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..48741ec 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -15,7 +15,7 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git cd /c/compass-ci || exit - [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" ln -s compass-ci /c/cci } @@ -24,9 +24,12 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/lkp-tests.git cd /c/lkp-tests || exit - [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" + [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" } -- 2.23.0

1 0

[PATCH compass-ci] os/debian: add apt-get update
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the url for downloading the package is displayed as 404 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/0-package/os/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 6a58a0c..c45e8c2 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -22,4 +22,5 @@ pkgs=( hostname ) +apt-get update apt-get install -y --fix-missing "${pkgs[@]}" -- 2.23.0

1 0