- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 6/6] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 30 +++++++------------ 2 files changed, 11 insertions(+), 23 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..94becdf 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,18 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

2 2

[PATCH v4 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 29 Dec '20

29 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] assign-account: add 'lab' to default yaml
by Luan Shengde 29 Dec '20

29 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 2

[PATCH v4 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..97ad757 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..985fdea 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -31,6 +31,8 @@ end options = {} options[:nr_jobs] = 10 +options[:desc_keyword] = nil + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +60,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/5-build: adapt for x86_64 machine
by Liu Yinsi 29 Dec '20

29 Dec '20

build ipxe according different arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/ipxe | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sparrow/5-build/ipxe b/sparrow/5-build/ipxe index a66e298..50a2af9 100755 --- a/sparrow/5-build/ipxe +++ b/sparrow/5-build/ipxe @@ -15,11 +15,25 @@ cd ipxe/src || exit cpus=$(nproc) -make ARCH=arm64 bin-arm64-efi/ipxe.efi -make ARCH=arm64 bin-arm64-efi/snponly.efi -make ARCH=arm64 bin-arm64-efi/snp.efi -make CONFIG=rpi bin-arm64-efi/rpi.efi - -mkdir -p /tftpboot/ipxe/bin-arm64-efi/ -cp -a bin-arm64-efi/*.efi /tftpboot/ipxe/bin-arm64-efi/ -chmod 664 /tftpboot/ipxe/bin-arm64-efi/*.efi +build_ipxe() +{ + make ARCH=$1 bin-$1-efi/ipxe.efi + make ARCH=$1 bin-$1-efi/snponly.efi + make ARCH=$1 bin-$1-efi/snp.efi + make CONFIG=rpi bin-$1-efi/rpi.efi + + mkdir -p /tftpboot/ipxe/bin-$1-efi/ + cp -a bin-$1-efi/*.efi /tftpboot/ipxe/bin-$1-efi/ + chmod 664 /tftpboot/ipxe/bin-$1-efi/*.efi +} + +ARCH=$(arch) + +case "$ARCH" in + "aarch64") + build_ipxe arm64 + ;; + "x86_64") + build_ipxe x86_64 + ;; +esac -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..47ef92f --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER caoxl78320(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..a1dbe77 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load_file(File.join('/srv', "#{result_root}", 'job.yaml')) +end -- 2.23.0

1 0

[PATCH compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 28 Dec '20

28 Dec '20

in x86_64 debian machine, UEFI support for qemu is ovmf/OVMF.fd, different arch use different UEFI support. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/nginx-server: spelling error
by Wei Jihui 28 Dec '20

28 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/depends/nginx-server b/distro/depends/nginx-server index da35d5f6..85e9cc2e 100644 --- a/distro/depends/nginx-server +++ b/distro/depends/nginx-server @@ -1,6 +1,6 @@ gcc glibc -ethool +ethtool libssl-dev zlib1g-dev libpcre3-dev -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: adapt kvm.sh for x86_64 and riscv64
by Liu Yinsi 28 Dec '20

28 Dec '20

put same options in public_options, different options in individual_options, merge options according different arch. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272042 /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-x86_64/z9.272046 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-riscv64/z9.272055 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-x86_64/z9.272057 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 78 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 48bacab..83b9606 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -20,8 +20,19 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-system-x86_64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [ -n "$(echo $kernel | grep "riscv64")" ] && qemu=qemu-system-riscv64 } write_logfile() @@ -31,6 +42,17 @@ write_logfile() cat $ipxe_script >> ${log_file} } +options_value() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + + helper=/usr/libexec/qemu-bridge-helper + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper + + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + get_rootfs() { append= @@ -78,18 +100,14 @@ print_message() sleep 5 } -kvm_command() +public_options() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -98,6 +116,46 @@ kvm_command() ) } +individual_options() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) +} + +merge_options() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") + ;; + esac +} + run_qemu() { "${kvm[@]}" --append "${append}" @@ -110,8 +168,12 @@ get_rootfs check_rootfs qemu_command +options_value print_message -kvm_command +public_options +individual_options +merge_options + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 28 Dec '20

28 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 28 Dec '20

28 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..48741ec 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -15,7 +15,7 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git cd /c/compass-ci || exit - [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" ln -s compass-ci /c/cci } @@ -24,9 +24,12 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/lkp-tests.git cd /c/lkp-tests || exit - [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" + [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" } -- 2.23.0

1 0

[PATCH compass-ci] os/debian: add apt-get update
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the url for downloading the package is displayed as 404 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/0-package/os/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 6a58a0c..c45e8c2 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -22,4 +22,5 @@ pkgs=( hostname ) +apt-get update apt-get install -y --fix-missing "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 28 Dec '20

28 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 1

[PATCH compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 1

[PATCH compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 1