November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] container/scheduler-https: raise warning if missing CA
by Wu Zhende 12 Nov '20

12 Nov '20

Not all environments require https scheduler. So start it only if CA certificate exists. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 38c4892..38427b7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -6,6 +6,10 @@ require 'set' require_relative '../defconfig.rb' +warn_msg = 'CA certificate is missing.Ignore if the HTTPS scheduler is not required' +ca_path = '/etc/compass-ci/ca' +raise warn_msg unless File.exist?("#{ca_path}/server.crt") && File.exist?("#{ca_path}/server.key") + names = Set.new %w[ SCHED_HOST SCHED_PORT -- 2.23.0

2 1

[PATCH v3 compass-ci] splitting the cifs services
by Xiao Shenwei 12 Nov '20

12 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: mount /srv/initrd client: mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd client: mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 24 +++++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 36 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 +++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..9590701 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,24 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..43d2b78 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,36 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp +[osimage] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..a9dbc48 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -dt + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

4 6

[PATCH lkp-tests] netperf/PKGBUILD: download new config.guess, config.sub if guess failed
by Lin Jiaxin 12 Nov '20

12 Nov '20

error: ./config.guess: unable to guess system type This script, last modified 2005-08-03, has failed to recognize the operating system you are using. It is advised that you download the most up to date version of the config scripts. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- pkg/netperf/PKGBUILD | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pkg/netperf/PKGBUILD b/pkg/netperf/PKGBUILD index 1b724948..b694a61c 100755 --- a/pkg/netperf/PKGBUILD +++ b/pkg/netperf/PKGBUILD @@ -10,6 +10,13 @@ md5sums=('e0d45b5bca1eee2aef0155de82366202') build() { cd "$srcdir/$pkgname-$pkgname-$pkgver.$pkgrel" + + # the orginal config.guess is too old to guess new system type + # so download latest version if it guess failed + ./config.guess >/dev/null 2>&1 || { + curl -s -o config.guess 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.gues…' + curl -s -o config.sub 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;…' + } # sendfile_tcp_stream() misses assignment of len and it is possible # for random len to trigger the following issue, so fix it. -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sched: format: crystal-format job.cr
by Ren Wen 12 Nov '20

12 Nov '20

Should do a 'crystal-format <filename>' after changing the crystal file. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index d10a1e9..2418c82 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -301,7 +301,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") @@ -429,7 +429,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

1 0

[PATCH compass-ci] sched: simplify logic
by Ren Wen 12 Nov '20

12 Nov '20

has_key?(key) Returns true when key given by key exists, otherwise false. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index d10a1e9..e3d47de 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -289,7 +289,7 @@ class Job "SCHED_PORT"] initialized_keys.each do |key| - if @hash.has_key?(key) == false + unless @hash.has_key?(key) return false end end @@ -301,7 +301,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") @@ -429,7 +429,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

1 1

[PATCH v6 lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 12 Nov '20

12 Nov '20

[why] when the task is failed(state), it need in the waiting state [how] when job2sh.rb defined sh_on_state, it will be invoked by the scheduler. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..3052cf39 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_state = job.sh_on_state sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_state } } -- 2.23.0

2 1

[PATCH v2 compass-ci 3/4] assign-account: generate rsa keys
by Luan Shengde 12 Nov '20

12 Nov '20

generate rsa private/public key and return public key on jumper server [why] register the public key on testbox enable user one click to login in the testbox [how] generate rsa private/public key for user on jumper server return and register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 32 ++++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..b5a2388 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -94,30 +94,36 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + password = 'Use pub_key to login' unless pub_key.nil? + + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name) if @data['my_jumper_pubkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + hostname = File.read('/etc/hostname') + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir -- 2.23.0

2 3

[PATCH v3 compass-ci 1/3] jobfile_operate.cr: reduce if nesting levels
by Xu Xijian 12 Nov '20

12 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 4ec9f96..baab515 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -58,14 +58,14 @@ module Jobfile::Operate def self.parse_one(script_lines, key, val) return false if val.as_h? - if valid_shell_variable?(key) - value = if val.as_a? - shell_escape(val.as_a) - else - shell_escape(val.to_s) - end - script_lines << "\texport #{key}=" + value if value - end + return false unless valid_shell_variable?(key) + + value = if val.as_a? + shell_escape(val.as_a) + else + shell_escape(val.to_s) + end + script_lines << "\texport #{key}=" + value if value end def self.sh_export_top_env(job_content : Hash) -- 2.23.0

2 2

[PATCH v7 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 12 Nov '20

12 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml: PKGBUILD_TAG: 1.0-20 About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..67d055a4 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + PKGBUILD_TAG_FILE="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 4

[PATCH v2 compass-ci] container/https-scheduler: raise warning if missing ca
by Wu Zhende 12 Nov '20

12 Nov '20

Not all environments require https scheduler. So start it only if ca certificate exists. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 38c4892..6d6dda0 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -6,6 +6,9 @@ require 'set' require_relative '../defconfig.rb' +warning = 'CA certificate is missing.Ignore if the HTTPS scheduler is not required' +raise warning unless File.exist?('/etc/compass-ci/ca/server.crt') && File.exist?('/etc/compass-ci/ca/server.key') + names = Set.new %w[ SCHED_HOST SCHED_PORT -- 2.23.0

3 2