November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] doc/job/: add os-os_verison-os_arch.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os-os_verison-os_arch.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/os-os_verison-os_arch.md diff --git a/doc/job/os-os_verison-os_arch.md b/doc/job/os-os_verison-os_arch.md new file mode 100644 index 0000000..9939414 --- /dev/null +++ b/doc/job/os-os_verison-os_arch.md @@ -0,0 +1,16 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch are also need to be specified, + then the machine will run an os according to these three fields. +- The os, os_version, os_arch are the key for users to specify the os related parameters. +- If os, os_version, os_arch are not given by users, it will use the default openeuler os related parameters. +- Here are some examples: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..844cdbb --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +It explicitly tells that `runtime` are the parameters of test script. +For any test case needing paramter(s), need such comment lines and +follow exactly the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

1 1

[PATCH compass-ci 4/4] assign-account: build email message for updating configuration
by Luan Shengde 25 Nov '20

25 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../assign-account/build-update-email.rb | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/assign-account/build-update-email.rb diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb new file mode 100755 index 0000000..40d6b3b --- /dev/null +++ b/container/assign-account/build-update-email.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_update_message(email, account_info, conf_info) + message = if conf_info['enable_login'] + + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account update notice + + Dear user: + + Your account has been update successfully. + You can use the following command to login the server. + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + regards + compass-ci + EMAIL_MESSAGE + else + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account expiration notice + + Dear user: + + your account has expired. + consult the manager for help if you want to continue using the account. + + regards + compass-ci + EMAIL_MESSAGE + end + + return message +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: reattach new dir for the container
by Luan Shengde 25 Nov '20

25 Nov '20

[why] enable execute command 'usermod -s shell' to enable/disable account to login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..793191a 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh +. ../defconfig.sh docker_rm assign-account @@ -11,8 +11,7 @@ cmd=( --restart=always --name=assign-account -d - -v /etc/passwd:/etc/passwd:ro - -v /etc/group:/etc/group:ro + -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw -p 29999:29999 @@ -20,14 +19,3 @@ cmd=( ) "${cmd[@]}" - -cat << EOF -port: 29999 ---- -Client: -./assign-tool.rb email_file - -email subject: "apply ssh account" - -Refer to compass-ci/doc/apply-ssh-account.md for more information. -EOF -- 2.23.0

1 0

[PATCH compass-ci 2/4] assign-account: add new function and optimize
by Luan Shengde 25 Nov '20

25 Nov '20

[why] meet new options for answerback-email meet update configuration function for answerback-email [how] add new function for the new options for answerback-email optimize current function to enable update Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 117 +++++++++++++++---- 1 file changed, 92 insertions(+), 25 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 0e91057..95bec14 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -29,11 +29,14 @@ API: call graph: setup_jumper_account_info - read_account_info - build_account_name + read account + read_account_info + build_account_info + reread_account_info read_jumper_info config_default_yaml config_authorized_key + permit_login_config generate_ssh_key the returned data for setup_jumper_account_info like: @@ -41,12 +44,14 @@ the returned data for setup_jumper_account_info like: "my_login_name" => "login_name", "password" => "password", "jumper_host" => "0.0.0.0", - "jumper_port" => "10000" + "jumper_port" => "10000", + "my_jumper_pubkey" => my_jumper_pubkey } =end require 'fileutils' +require 'yaml' # get jumper and account info class AccountStorage @@ -95,14 +100,21 @@ class AccountStorage end def setup_jumper_account_info - login_name, password = read_account_info + if @data.key?('is_update_account') && @data['is_update_account'] + login_name = @data['my_login_name'] + password = reread_account_info + else + login_name, password = read_account_info + end + jumper_host, jumper_port = read_jumper_info - pub_key = @data['my_ssh_pubkey'][0] - ssh_dir = File.join('/home/', login_name, '.ssh') - config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_authorized_key(login_name) config_default_yaml(login_name) - my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true + permit_login_config(login_name) + if @data.key?('gen_sshkey') && @data['gen_sshkey'] + my_jumper_pubkey = generate_ssh_key(login_name) + end jumper_account_info = { 'my_login_name' => login_name, @@ -115,34 +127,89 @@ class AccountStorage return jumper_account_info end - def generate_ssh_key(login_name, ssh_dir) + def permit_login_config(login_name) + if @data.key?('enable_login') && @data['enable_login'] + %x(usermod -s /usr/bin/zsh #{login_name}) + else + %x(usermod -s /sbin/nologin #{login_name}) + end + end + + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + pub_key_file = File.join(ssh_dir, 'id_rsa.pub') + + return if File.exist?(pub_key_file) && \ + @data['my_ssh_pubkey'].include?(File.read(pub_key_file).strip) + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) - File.read("/home/#{login_name}/.ssh/id_rsa.pub") + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.read("/home/#{login_name}/.ssh/id_rsa.pub").strip end - def config_default_yaml(login_name) + def touch_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') - FileUtils.mkdir_p default_yaml_dir + FileUtils.mkdir_p default_yaml_dir unless File.exist? default_yaml_dir + + default_yaml = File.join(default_yaml_dir, 'account.yaml') + FileUtils.touch default_yaml unless File.exist? default_yaml + default_yaml + end + def config_default_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + + account_yaml = YAML.load_file(default_yaml) || {} # my_email, my_name, my_uuid is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request - File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| - file.puts "my_email: #{@data['my_email']}" - file.puts "my_name: #{@data['my_name']}" - file.puts "my_uuid: #{@data['my_uuid']}" - end - %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + account_yaml['my_email'] = @data['my_email'] + account_yaml['my_name'] = @data['my_name'] + account_yaml['my_uuid'] = @data['my_uuid'] + + f = File.new(default_yaml, 'w') + f.puts account_yaml.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name) + pub_key = @data['my_ssh_pubkey'][0] + + return if pub_key.nil? + return if pub_key.strip.end_with?('account-vm') + + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + + authorized_file = File.join(ssh_dir, 'authorized_keys') + FileUtils.touch authorized_file unless File.exist? authorized_file + + store_pubkey(ssh_dir, login_name, authorized_file, pub_key) end - def config_authorized_key(login_name, pub_key, ssh_dir) - Dir.mkdir ssh_dir, 0o700 - Dir.chdir ssh_dir - f = File.new('authorized_keys', 'w') + def store_pubkey(ssh_dir, login_name, authorized_file, pub_key) + authorized_keys = File.read(authorized_file).split("\n") + + return if authorized_keys.include?(pub_key.strip) + + f = File.new(authorized_file, 'a') f.puts pub_key f.close - File.chmod 0o600, 'authorized_keys' - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.chmod 0o600, authorized_file + end + + def reread_account_info + my_login_name_file = File.join(@account_dir, 'assigned-users', @data['my_login_name']) + + message = "No such assigned account exists: #{my_login_name_file}." + raise message unless File.exist? my_login_name_file + + password = File.readlines(my_login_name_file)[0].chomp + + return password end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will user 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 20 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 108 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..e54adc8b --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,20 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

2 1

[PATCH lkp-tests 1/2] daemon/multi-qemu: fix default parameter format
by Xiao Shenwei 25 Nov '20

25 Nov '20

before: : ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} now: : ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} this parameter will be used by ruby script for ruby command line parameter, if use space like: multi-qemu -q vm-2p8g.taishan200-2280-2s48p-256g--a1, vm-2p8g.aarch64 the vm-2p8g.aarch64 doesn't work correct usage: multi-qemu -q vm-2p8g.taishan200-2280-2s48p-256g--a1,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 7cc625f4..aa2348b4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,7 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

2 1

[PATCH compass-ci] job.cr: delete 2 variables from METHOD_KEYS
by Xu Xijian 25 Nov '20

25 Nov '20

Since linux_modules_initrd and linux_headers_initrd are only used in initramfs, they don't need to be initialized in all jobs, so delete them from METHOD_KEYS. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a9923cc..378fd00 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -97,8 +97,6 @@ class Job docker_image kernel_version linux_vmlinuz_path - linux_modules_initrd - linux_headers_initrd ) macro method_missing(call) @@ -390,9 +388,9 @@ class Job temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{linux_modules_initrd}")}" + "#{JobHelper.service_path(self["linux_modules_initrd"])}" temp_initrds << "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{linux_headers_initrd}")}" + "#{JobHelper.service_path(self["linux_headers_initrd"])}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/bootstrap.sh: fix redirect_stdout_stderr for docker
by Xiao Shenwei 25 Nov '20

25 Nov '20

refer-to: compass-ci: c7c4ba94b3ef00a3f5c91200f85ea2d924726132 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/bootstrap.sh | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ce34a632..46e77e72 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -270,11 +270,7 @@ announce_bootup() redirect_stdout_stderr() { - if is_docker; then - exec > /tmp/stdout - exec 2> /tmp/stderr - return - fi + is_docker && return [ -c /dev/kmsg ] || return has_cmd tail || return -- 2.23.0

1 0

[PATCH compass-ci] provider/docker: forward correct and error logs to docker log collector
by Xiao Shenwei 25 Nov '20

25 Nov '20

[why] $LKP_SRC/lib/bootstrap.sh:271 redirect_stdout_stderr() result file: output stderr stdout these file will based on /tmp/stdout and /tmp/stderr the function will execute: exec > /tmp/stdout exec 2> /tmp/stderr docker log based on /dev/stdout and /dev/stderr, so docker log can't collect the job's output. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..3d04406 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap +# forward correct and error logs to docker log collector +touch /tmp/stderr /tmp/strout +tail -f /tmp/stdout > /dev/stdout & +echo $! >> /tmp/pid-tail-global +tail -f /tmp/stderr > /dev/stderr & +echo $! >> /tmp/pid-tail-global + +sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap 1> /tmp/stdout 2> /tmp/stderr -- 2.23.0

1 0