November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] provide my-docker script for user
by Xiao Shenwei 18 Nov '20

18 Nov '20

Different users can run it concurrently while avoiding hostname collision. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-docker | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100755 providers/my-docker diff --git a/providers/my-docker b/providers/my-docker new file mode 100755 index 0000000..c8420c7 --- /dev/null +++ b/providers/my-docker @@ -0,0 +1,13 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './docker/docker' + +tbox_group = 'dc-1g' +hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" +# specify which queues will be request, use "," to separate more than 2 values +queues = "#{tbox_group}~#{ENV['USER']}" + +main(hostname, queues) -- 2.23.0

4 4

[PATCH compass-ci] sbin/docker-pull: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/providers% ./my-docker dc-1g.lys dc-1g~lys Donehttp://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/boot.container/hostname/dc-1g.lys-501708 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos because lacking of judgment before pull image from local repository, if local repository not found images, it will show error message. [how] if no images in local repository, not pull from local repository, just pull from docker hub. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/docker-pull | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sbin/docker-pull b/sbin/docker-pull index 5972913..d5b90dc 100755 --- a/sbin/docker-pull +++ b/sbin/docker-pull @@ -25,9 +25,16 @@ docker_hub() docker pull $image_name } +query_local_repository() +{ + local image_array=($(echo "$image_name"|tr ':' ' ')) + image=$(curl -s -X GET http://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/v2/${image_array[0]}/manifests/${image_array[1]}) + echo $image| grep -q ${image_array[0]} && local_repository +} + main() { - local_repository || docker_hub + query_local_repository || docker_hub } main -- 2.23.0

2 2

[PATCH compass-ci 2/2] sched: fix taskqueue hand over cluster job 409
by Cao Xueliang 18 Nov '20

18 Nov '20

[why] When add a cluster job to taskqueue use host, but we don't change the queue of job to the host. So when finished job we hand over the queue of job to extract queue will get 409 error. [how] Set the queue of job to the host. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/submit_job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8871464..8838e40 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -49,8 +49,8 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| - tbox_group = host.to_s - job_id = add_task(tbox_group, lab) + queue = host.to_s + job_id = add_task(queue, lab) # return when job_id is '0' # 2 Questions: @@ -58,15 +58,16 @@ class Sched # - may consume job before all jobs done return job_messages << { "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", } unless job_id job_ids << job_id # add to job content when multi-test - job["testbox"] = tbox_group - job.update_tbox_group(tbox_group) + job["testbox"] = queue + job["queue"] = queue + job.update_tbox_group(queue) job["node_roles"] = config["roles"].as_a.join(" ") direct_macs = config["macs"].as_a direct_ips = [] of String -- 2.23.0

3 3

[PATCH compass-ci] scheduler: fix: check whether a job is cluster job or not
by Ren Wen 18 Nov '20

18 Nov '20

How to check whether a job is cluster job or not at LKP? should_wait_cluster() { [ -z "$LKP_SERVER" ] && return 1 [ -z "$node_roles" ] && return 1 [ "$cluster" = "cs-localhost" ] && return 1 return 0 } So when you need a cluster job, the value of cluster field should not be "cs-localhost". Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8838e40..1297b57 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -9,10 +9,10 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' + # how to execute a cluster job: + # has a cluster field, and the value is not "cs-localhost" cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + if cluster_file.empty? || cluster_file == "cs-localhost" return submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, -- 2.23.0

2 3

[PATCH v2 lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..afe511ca1 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { - reboot + if [ "${os}" = "archlinux" ]; then + systemctl reboot + else + reboot + fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..5551dc586 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { + [ ${os} == "archlinux" ] && { + systemctl reboot + } + reboot } -- 2.23.0

2 2

[PATCH v3 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec: change existing spec cases input and output directory
by Wei Jihui 17 Nov '20

17 Nov '20

[Why] we will add more spec cases, so it should give each type of cases for different input and output directories. [How] move existing spec case to link_jobs, they all from lkp-tests/jobs. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 0 .../ftq/ftq-10000-2-100000ss-add.yaml | 0 .../ftq/ftq-20-2-6000ss-cache.yaml | 0 ...-68000000-1-65535-set_get-never-never.yaml | 0 spec/submit/redis.yaml | 1 - spec/submit_spec.rb | 27 ++++++++++--------- 7 files changed, 14 insertions(+), 15 deletions(-) delete mode 120000 spec/submit/ftq.yaml rename spec/submit/{ => link_jobs}/ftq/ftq-1000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-10000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-20-2-6000ss-cache.yaml (100%) rename spec/submit/{ => link_jobs}/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml (100%) delete mode 120000 spec/submit/redis.yaml diff --git a/spec/submit/ftq.yaml b/spec/submit/ftq.yaml deleted file mode 120000 index 0ba22f05..00000000 --- a/spec/submit/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/redis.yaml b/spec/submit/redis.yaml deleted file mode 120000 index 196e182b..00000000 --- a/spec/submit/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 69bd58af..67e05749 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -21,19 +21,20 @@ def traverse_file(output_dir) end describe 'submit job spec' do - Dir.glob("#{LKP_SRC}/spec/submit/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/#{job_name}" - it 'save atomic yaml' do - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) + it 'link jobs spec' do + Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| + p yaml_file + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) end end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/sysbench-mysql: adapt to the sysbench test of a specified version
by Zhang Yu 17 Nov '20

17 Nov '20

[why] use the special version of sysbench test mysql required by the customer. so need to use different sysbench script. [how] add parameters to test, and use different sysbench script. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..106ee0b9 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=7200}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) + +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql -- 2.23.0

1 0