- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] scheduler: fix: check whether a job is cluster job or not
by Ren Wen 18 Nov '20

18 Nov '20

How to check whether a job is cluster job or not at LKP? should_wait_cluster() { [ -z "$LKP_SERVER" ] && return 1 [ -z "$node_roles" ] && return 1 [ "$cluster" = "cs-localhost" ] && return 1 return 0 } So when you need a cluster job, the value of cluster field should not be "cs-localhost". Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8838e40..1297b57 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -9,10 +9,10 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' + # how to execute a cluster job: + # has a cluster field, and the value is not "cs-localhost" cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + if cluster_file.empty? || cluster_file == "cs-localhost" return submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, -- 2.23.0

2 3

[PATCH v2 lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..afe511ca1 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { - reboot + if [ "${os}" = "archlinux" ]; then + systemctl reboot + else + reboot + fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..5551dc586 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { + [ ${os} == "archlinux" ] && { + systemctl reboot + } + reboot } -- 2.23.0

2 2

[PATCH v3 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec: change existing spec cases input and output directory
by Wei Jihui 17 Nov '20

17 Nov '20

[Why] we will add more spec cases, so it should give each type of cases for different input and output directories. [How] move existing spec case to link_jobs, they all from lkp-tests/jobs. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 0 .../ftq/ftq-10000-2-100000ss-add.yaml | 0 .../ftq/ftq-20-2-6000ss-cache.yaml | 0 ...-68000000-1-65535-set_get-never-never.yaml | 0 spec/submit/redis.yaml | 1 - spec/submit_spec.rb | 27 ++++++++++--------- 7 files changed, 14 insertions(+), 15 deletions(-) delete mode 120000 spec/submit/ftq.yaml rename spec/submit/{ => link_jobs}/ftq/ftq-1000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-10000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-20-2-6000ss-cache.yaml (100%) rename spec/submit/{ => link_jobs}/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml (100%) delete mode 120000 spec/submit/redis.yaml diff --git a/spec/submit/ftq.yaml b/spec/submit/ftq.yaml deleted file mode 120000 index 0ba22f05..00000000 --- a/spec/submit/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/redis.yaml b/spec/submit/redis.yaml deleted file mode 120000 index 196e182b..00000000 --- a/spec/submit/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 69bd58af..67e05749 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -21,19 +21,20 @@ def traverse_file(output_dir) end describe 'submit job spec' do - Dir.glob("#{LKP_SRC}/spec/submit/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/#{job_name}" - it 'save atomic yaml' do - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) + it 'link jobs spec' do + Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| + p yaml_file + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) end end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/sysbench-mysql: adapt to the sysbench test of a specified version
by Zhang Yu 17 Nov '20

17 Nov '20

[why] use the special version of sysbench test mysql required by the customer. so need to use different sysbench script. [how] add parameters to test, and use different sysbench script. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..106ee0b9 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=7200}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) + +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..1e395231 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + time = $1.to_f + + puts "total time: #{time}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..28e039c3 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file [input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms
by Hu Xuejiao 17 Nov '20

17 Nov '20

[output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..df8181bd --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end + -- 2.23.0

1 0