- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/docker-pull: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/providers% ./my-docker dc-1g.lys dc-1g~lys Donehttp://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/boot.container/hostname/dc-1g.lys-501708 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos because lacking of judgment before pull image from local repository, if local repository not found images, it will show error message. [how] if no images in local repository, not pull from local repository, just pull from docker hub. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/docker-pull | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sbin/docker-pull b/sbin/docker-pull index 5972913..d5b90dc 100755 --- a/sbin/docker-pull +++ b/sbin/docker-pull @@ -25,9 +25,16 @@ docker_hub() docker pull $image_name } +query_local_repository() +{ + local image_array=($(echo "$image_name"|tr ':' ' ')) + image=$(curl -s -X GET http://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/v2/${image_array[0]}/manifests/${image_array[1]}) + echo $image| grep -q ${image_array[0]} && local_repository +} + main() { - local_repository || docker_hub + query_local_repository || docker_hub } main -- 2.23.0

2 2

[PATCH compass-ci 2/2] sched: fix taskqueue hand over cluster job 409
by Cao Xueliang 18 Nov '20

18 Nov '20

[why] When add a cluster job to taskqueue use host, but we don't change the queue of job to the host. So when finished job we hand over the queue of job to extract queue will get 409 error. [how] Set the queue of job to the host. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/submit_job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8871464..8838e40 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -49,8 +49,8 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| - tbox_group = host.to_s - job_id = add_task(tbox_group, lab) + queue = host.to_s + job_id = add_task(queue, lab) # return when job_id is '0' # 2 Questions: @@ -58,15 +58,16 @@ class Sched # - may consume job before all jobs done return job_messages << { "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", } unless job_id job_ids << job_id # add to job content when multi-test - job["testbox"] = tbox_group - job.update_tbox_group(tbox_group) + job["testbox"] = queue + job["queue"] = queue + job.update_tbox_group(queue) job["node_roles"] = config["roles"].as_a.join(" ") direct_macs = config["macs"].as_a direct_ips = [] of String -- 2.23.0

3 3

[PATCH compass-ci] scheduler: fix: check whether a job is cluster job or not
by Ren Wen 18 Nov '20

18 Nov '20

How to check whether a job is cluster job or not at LKP? should_wait_cluster() { [ -z "$LKP_SERVER" ] && return 1 [ -z "$node_roles" ] && return 1 [ "$cluster" = "cs-localhost" ] && return 1 return 0 } So when you need a cluster job, the value of cluster field should not be "cs-localhost". Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8838e40..1297b57 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -9,10 +9,10 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' + # how to execute a cluster job: + # has a cluster field, and the value is not "cs-localhost" cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + if cluster_file.empty? || cluster_file == "cs-localhost" return submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, -- 2.23.0

2 3

[PATCH v2 lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..afe511ca1 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { - reboot + if [ "${os}" = "archlinux" ]; then + systemctl reboot + else + reboot + fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..5551dc586 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { + [ ${os} == "archlinux" ] && { + systemctl reboot + } + reboot } -- 2.23.0

2 2

[PATCH v3 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec: change existing spec cases input and output directory
by Wei Jihui 17 Nov '20

17 Nov '20

[Why] we will add more spec cases, so it should give each type of cases for different input and output directories. [How] move existing spec case to link_jobs, they all from lkp-tests/jobs. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 0 .../ftq/ftq-10000-2-100000ss-add.yaml | 0 .../ftq/ftq-20-2-6000ss-cache.yaml | 0 ...-68000000-1-65535-set_get-never-never.yaml | 0 spec/submit/redis.yaml | 1 - spec/submit_spec.rb | 27 ++++++++++--------- 7 files changed, 14 insertions(+), 15 deletions(-) delete mode 120000 spec/submit/ftq.yaml rename spec/submit/{ => link_jobs}/ftq/ftq-1000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-10000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-20-2-6000ss-cache.yaml (100%) rename spec/submit/{ => link_jobs}/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml (100%) delete mode 120000 spec/submit/redis.yaml diff --git a/spec/submit/ftq.yaml b/spec/submit/ftq.yaml deleted file mode 120000 index 0ba22f05..00000000 --- a/spec/submit/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/redis.yaml b/spec/submit/redis.yaml deleted file mode 120000 index 196e182b..00000000 --- a/spec/submit/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 69bd58af..67e05749 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -21,19 +21,20 @@ def traverse_file(output_dir) end describe 'submit job spec' do - Dir.glob("#{LKP_SRC}/spec/submit/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/#{job_name}" - it 'save atomic yaml' do - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) + it 'link jobs spec' do + Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| + p yaml_file + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) end end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/sysbench-mysql: adapt to the sysbench test of a specified version
by Zhang Yu 17 Nov '20

17 Nov '20

[why] use the special version of sysbench test mysql required by the customer. so need to use different sysbench script. [how] add parameters to test, and use different sysbench script. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..106ee0b9 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=7200}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) + +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..1e395231 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + time = $1.to_f + + puts "total time: #{time}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..28e039c3 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file [input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms
by Hu Xuejiao 17 Nov '20

17 Nov '20

[output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..df8181bd --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end + -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/4] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 3914f542..44be44b3 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -111,7 +111,8 @@ build_source_pkg() echo "$makepkg" PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + [ "$?" == 0 ] && update_softlink } build_source_pkg -update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v4 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..3914f542 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg() +{ + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..4aaa4fef 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] && create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/4] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 43ba04ad..a3b5e960 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -110,7 +110,8 @@ build_source_pkg(){ echo "$makepkg" PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + [ "$?" == 0 ] && update_softlink } build_source_pkg -update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..43ba04ad 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,12 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg(){ + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..4aaa4fef 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] && create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

2 1

[PATCH v8 lkp-tests 3/3] execute on_fail when exit-code isn't 0
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] When a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..b422a7eb 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,10 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + + source $job_script + [ "$(type -t on_fail)" = "function" ] && $LKP_DEBUG_PREFIX $job_script on_fail + elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

2 1

[PATCH v4 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..08faf4e0 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end -- 2.23.0

2 1

[PATCH v8 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec1a866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## usage + +build email as the following section: allowed email format +send mail to: compass-ci(a)qq.com + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_mail_content + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there has an attachment file to the email file + attachment: + first attachment + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + - no more available account + build_error_email with raised error message + + - send_mail + - build success email + - build failed email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v8 compass-ci 10/11] mail-robot: start file
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..5827e18 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0