- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/dracut-initrd: set yes for choices when pvcreate and vgcreate
by Xu Xijian 13 Apr '21

13 Apr '21

[why] When create physical volume, some choices need to be made, we should set it as yes(-y) to keep going instead of skipping [errlog] [ 260.644037] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): lvm pvcreate /dev/sda [ 260.664078] dracut-pre-mount[4008]: WARNING: ext4 signature detected on /dev/sda at offset 1080. Wipe it? [y/n]: [n] [ 260.684036] dracut-pre-mount[4008]: Aborted wiping of ext4. [ 260.700034] dracut-pre-mount[4008]: 1 existing signature left on the device. [ 260.716076] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): reboot Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 2a07270..9e83742 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -29,13 +29,13 @@ sync_src_lv() { # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot } } -- 2.23.0

1 0

[PATCH compass-ci] es-jobs: reuse results of other programs
by Wu Zhende 13 Apr '21

13 Apr '21

[Why] es-jobs's result may need use in other programs. But it's hard to get it right now. [How] use: es_jobs.generate_result can get the hash result Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 6 +++++- sbin/es-jobs | 1 + 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index b0d0fb2..25e54b4 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -168,6 +168,10 @@ class ESJobs end def output + output_yaml('', @result) + end + + def generate_result if @jobs.empty? puts "No query result is found: #{@es_query}" return @@ -175,6 +179,6 @@ class ESJobs @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h - output_yaml('', @result) + @result end end diff --git a/sbin/es-jobs b/sbin/es-jobs index ff0d540..6d17d6b 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -52,4 +52,5 @@ items = parse_argv raise 'Please enter a query' if items.empty? es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.generate_result es_jobs.output -- 2.23.0

1 0

[PATCH compass-ci] lib/sched.cr: change testbox's queues
by Wu Zhende 13 Apr '21

13 Apr '21

[Before] ["sched/dc-8g.aarch64/ready"] [After] ["dc-8g.aarch64"] [Why] "sched/.*/ready" is a fixed format. The length of each column on kibana is limited, and removing the fixed format can also highlinght the point. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..35315ce 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -158,6 +158,16 @@ class Sched spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end + # input: ["sched/dc-8g.aarch64/ready"] + # output: ["dc-8g.aarch64"] + def fetch_queues(queues) + new_queues = [] of String + queues.each do |queue| + new_queues << queue.split("/")[1] + end + new_queues + end + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline @@ -172,7 +182,7 @@ class Sched end type = get_type(testbox) - queues = JSON.parse(queues.to_json) + queues = JSON.parse(fetch_queues(queues).to_json) hash = { "job_id" => job_id, "state" => state, -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 13 Apr '21

13 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

2 4

[PATCH compass-ci] lib/git_mirror.rb: retry es query
by Li Yuanchao 13 Apr '21

13 Apr '21

[errmsg] git-mirror: /usr/lib/ruby/2.5.0/socket.rb:452:in `__read_nonblock': Connection reset by peer (Faraday::ConnectionFailed) Sometimes the error above occurs, and container git-mirror restart. It's not a stable error, and next time es query can be success. So retry here. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bcdb077..6100d6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -432,7 +432,14 @@ class MirrorMain query = { query: { match: { _id: git_repo } } } return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? - result = @es_client.search(index: 'repo', body: query)['hits'] + begin + result = @es_client.search(index: 'repo', body: query)['hits'] + rescue StandardError + puts $ERROR_INFO + sleep 1 + retry + end + fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: post data to rpm-repo docker
by Wang Yong 13 Apr '21

13 Apr '21

post rpm data to rpm-repo in order to createrepo ref: commit 5dd22c1e010734bc5e610f4abe46baec0f36f5bb Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a91ae4a08..312582d16 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -108,6 +108,32 @@ upload_rpm_pkg() done } +handle_rpm_list() +{ + rpm_list=($(show_rpm_files | xargs basename -a)) + srpm_list=($(show_src_rpm_files | xargs basename -a)) + full_list=(${rpm_list[@]} ${srpm_list[@]}) + j=0 + + for rpm_name in ${full_list[@]} + do + full_list[$j]="\"/srv${rpm_dest}/${rpm_name}\"" + [[ ${rpm_name} =~ "src.rpm" ]] && full_list[$j]="\"/srv${src_rpm_dest}/${rpm_name}\"" + ((j++)) + done + + full_list=$(echo ${full_list[@]} | sed 's; ;,;g') +} + +update_repo_mq() +{ + handle_rpm_list + + curl -H 'Content-Type: Application/json' -XPOST "${LKP_SERVER}:10003/upload" \ + -d "{\"upload_rpms\": ["${full_list}"]}" +} + [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg +update_repo_mq -- 2.23.0

1 0

[PATCH compass-ci] container/scheduler: fix Missing ENV key: NODE_PORT
by Cao Xueliang 13 Apr '21

13 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler/my-start b/container/scheduler/my-start index 8d800f9..e5855af 100755 --- a/container/scheduler/my-start +++ b/container/scheduler/my-start @@ -65,6 +65,7 @@ cmd = %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH compass-ci] scheduler: use NODE_PORT to run kemal
by Cao Xueliang 13 Apr '21

13 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 5 +++-- src/scheduler.cr | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index b13b396..598b263 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -7,6 +7,7 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ + SCHED_HOST SCHED_PORT SCHED_NODES REDIS_HOST @@ -38,7 +39,7 @@ defaults['SCHED_PORT'] ||= '3000' defaults['MASTER_FLUENTD_PORT'] ||= '24224' defaults['SRV_HTTP_CCI_PORT'] ||= '11307' sched_nodes = defaults.delete('SCHED_NODES') -SCHED_PORT = defaults.delete('SCHED_PORT') +SCHED_PORT = defaults['SCHED_PORT'] sched_nodes.each do |node| docker_rm "s001-alpine-#{node['port']}" @@ -69,7 +70,7 @@ sched_nodes.each do |node| -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_PORT=#{node_port} + -e NODE_PORT=#{node_port} -v /srv/tmp/public:/c/cci/scheduler/public -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} diff --git a/src/scheduler.cr b/src/scheduler.cr index 07b38a3..4e4bf24 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -9,7 +9,7 @@ module Scheduler log = JSONLogger.new begin - Kemal.run(SCHED_PORT) + Kemal.run(ENV["NODE_PORT"].to_i32) rescue e log.error(e) end -- 2.23.0

2 2

[PATCH compass-ci 2/2] src/extract-stats/stats_worker.cr: optimize output
by Lu Weitao 13 Apr '21

13 Apr '21

[Why] standardized output log for kibana Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index e371bf3..47c3a59 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -2,6 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "../lib/etcd_client" +require "../lib/json_logger" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" require "../scheduler/constants" @@ -16,6 +17,7 @@ class StatsWorker @es = Elasticsearch::Client.new @etcd = EtcdClient.new @rc = RegressionClient.new + @log = JSONLogger.new end def handle(queue_path, channel) @@ -30,7 +32,7 @@ class StatsWorker @etcd.delete(queue_path) rescue e channel.send(queue_path) - STDERR.puts e.message + @log.error(e.message) # incase of many error message when task-queue, ES does not work sleep(10) ensure @@ -95,16 +97,16 @@ class StatsWorker new_error_ids = check_new_error_ids(error_ids, job_id) unless new_error_ids.empty? sample_error_id = new_error_ids.sample - STDOUT.puts "send a delimiter task: job_id is #{job_id}" + @log.info("send a delimiter task: job_id is #{job_id}") queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" value = {"job_id" => job_id, "error_id" => sample_error_id} @etcd.put(queue, value) msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) - system "echo '#{msg}'" + @log.info(msg) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) - system "echo '#{msg}'" + @log.info(msg) end def check_new_error_ids(error_ids : Array, job_id : String) @@ -113,7 +115,7 @@ class StatsWorker begin is_exists = @rc.check_error_id error_id rescue e - STDERR.puts e.message + @log.error(e.message) next end next if is_exists -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/dump_stat.rb: optimize useless special chars
by Lu Weitao 13 Apr '21

13 Apr '21

[Why] we already use strip, so needn't chomp Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 383ff09..04a7632 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -38,7 +38,7 @@ module DumpStat stat_result.each do |key, value| key = key.resolve_invalid_bytes - key = key.chomp.strip + key = key.strip next if key[0] == '#' next if value.empty? || value == 0 next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ -- 2.23.0

1 0