- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 1/2] reset-tbox to reboot machine testbox
by Zhang Dewan 13 Jan '21

13 Jan '21

[why] sumbit a job to reboot machine testbox is better. [how] cp ~/lkp-test/jobs/reset-tbox-yaml . submit reset-tbox.yaml host_name="full-name-of-machine-testbox" Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 17 +++++++++++++++++ tests/reset-tbox.yaml | 12 ++++++++++++ 3 files changed, 39 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox create mode 100644 tests/reset-tbox.yaml diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..9c3f4f01 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +testcase: reset-tbox + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset + +host_name: +reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..b0f645d7 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,17 @@ +#!/bin/bash +# host_name + +source "/etc/lab/.account.info" +hosts_file=/etc/lab/hosts_file + +# require $host_name +host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + +if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then + echo ">>> exec ipmitool commamd!" + ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + [ $? -eq 0 ] && echo "succeed!" +else + echo "specify host cannot found0!" + exit +fi diff --git a/tests/reset-tbox.yaml b/tests/reset-tbox.yaml new file mode 100644 index 00000000..f557085f --- /dev/null +++ b/tests/reset-tbox.yaml @@ -0,0 +1,12 @@ +short_description: > + To reboot machine testbox like taishan200-2280-2s64p-256g--aN etc. + +description: | + reset-tbox executed by a special docker which running on ibmc-mgnt, where you can access all server's iBMC-os by ipmitool, and then reboot the host. + +homepage: + +parameters: + host_name: see it on the first line about 'latbox aN' + +results: -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: install libvirt
by Xiao Shenwei 13 Jan '21

13 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 1 + sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 4a9da60..2d7c9e3 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -51,3 +51,4 @@ sed '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service systemctl daemon-reload systemctl restart docker +systemctl start libvirtd diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index a63964f..37c74d0 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -21,6 +21,7 @@ pkgs=( busybox iproute hostname + libvirt ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: set default values
by Liu Yinsi 13 Jan '21

13 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f2366c2..ca91da5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -20,6 +20,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} : ${ASSIST_RESULT_PORT:=8102} : ${RESULT_WEBDAV_PORT:=3080} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -45,6 +47,9 @@ LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT +lab: nolab +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 13 Jan '21

13 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 21 +++++++++++++++++++++ 3 files changed, 47 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..a407ad40 --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed. you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..2911d3b8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,21 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + + +send_email $subject -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/3] stats/crystal/memory--stat.cr: create memory--stat.cr with memory--stat
by Lu Kaiyi 13 Jan '21

13 Jan '21

create memory--stat.cr as same content with memory--stat Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/memory--stat.cr diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr new file mode 100644 index 000000000..1a39e4075 --- /dev/null +++ b/stats/crystal/memory--stat.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby + +# time: 123456 +# cgroup: pmbench.1 +# cache 0 + +def parse + cg = 'nocg' + $stdin.each_line do |line| + case line + when /^time:/ + puts line + when /^cgroup: (\S+)/ + cg = $1 + when /(\S+)\s*(\S+)/ + k, v, = line.split + puts "#{cg}.#{k}: #{v}" + end + end +end + +parse -- 2.23.0

2 1

[PATCH v8 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

3 3

[PATCH compass-ci] lib: remove file executable permissions
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] the files in lib dir don't need executable permissions. [how] chmod a-x assign_account_client.rb chmod a-x build_my_info_client.rb chmod a-x redis_client.rb Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/assign_account_client.rb | 0 lib/build_my_info_client.rb | 0 lib/redis_client.rb | 0 3 files changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 lib/assign_account_client.rb mode change 100755 => 100644 lib/build_my_info_client.rb mode change 100755 => 100644 lib/redis_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb old mode 100755 new mode 100644 diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100755 new mode 100644 diff --git a/lib/redis_client.rb b/lib/redis_client.rb old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare remove change of failure metric
by Lu Weitao 13 Jan '21

13 Jan '21

[Why] We only display the failure metrics that belong the latest job, aimed to express the latest job exists failure. so the change rate of failure metric is useless like: centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs change fails:runs | | | 1:12 -8.3% 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied [How] centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs fails:runs | | 0:12 20:24 stderr.which:no_time_in(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/lkp/lkp/src/bin) centos openeuler metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2.638681e+10 ± 23% -0.1% 2.636343e+10 ± 22% iperf.tcp.receiver.bps 2.638853e+10 ± 23% +0.1% 2.641691e+10 ± 22% iperf.tcp.sender.bps 6.80 ± 141% -50.4% 3.37 ± 223% iperf.time.elapsed_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 58 ++++++++--------------------------------- 1 file changed, 11 insertions(+), 47 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..5bb9d26 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -83,15 +83,12 @@ def get_values(value_list, success) end def get_compare_value(base_value_average, value_average, success) - # get compare value(change or reproduction) + # get compare value(change) # - if success - return if base_value_average.zero? + return unless success + return if base_value_average.zero? - (100 * value_average / base_value_average - 100).round(1) - else - (100 * (value_average - base_value_average)).round(1) - end + (100 * value_average / base_value_average - 100).round(1) end def field_changed?(base_values, values, success, field, options) @@ -264,7 +261,7 @@ def get_html_header(matrixes_titles, success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end header = get_html_index(matrixes_titles) @@ -550,35 +547,6 @@ def format_fails_runs(fails, runs) fails_str + ':' + runs_str end -def get_reproduction_index_str(reproduction, compare_index) - reproduction_str = format('%+.1f%%', reproduction) - reproduction_index = compare_index - reproduction_str.index('.') - if reproduction_index.negative? - reproduction_str = format('%+.1e%%', reproduction).sub('e+0', 'e+').sub('e-0', 'e-') - reproduction_index = compare_index - reproduction_str.index('e') - 1 - end - return reproduction_index, reproduction_str -end - -def get_reprodcution(reproduction, compare_index) - if reproduction - reproduction_index, reproduction_str = get_reproduction_index_str(reproduction, compare_index) - space_str = ' ' * (SUB_SHORT_COLUMN_WIDTH - reproduction_str.length) - reproduction_str = space_str.insert(reproduction_index, reproduction_str) - else - reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '0') - end - reproduction_str -end - -def format_reproduction(reproduction, theme, compare_index) - color = get_compare_value_color(reproduction, theme) - colorize( - color, - get_reprodcution(reproduction, compare_index) - ) -end - def get_change_index_str(change, compare_index) change_str = format('%+.1f%%', change) change_index = compare_index - change_str.index('.') @@ -693,7 +661,7 @@ def get_title_name(success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end return common_title, compare_title end @@ -718,14 +686,14 @@ def get_base_matrix_title_symbol(common_index, success) title_symbol end -def get_other_matrixes_title_symbol(_compare_title, matrixes_number, common_index, success) +def get_other_matrixes_title_symbol(compare_title, matrixes_number, common_index, success) title_symbol = ' ' * ( (INTERVAL_WIDTH + COLUMN_WIDTH) * matrixes_number ) start_point = 0 common_symbol = success ? '\\' : '|' - compare_symbol = '|' + compare_symbol = compare_title.empty? ? '' : '|' compare_index = SUB_SHORT_COLUMN_WIDTH / 2 (matrixes_number - 1).times do |_| @@ -786,12 +754,8 @@ def get_success_str(values, index, theme, compare_index) (change_str || '') + stddev_str end -def get_failure_str(values, index, theme, compare_index) - unless index.zero? - reproduction_str = format_reproduction( - values[:reproduction], theme, compare_index - ) - end +def get_failure_str(values, index) + reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '') unless index.zero? fails_runs_str = format_fails_runs( values[:fails], @@ -810,7 +774,7 @@ def get_values_str(matrixes, success, theme) ) + INTERVAL_BLANK else get_failure_str( - values, index, theme, compare_index + values, index ) + INTERVAL_BLANK end end -- 2.23.0

1 0

[PATCH v8 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] stats/crystal/memory--stat.cr: fix some errors manually
by Lu Kaiyi 13 Jan '21

13 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index da7ac0d86..f45ab5fb1 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -6,14 +6,14 @@ def parse cg = "nocg" - $stdin.each_line do |line| + STDIN.each_line do |line| case line when /^time:/ puts line when /^cgroup: (\S+)/ cg = $1 when /(\S+)\s*(\S+)/ - k, v, = line.split + k, v = line.split puts "#{cg}.#{k}: #{v}" end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] stats/crystal/memory--stat.cr: do first conversion by ruby2crystal tools
by Lu Kaiyi 13 Jan '21

13 Jan '21

use the ruby2crystal tools do first conversion like below: /c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod memory--stat.cr /c/lkp-tests/tools/ruby2crystal.sh memory--stat.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index 1a39e4075..da7ac0d86 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -1,11 +1,11 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # time: 123456 # cgroup: pmbench.1 # cache 0 def parse - cg = 'nocg' + cg = "nocg" $stdin.each_line do |line| case line when /^time:/ -- 2.23.0

1 0

[PATCH compass-ci] doc/send-mail.md: format markdown file
by Wei Jihui 13 Jan '21

13 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/send-mail.md | 51 +++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 27 deletions(-) diff --git a/doc/send-mail.md b/doc/send-mail.md index f503a08..5af8d17 100644 --- a/doc/send-mail.md +++ b/doc/send-mail.md @@ -1,13 +1,5 @@ - # send mail - - - - - - - ## purpose The service is used to send mails with formatted mail data @@ -23,38 +15,42 @@ The service is used to send mails with formatted mail data ## data format - send_mail_yaml - - data format - data='{ - "subject": "email subject", - "to": "email_to_addr", - "body": "email message" - }' + data='{ + "subject": "email subject", + "to": "email_to_addr", + "body": "email message" + }' or - data=" - subject: email subject - to: email_to_addr - body: email message - " + data=" + subject: email subject + to: email_to_addr + body: email message + " - send_mail_text - - data format - data=" - To: email_to_addr - Subject: email_subject + data=" + To: email_to_addr + Subject: email_subject - mail_msg_line1 - mail_msg_line2 - ... - " + mail_msg_line1 + mail_msg_line2 + ... + " ## usage: - send_mail_yaml + ```shell curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_yaml' -d "$data" + ``` + - send_mail_text + ```shell curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "$data" + ``` ## example + ```ruby data = " To: test_email(a)163.com Subject: test mail 10 @@ -63,3 +59,4 @@ The service is used to send mails with formatted mail data " %x(curl -XPOST 'localhost:49000/send_mail_text' -d "#{data}") + ``` -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: modify job_id to job id
by Hu Xuejiao 13 Jan '21

13 Jan '21

[why] We use "es-find (id=xxx)" to search id-information,but when submit, it is (job_id=xxx). so it is recommended that the value be the same as the id=xxx. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index 5273896b..39cdcf57 100755 --- a/sbin/submit +++ b/sbin/submit @@ -168,10 +168,10 @@ ARGV.each do |jobfile| JSON.parse(messages).each do |msg| if msg['message'].empty? job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job_id=#{msg['job_id']}") + puts("submit #{jobfile}, got job id=#{msg['job_id']}") else opt_monitor = false - puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") end end end -- 2.23.0

2 1

[PATCH compass-ci] doc: add a doc about catalogs under /srv/git
by Li Yuanchao 13 Jan '21

13 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/about_srv_git.md | 182 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 doc/about_srv_git.md diff --git a/doc/about_srv_git.md b/doc/about_srv_git.md new file mode 100644 index 0000000..3374828 --- /dev/null +++ b/doc/about_srv_git.md @@ -0,0 +1,182 @@ + +# ��`/srv/git`�� + +``` +delimiter: + ��git-bisect,��good commit�� + +auto-submit�� + ��job��rabbitmq��git-mirror��job�� + +git-daemon�� + ��git��/srv/git��git clone/git push�� + �� "git clone git://localhost/$project" ��git-daemon�� + +git-mirror�� + ��clone�� + +remote-git�� + ��/srv/git��curl��/srv/git��git�� +``` + + +# /srv/git�� + + +��a-z��git-mirror�� +``` +drwxrwsr-x. 55 lkp committer 4.0K 2020-11-20 03:54 u +drwxrwsr-x. 78 lkp committer 4.0K 2020-12-04 11:24 v +drwxrwsr-x. 75 lkp committer 4.0K 2020-12-04 11:25 i +drwxrwsr-x. 986 lkp committer 36K 2020-12-04 11:37 d +drwxrwsr-x. 1612 lkp committer 60K 2020-12-05 03:50 g +drwxrwsr-x. 827 lkp committer 28K 2020-12-05 04:00 b +drwxrwsr-x. 66 lkp committer 4.0K 2020-12-05 04:05 w +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-05 04:17 y +drwxrwsr-x. 706 lkp committer 28K 2020-12-05 05:49 e +drwxrwsr-x. 837 lkp committer 28K 2020-12-08 22:05 f +drwxrwsr-x. 679 lkp committer 24K 2020-12-10 18:18 k +drwxrwsr-x. 948 lkp committer 36K 2020-12-11 02:31 h +drwxrwsr-x. 384 lkp committer 12K 2020-12-11 03:12 j +drwxrwsr-x. 927 lkp committer 36K 2020-12-15 03:11 n +drwxrwsr-x. 1757 lkp committer 56K 2020-12-15 11:09 l +drwxrwsr-x. 54 lkp committer 4.0K 2020-12-15 14:29 t +drwxrwsr-x. 1018 lkp committer 36K 2020-12-15 15:57 a +drwxrwsr-x. 1344 lkp committer 40K 2020-12-15 22:28 m +drwxrwsr-x. 809 lkp committer 28K 2020-12-16 05:21 o +drwxrwsr-x. 1342 lkp committer 36K 2020-12-21 19:19 c +drwxrwsr-x. 5800 lkp committer 212K 2020-12-22 03:08 p +drwxrwsr-x. 395 lkp committer 12K 2020-12-22 22:16 q +drwxrwsr-x. 1632 lkp committer 64K 2020-12-28 21:25 r +drwxrwsr-x. 445 lkp committer 20K 2020-12-28 23:16 x +drwxrwsr-x. 1796 lkp committer 56K 2021-01-05 09:20 s +drwxrwsr-x. 39 lkp committer 4.0K 2021-01-08 10:48 z +``` + +��lkp:committer��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 10:57 git.code.sf.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 11:22 code.reversed.top +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-20 03:56 source.ffmpeg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-21 02:46 git.kernel.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-22 04:05 gitlab.univ-nantes.fr +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-02 18:14 git.musl-libc.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 05:56 gitlab.dkrz.de +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 06:17 git.sv.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:17 git.ageinghacker.net +drwxrwsr-x. 11 lkp committer 4.0K 2020-12-05 06:39 git.qemu-project.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:44 git.sigxcpu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:56 git.osmocom.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 07:38 gitlab.linphone.org +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 07:52 gitlab.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-10 14:40 webrtc.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:33 www.kismetwireless.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:34 src.fedoraproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-16 05:53 git.tuxfamily.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:14 chriso(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:55 nedko.arnaudov.name +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:59 git.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 18:01 android.googlesource.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-17 18:40 gopkg.in +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 19:53 git.cryptomilk.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:05 git.dpkg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:09 kernel.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-19 14:15 aur(a)archlinux.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-21 05:57 boringssl.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:44 hub.spigotmc.org +drwxrwsr-x. 14 lkp committer 4.0K 2020-12-22 04:49 repo.or.cz +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:52 luajit.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:02 git.assembla.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-22 05:07 git.purrdata.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:09 brettc(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:11 git.teknik.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 14:40 code.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-23 01:50 git.qemu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:51 git.merproject.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-23 01:52 code.qt.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:57 git.tartarus.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-27 22:17 gist.github.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:28 invent.kde.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:55 git.sv.nongnu.org +drwxrwsr-x. 6 lkp committer 4.0K 2020-12-29 02:57 git.savannah.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:57 git.xfce.org +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-29 02:58 anongit.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:59 git.ejiek.com +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-29 03:00 gitlab.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-30 22:00 git.torproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-02 05:58 skia.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-03 05:57 aur(a)aur4.archlinux.org +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-05 10:56 gitlab.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 gitlab.xiph.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 git.savannah.nongnu.org +drwxrwsr-x. 4 lkp committer 4.0K 2021-01-05 11:01 git.tukaani.org +drwxrwsr-x. 13 lkp committer 4.0K 2021-01-05 11:04 aur.archlinux.org +drwxrwsr-x. 6 lkp committer 4.0K 2021-01-05 11:07 bitbucket.org +drwxrwsr-x. 1664 lkp committer 40K 2021-01-07 09:08 github.com +drwxrwsr-x. 5 lkp committer 4.0K 2021-01-08 05:03 go.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-08 05:24 chromium.googlesource.com +drwxrwsr-x. 14 lkp committer 4.0K 2021-01-08 06:00 aur(a)aur.archlinux.org +``` + +��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:30 coccinelle +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 oec-hardware +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 libbpf +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:46 qemu +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 smatch +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 gtk +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-21 11:02 pixz +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:26 sparse +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:27 pahole +drwxrwsr-x. 3 lkp committer 4.0K 2020-09-22 15:10 upstream-repos +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-09 19:31 linux +drwxrwsr-x. 24 lkp committer 4.0K 2020-11-10 16:40 submodules +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-02 10:00 compass-ci +``` + +��pkgbuild��make�� +``` +drwxrwsr-x. 43 lkp team 200K 2020-11-11 15:17 archlinux +drwxrwsr-x. 82 root team 4.0K 2020-12-15 18:12 pkg +drwxrwsr-x. 13 wfg team 4.0K 2020-12-16 12:55 make +``` + +��rpm��spec�� +``` +drwxrwsr-x. 3 root committer 4.0K 2020-12-08 11:15 rpmbuild-pkg +``` + +��clone�� +``` +drwxrwsr-x. 3 zhy committer 4.0K 2020-06-16 11:25 mysql-server +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 atomic +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 spinlock +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:29 mutex +drwxrwsr-x. 4 zhy committer 4.0K 2020-08-24 11:52 avx2neon +drwxrwsr-x. 5 liping committer 4.0K 2020-08-24 11:52 AvxToNeon +``` + +��compass-ci/lkp-tests/.../upstream-repos�� +``` +drwxrwsr-x. 3 wfg committer 4.0K 2020-09-14 15:26 gitee.com +``` + +�� +``` +drwxrwxrwx. 3 weijihui weijihui 4.0K 2020-07-27 15:26 intrintesttool +``` + +ansible�� +``` +drwxrwsr-x. 17 root committer 4.0K 2020-12-29 10:33 ansible-examples +``` + +�� +``` +drwxrwsr-x. 2 root committer 16K 2020-03-27 10:45 lost+found +drwxrwsr-x. 2 wfg committer 4.0K 2020-07-24 16:58 gzip +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-01 11:15 cci +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-04 16:27 lkp-tests +drwxrwsr-x. 4 wfg committer 4.0K 2020-09-19 14:15 crystal-ci +``` -- 2.23.0

2 1

[PATCH compass-ci] tool/es-jobs: add a new tool
by Wu Zhende 12 Jan '21

12 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'id,job_state,dmesg.timestamp:last,stats.sum,stats.avg' -r '0' means: 1.query the job whose submit_id is xxx 2.I want to get id,job_state,dmesg.timestamp:last,stats.sum,stats.avg 3.refine stats.success jobs [Output] { "id": [ "crystal.597090", "crystal.597093" ], "job_state": [ "finished", "finished" ], "dmesg.timestamp:last": [ 75.256233, 48.990905 ], "stats.sum": { "kmsg.timestamp:last": 124.24713799999999, "dmesg.timestamp:last": 124.24713799999999 }, "stats.avg": { "iperf.tcp.receiver.bps": 17008962077.755077, "iperf.tcp.sender.bps": 17036843967.505566, "iperf.udp.bps": 524286.6541701214 }, "stats.count": { "stats.success": 2, "stats.success_jobs": [ "crystal.597090", "crystal.597093" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 53 +++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..78db8f0 --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [-1], fields = []) + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @refine_jobs = [] + @query_result = @es.multi_field_query(es_query) + @jobs = {} + @stats_count = Hash.new(0) + @stats_sum = Hash.new(0) + @stats_avg = Hash.new(0) + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknow', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + @query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + @result[field] << job[field] if job[field] + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + @stats_count[@stats_level[level]] += 1 + stat_key = @stats_level[level] + '_jobs' + @stats_count[stat_key] = [] unless @stats_count.key?(stat_key) + @stats_count[stat_key] << job_id + end + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + if function_stat?(key) + @stats_sum[key] += value + else + @stats_avg[key] = (@stats_avg[key] + value) / 2 + end + + level = get_stat_level(key, level) + end + return level + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats_count + } + result['stats.sum'] = @stats_sum if @fields.include?('stats.sum') + result['stats.avg'] = @stats_avg if @fields.include?('stats.avg') + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..f075ef2 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be id, suite, os, etc.' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "-1,0,1,2,3"' + opts.separator '-1 means not refine' + opts.separator '{ + -1 => null, + 0 => stats.succes, + 1 => stats.unknow, + 2 => stats.warning, + 3 => stats.has_error +}' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields) +es_jobs.output -- 2.23.0

3 2

[PATCH v2 lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 12 Jan '21

12 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 21 +++++++++++++++++++++ 3 files changed, 47 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..d7c2c44a --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..2911d3b8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,21 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + + +send_email $subject -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..6642703 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,30 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + pkg_style = @hash[item]? + break if pkg_style end + return nil unless pkg_style && pkg_style != nil + + tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] + tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] + tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + + tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +284,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 12 Jan '21

12 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 16 +++++++++++++--- tests/cci-makepkg | 14 +++++++++++--- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..5a473a00c 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,12 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" - pkg_args="-A --check --skippgpcheck" + + if [ "${os}" == "archlinux" ]; then + pkg_args="-A --check --skippgpcheck -s --needed --noconfirm" + else + pkg_args="-A --check --skippgpcheck" + fi [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,8 +158,13 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + if [ "${os}" == "archlinux" ]; then + BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + else + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + fi local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..e4bab41eb 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,18 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + if [ "${DISTRO}" == "archlinux" ]; then + local makepkg="$LKP_SRC/sbin/makepkg -A -s --needed --noconfirm --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + else + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + fi [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping - search all email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue email_mapping --search-all options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings -s, --search search email mapping -h, --help show this message --delete-queue delete mapping queue --search-all search all email mappings the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 119 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..365aabf --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,119 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false, + 'search_all_map' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + opts.banner += " email_mapping --search-all\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on('--search-all', 'search all email mappings') do + map_conf['search_all_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +required_opts = ['-n', '-f', '--delete-queue', '--search-all'] + +if ARGV.empty? + ARGV << '-h' +elsif (required_opts - ARGV).eql? required_opts + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = RedisClient.new('email_mapping') + if map_conf['re_mapping'] + email_mapping.reset_hash_key(r_name, d_email) + elsif map_conf['del_map'] + email_mapping.delete_hash_key(r_name) + elsif map_conf['search_map'] + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end + elsif map_conf['search_all_map'] + all_mappings = email_mapping.search_all_hash_key + + all_mappings.each do |k, v| + puts "#{k}: #{v}\n" + end + elsif map_conf['delete_queue'] + email_mapping.delete_queue + else + return if email_mapping.add_hash_key(r_name, d_email) + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key search all redis hash keys, values delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 49 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 lib/redis_client.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..0c05252 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to handle the redis queues +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end + + def search_all_hash_key + h_values = @redis.hscan_each(@queue).to_h + + return h_values + end +end -- 2.23.0

1 0

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37da..ea55b2ebc 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -25,28 +25,17 @@ postgres_run_log() init_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) - init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) + [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 10 +++++----- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..31762384d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,12 +7,12 @@ # - config # upstream_url -# We'll use the below global env vars, but not treat them as parameter to avoid +# We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..c89737aca 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..7d1e5cb15 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change possible matches for webhook
by Li Yuanchao 12 Jan '21

12 Jan '21

As directory structure of upstream-repos changed, the possible git_repo of repository to match also need to change. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index f5c653b..2adaaa1 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -319,16 +319,12 @@ class MirrorMain end # example - # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc - # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) fork_name, project = webhook_url.split('/')[-2, 2] - git_repo = "#{project}/#{project}" - return git_repo if check_git_repo(git_repo, webhook_url) - - git_repo = "#{project}/#{fork_name}" + git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" return git_repo if check_git_repo(git_repo, webhook_url) git_repo = "#{project[0].downcase}/#{project}/#{project}" -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb index 9d72ecc..d3a9a62 100755 --- a/sbin/email-mapping.rb +++ b/sbin/email-mapping.rb @@ -9,7 +9,8 @@ require 'optparse' REDIS_HOST = 'localhost' REDIS_PORT = '6379' -require "#{ENV['CCI_SRC']}/lib/email_mapping_client" +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' r_name = nil d_email = nil @@ -74,17 +75,22 @@ end options.parse!(ARGV) def email_mapping(r_name, d_email, map_conf) - email_mapping = EmailMapping.new(r_name, d_email) + email_mapping = RedisClient.new('email_mapping') if map_conf['re_mapping'] - email_mapping.re_map_email + email_mapping.reset_hash_key(r_name, d_email) elsif map_conf['del_map'] - email_mapping.delete_email_mapping + email_mapping.delete_hash_key(r_name) elsif map_conf['search_map'] - puts email_mapping.search_map + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end elsif map_conf['delete_queue'] - email_mapping.delete_mapping_queue + email_mapping.delete_queue else - return if email_mapping.add_email_mapping + return if email_mapping.add_hash_key(r_name, d_email) puts "#{r_name} has already mapped an email." end -- 2.23.0

1 1

[PATCH v4 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 43 +++++++++++++++++++ sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100755 lib/redis_client.rb create mode 100755 sbin/email-mapping.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..142d7f0 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end +end diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..9530d47 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = '/etc/compass-ci/account/autosubmit.yaml' +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..7405ca0 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,18 @@ run_docker() done } +kill_all() +{ + for line in $(cat "$1") + do + [ -z "$ line" ] && break + kill -9 "$line" + done +} + run_qemu run_docker wait + +kill_all pid +kill_all $CCI_SRC/providers/dc.pid -- 2.23.0

2 1

[PATCH compass-ci] fix: the error docker name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] + docker cp -a init_docker:/tmp/openeuler-20.09-20210112.cgz /home/wcl/ Error: No such container:path: init_docker:/tmp/openeuler-20.09-20210112.cgz --- container/osimage/openeuler/lib | 2 +- container/osimage/openeuler/run | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/osimage/openeuler/lib b/container/osimage/openeuler/lib index e0b7017..34dd0ec 100755 --- a/container/osimage/openeuler/lib +++ b/container/osimage/openeuler/lib @@ -40,6 +40,6 @@ load_docker_image() cp_image_to_host() { - docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + docker cp -a initramfs_${image}:/tmp/${IMAGE_NAME} $HOME/ echo "result: $(ls $HOME/${IMAGE_NAME})" } diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run index 5d7da72..6ed4069 100755 --- a/container/osimage/openeuler/run +++ b/container/osimage/openeuler/run @@ -24,5 +24,5 @@ cmd=( load_docker_image "${cmd[@]}" cp_image_to_host -docker_rm init_docker &> /dev/null +docker_rm initramfs_${image} &> /dev/null echo "build finished" -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..c946bec 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,13 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = nil) @lab = lab || 'nolab' + @my_token = my_token || %x(uuidgen).chomp @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => @my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..8b8cc8c 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = "/etc/compass-ci/account/autosubmit.yaml" +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] bin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 1

[PATCH v2 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 8 ++++---- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..48c738f52 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -9,10 +9,10 @@ # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..487121454 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..4f5f4c2f2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] fix: unusable package name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] Packing package. Please wait. /usr/bin/setup-image: line 41: ./tmp/$IMAGE_PACK: ambiguous redirect before: IMAGE_PAVK=' openeuler-20.03.cgz' after: IMAGE_PAVK=openeuler-20.03.cgz Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index ce51ba3..8a7b315 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -12,7 +12,7 @@ check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag IMAGE_NAME=${docker_name_tag%%:*} -IMAGE_PACK=${IMAGE_NAME##*/}.cgz +IMAGE_PACK=$(echo ${IMAGE_NAME##*/}.cgz) start=( docker run -- 2.23.0

1 0

[PATCH v2 compass-ci] container/defconfig: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

[why] when submitting jobs, the job will read data from /etc/compass-ci/defaults/*.yaml. the read data is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..97d924d 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH v5 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, network is ok, but actually network is unreachable, because routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so put export_ip_mac in network_ok to ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0

[PATCH v4 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, but routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so check export_ip_mac in network_ok to make ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0

[PATCH v2 compass-ci] container/add-software: add to clean hidden files in runtime dirs
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the first pkgbuild_repo if no '-git'
by Li Yuanchao 12 Jan '21

12 Jan '21

As new pkgbuild_repos don't have '-git' at the end, return the first one instead. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 8845b13..d765094 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -18,7 +18,7 @@ class AutoSubmit end def get_pkgbuild_repo(repo_array) - pkgbuild_repo = nil + pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| next unless repo =~ /-git$/ -- 2.23.0

2 1

[PATCH compass-ci] container: mount default config dir to delimiter container for submitting job
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/delimiter/start b/container/delimiter/start index 27e8d45..5d7037d 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -23,6 +23,7 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm 'delimiter' cmd = %w[ @@ -34,8 +35,9 @@ cmd = %w[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} - -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter -- 2.23.0

2 1

[PATCH v2 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 12 Jan '21

12 Jan '21

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..4ff4285 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,32 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +98,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

2 1

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 12 Jan '21

12 Jan '21

>+ def lftp_mirror >+ @result_root.delete_prefix!('/srv') >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' >+ url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" >+ system "lftp -c mirror #{url} >/dev/null 2>&1" >+ end >+ >+ def lftp(data) >+ if @result_root >+ lftp_mirror Seems it'll repeatedly call lftp on every new message? When do we start downloading? Ideally do it once after extract stats. >+ else >+ set_result_root(data) That else seems illogical. Typically we set a var before checking it. Maybe lftp_mirror will be called at the next message, however it's still illogical thing. Thanks, Fengguang >+ end >+ end >+ > def stop(data, web_socket, code = 1000, reason = 'normal') > @stop_query.each do |key, value| > return false unless data[key] == value >@@ -109,7 +136,7 @@ class Monitor > > @query.each do |k, v| > @query[k] = JSON.parse(v) >- rescue >+ rescue StandardError > end > query = @query.to_json > puts "query=>#{query}" >@@ -136,6 +163,7 @@ class Monitor > > output(data) > connect(data, ws) >+ lftp(data) > > stop(data, ws) if @action['stop'] > end >diff --git a/sbin/submit b/sbin/submit >index 05b3e86dd..a92556fff 100755 >--- a/sbin/submit >+++ b/sbin/submit >@@ -47,6 +47,10 @@ options = OptionParser.new do |opts| > actions << 'connect' > end > >+ opts.on('-l', '--lftp', 'lftp mirror job result') do >+ actions << 'lftp_result' >+ end >+ > opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do > opt_monitor = true > filter = YAML.load ARGV[0] if ARGV[0] >-- >2.23.0 >

1 0

[PATCH compass-ci] lib/job.cr: set sshr info when submit reproduce job
by Wu Zhende 12 Jan '21

12 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 6f344fe..218e2e1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -62,6 +62,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + set_sshr_info() return end end @@ -203,7 +204,7 @@ class Job end private def set_sshr_info - return unless @hash["ssh_pub_key"]? + return unless @hash.has_key?("ssh_pub_key") self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] -- 2.23.0

2 1

[PATCH lkp-tests 1/3] stats/crystal/memory--stat.cr: create memory--stat.cr with memory--stat
by Lu Kaiyi 12 Jan '21

12 Jan '21

create memory--stat.cr as same content with memory--stat Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/memory--stat.cr diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr new file mode 100644 index 000000000..1a39e4075 --- /dev/null +++ b/stats/crystal/memory--stat.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby + +# time: 123456 +# cgroup: pmbench.1 +# cache 0 + +def parse + cg = 'nocg' + $stdin.each_line do |line| + case line + when /^time:/ + puts line + when /^cgroup: (\S+)/ + cg = $1 + when /(\S+)\s*(\S+)/ + k, v, = line.split + puts "#{cg}.#{k}: #{v}" + end + end +end + +parse -- 2.23.0

2 1