- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] fix network problem: add domestic resource url for Debian packages
by Luan Shengde 18 Mar '21

18 Mar '21

[why] the default url for Debian package resources may lead to download timeout problems: E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is expired (invalid since 9h 8min 10s). Updates for this repository will not be applied. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 2 ++ container/mail-robot/conf/sources.list | 9 +++++++++ container/send-internet-mail/Dockerfile | 2 ++ container/send-internet-mail/conf/sources.list | 9 +++++++++ container/send-mail/Dockerfile | 2 ++ container/send-mail/conf/sources.list | 9 +++++++++ 6 files changed, 33 insertions(+) create mode 100644 container/mail-robot/conf/sources.list create mode 100644 container/send-internet-mail/conf/sources.list create mode 100644 container/send-mail/conf/sources.list diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index 4402543..e4b3e7c 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -7,6 +7,8 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +COPY conf/sources.list* /etc/apt/ + RUN apt-get update && \ apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client diff --git a/container/mail-robot/conf/sources.list b/container/mail-robot/conf/sources.list new file mode 100644 index 0000000..8179e01 --- /dev/null +++ b/container/mail-robot/conf/sources.list @@ -0,0 +1,9 @@ +deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib + +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index 29f1bc6..fa9ec2c 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -5,5 +5,7 @@ FROM debian MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +COPY conf/sources.list* /etc/apt/ + RUN apt-get update && \ apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler diff --git a/container/send-internet-mail/conf/sources.list b/container/send-internet-mail/conf/sources.list new file mode 100644 index 0000000..8179e01 --- /dev/null +++ b/container/send-internet-mail/conf/sources.list @@ -0,0 +1,9 @@ +deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib + +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 29f1bc6..fa9ec2c 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -5,5 +5,7 @@ FROM debian MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +COPY conf/sources.list* /etc/apt/ + RUN apt-get update && \ apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler diff --git a/container/send-mail/conf/sources.list b/container/send-mail/conf/sources.list new file mode 100644 index 0000000..8179e01 --- /dev/null +++ b/container/send-mail/conf/sources.list @@ -0,0 +1,9 @@ +deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib + +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib -- 2.23.0

1 0

[PATCH compass-ci] dracut-initrd: add xz-utils
by Wang Chenglong 18 Mar '21

18 Mar '21

[Why] wcl@crystal ~/compass-ci/container/dracut-initrd% ./run /srv/os/centos/x86_64/6.10/lib/modules/3.10.0-957.1.3.el7.x86_64 dracut: Executing: /usr/bin/dracut -o iscsi --omit-drivers iscsi --force --kver 3.10.0-957.1.3.el7.x86_64 -k /lib/modules/3.10.0-957.1.3.el7.x86_64 /boot/initramfs.lkp-3.10.0-957.1.3.el7.x86_64.img dracut: Kernel modules are compressed with xz, but xz is not available. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 616183f..b8eb6f5 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -6,7 +6,7 @@ export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y --fix-missing --no-install-recommends -qq apt-utils \ -nfs-common netbase cifs-utils kmod rsync dracut dracut-network \ +nfs-common netbase cifs-utils kmod rsync dracut dracut-network xz-utils \ dracut-config-generic apt-get clean -- 2.23.0

2 1

[PATCH v3 lkp-tests] stat/ansible_test: stats result for ansible_test
by Wang Chenglong 18 Mar '21

18 Mar '21

The result of ansible_test are counted and recorded in the ES. query it with the 'es-jobs' command. sum.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng: 1 raw.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng.message: ["{\"changed\": false, \"msg\": \"Unsupported pkg mgr. This role is depends on apt, yum, or pkgng.\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 47 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..d6acad1ab 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,40 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + return if error_msg.nil? + + if error_msg.is_a? Array + error_msg.each do |i| + error_id = common_error_id i + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" + end + elsif error_msg.is_a? String + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" end end + +def common_error_id(line) + line.gsub!(/[^\w]/, '-') + line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one + line.gsub!(/^-|-$/, '') + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + output_error ansible_failed_json['msg'] + output_error ansible_failed_json['message'] + output_error ansible_failed_json['cmd'] + output_error ansible_failed_json['failures'] +end -- 2.23.0

1 0

[PATCH compass-ci] container/crystal-shards: replace https:// with git:// for building image
by Cao Xueliang 17 Mar '21

17 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-shards/Dockerfile | 7 +++++-- container/crystal-shards/git | 7 +++++++ 2 files changed, 12 insertions(+), 2 deletions(-) create mode 100755 container/crystal-shards/git diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 2311554..15fe523 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -5,7 +5,10 @@ FROM alpine:crystal-base MAINTAINER chief <taxcom(a)tom.com> -copy shard.yml /usr/share/crystal/app/shard.yml +COPY git /usr/local/sbin/git +RUN echo -e "[url \"git://\"]\n\tinsteadOf = https://" >> /etc/gitconfig + +COPY shard.yml /usr/share/crystal/app/shard.yml WORKDIR /usr/share/crystal/app RUN shards @@ -14,7 +17,7 @@ sed -i '99s/arguments\[:id]/arguments\[:id]?/' /usr/share/crystal/app/lib/elast sed -i 's/, Utils.__listify(arguments\[:q].as(String))/ /' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr;\ sed -i '205a \ params.clear' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr; -copy shard-amqp.yml /usr/share/crystal/app/shard.yml +COPY shard-amqp.yml /usr/share/crystal/app/shard.yml RUN shards CMD ["bash"] diff --git a/container/crystal-shards/git b/container/crystal-shards/git new file mode 100755 index 0000000..7bb0b85 --- /dev/null +++ b/container/crystal-shards/git @@ -0,0 +1,7 @@ +#!/bin/bash + +if [[ "$*" = "ls-remote --get-url origin" ]]; then + /usr/bin/git "$@" | sed 's|git://|https://|g' +else + /usr/bin/git "$@" +fi -- 2.23.0

1 0

[PATCH compass-ci] container/crystal-shards: delete gitcache host and port
by Cao Xueliang 17 Mar '21

17 Mar '21

The gitcache service cannot be used to maintain branches when build crystal-shards images. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-shards/Dockerfile | 4 ---- container/crystal-shards/build | 2 +- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index bdc5b66..2311554 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -5,10 +5,6 @@ FROM alpine:crystal-base MAINTAINER chief <taxcom(a)tom.com> -ARG GITCACHE_HOST -ARG GITCACHE_PORT -RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - copy shard.yml /usr/share/crystal/app/shard.yml WORKDIR /usr/share/crystal/app diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 7c2a712..00153ab 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -11,4 +11,4 @@ docker_skip_rebuild "alpine:crystal-shards" load_cci_defaults -docker build --build-arg GITCACHE_HOST=$GITCACHE_HOST --build-arg GITCACHE_PORT=$GITCACHE_PORT -t alpine:crystal-shards . +docker build -t alpine:crystal-shards . -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 17 Mar '21

17 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming" : { "all" : $all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fali }, "chenqun" : { "all" : $all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fali }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..4a5a51f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -548,3 +548,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'all' => all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 17 Mar '21

17 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..ea45d59 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -548,3 +548,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'all' => all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_table(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 17 Mar '21

17 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..131cff0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_group_jobs_table?suite=iperf&dimension=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - dimension +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# "chenqun" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# ... +# } +get '/get_group_jobs_table' do + group_jobs_table(params) +end -- 2.23.0

2 2

[PATCH compass-ci] sparrow/3-code/dev-env: set $lab default value
by Liu Yinsi 17 Mar '21

17 Mar '21

before: $lab always is nolab after: if env $lab already exists, use env value. it can make code more flexible. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 6072ce8..e182b65 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -4,6 +4,7 @@ [ -z $server_ip ] && server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') server_name=$(hostname | cut -f1 -d.) +: ${lab:=nolab} : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${ES_HOST:=$server_ip} @@ -38,7 +39,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port -lab: nolab +lab: $lab EOF mkdir -p /etc/compass-ci/service -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-ci: delete env GITCACHE_HOST and GITCACHE_PORT
by Liu Yinsi 17 Mar '21

17 Mar '21

[why] gitcache service cannot be used to maintain branches when build compass-ci/container/crystal-shards images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 8e6acbac4..6f97d7151 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -6,8 +6,6 @@ export_env() { export initrd_http_host=$INITRD_HTTP_HOST export initrd_http_port=$INITRD_HTTP_PORT - export gitcache_host=$GITCACHE_HOST - export gitcache_port=$GITCACHE_PORT } deploy() -- 2.23.0

1 0