- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/error_messages.rb: fix error when file is not existed
by Lin Jiaxin 19 Dec '20

19 Dec '20

Traceback (most recent call last): 4: from compare_error_messages.rb:63:in `<main>' 3: from compare_error_messages.rb:20:in `get_compare_result' 2: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `obtain_error_messages' 1: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `open' /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `initialize': No such file or directory @ rb_sysopen - /srv/result/build-pkg/2020-12-16/vm-2p32g.linjiaxin/openeuler-20.03-aarch64/archlinux-community--fcgi-trunk-HEAD/crystal.193087/build-pkg (Errno::ENOENT) Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..61f8070 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -30,12 +30,14 @@ class ErrorMessages end def obtain_error_messages - File.open((a)file).each_line do |line| - next if extract_error_message(line) - next unless @in_stderr - next unless @error_message['error_line'] =~ /error|warning/i + if FileTest.exist?(@file) + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i - @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end end return @error_messages end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..0b51fc3 --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = get_new_error_ids(later_error_ids, previous_error_ids) + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def get_new_error_ids(later_error_ids, previous_error_ids) + return nil_to_empty_array(later_error_ids) - nil_to_empty_array(previous_error_ids) +end + +def nil_to_empty_array(array) + array = [] if array.nil? + return array +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 50 +++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..c96edef --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = later_error_ids - previous_error_ids + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] container: http-initrd container support job read /srv/cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..18813a2 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /cci { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..8341050 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci:/usr/share/nginx/html/cci:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] container: http-initrd container support job write cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..546e824 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /build-config { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..ac8506e 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci/build-config:/usr/share/nginx/html/build-config:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH compass-ci] doc: update documents
by Luan Shengde 19 Dec '20

19 Dec '20

update documents: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 2 +- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..1e8d618 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid - generate uuid + my_token + generate my_token apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - my_email - my_name - - my_uuid + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..8ae6c6d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -25,7 +25,7 @@ Which contains account information for you: my_email my_name - my_uuid + my_token ## step 3: one-time setup diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..160fdd0 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -14,7 +14,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ```SHELL mkdir -p ~/.config/compass-ci/defaults/ cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx my_email: xxx my_name: xx EOF -- 2.23.0

2 2

[PATCH v3 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH v2 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci] modify options for auto submit api-avx2neon.yaml
by Li Ping 19 Dec '20

19 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sbin/auto_submit.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index d0f4fbe..e1eb39b 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,5 +1,5 @@ a/AvxToNeon/AvxToNeon: -- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml +- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..85159f5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,27 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

2 3

[PATCH lkp-tests] lib/monitor: get host from job
by Wu Zhende 19 Dec '20

19 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH compass-ci] modify assign-account-email
by Li Ping 19 Dec '20

19 Dec '20

[why]: source /root/.bashrc && /root/.bash_profile follow this step can not success. Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..dccbd40 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ~/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..5644ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,25 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end + end - self["upload_dirs"] = "#{result_root}#{package_dir}" + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci 3/3] rename my_uuid to my_token
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 4 ++-- sbin/build-my-info.rb | 6 +++--- sbin/es-accounts-mapping.sh | 2 +- src/delimiter/utils.rb | 4 ++-- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..d3778e7 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid + my_token generate uuid apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - - my_email - my_name - - my_uuid + - my_email + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..7e1a6a6 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -23,9 +23,9 @@ Email template: Which contains account information for you: - my_email my_name - my_uuid + my_email + my_token ## step 3: one-time setup diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 73f7390..d75f767 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,12 +12,12 @@ print 'email: ' my_email = $stdin.echo = gets.chomp print 'name: ' my_name = $stdin.echo = gets.chomp -my_uuid = %x(uuidgen).chomp +my_token = %x(uuidgen).chomp my_info = { - 'my_email' => my_email, 'my_name' => my_name, - 'my_uuid' => my_uuid + 'my_email' => my_email, + 'my_token' => my_token } def store_account_info(my_info) diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index d5566dc..d04b0f3 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -18,7 +18,7 @@ else "my_commit_url": { "type": "keyword" }, - "my_uuid": { + "my_token": { "type": "keyword" }, "my_email": { diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index e87bbbe..41803f3 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -101,9 +101,9 @@ module Utils raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info job['suite'] = 'bisect' - job['my_uuid'] = account_info['my_uuid'] job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] + job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id job.delete('error_ids') @@ -127,7 +127,7 @@ module Utils def create_bisect_log(git_dir) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT - log_file = File.join(TMP_RESULT_ROOT, "bisect.log") + log_file = File.join(TMP_RESULT_ROOT, 'bisect.log') log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } -- 2.23.0

1 0

[PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 1/3] doc/manual: improve install client document
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- ...ci\345\256\242\346\210\267\347\253\257.md" | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..dd115f7 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -2,32 +2,25 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 +## 申请帐号 + :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email - receive email from compass-ci(a)qq.com +- get reply email and follow its instructions to + - setup default config -## Getting started +## 下载安装 lkp-tests -1. setup default config +Run the following command to install/setup lkp-test: - run the following command to add the below setup to default config file -```SHELL - mkdir -p ~/.config/compass-ci/defaults/ - cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx - my_email: xxx - my_name: xx - EOF -``` -2. download lkp-tests and dependencies - - run the following command to install the lkp-test and effect the configuration ```SHELL git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ~/.bashrc && source ~/.bash_profile + source ~/.${SHELL##*/}rc ``` -3. submit job - Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +## submit job + +Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] fetch-mail: update Dockerfile
by Luan Shengde 19 Dec '20

19 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile index 0434e09..99abf86 100644 --- a/container/fetch-mail/Dockerfile +++ b/container/fetch-mail/Dockerfile @@ -5,6 +5,7 @@ FROM centos:latest MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +RUN groupadd -g 1147 mailer RUN useradd -u 1144 -g 1147 mailer RUN yum update -y RUN yum install -y procmail fetchmail -- 2.23.0

1 0

[PATCH lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] ssh password-free login for testbox of docker [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

help star
by Wu Fengguang 18 Dec '20

18 Dec '20

Wei Jihui is our help star, thanks Wei Jihui! 24.0 Wu Fengguang 22.8 Wei Jihui 21.4 Yu Chuan 18.7 Cao Xueliang 12.0 Wu Zhende 10.6 Xiao Shenwei 9.2 Lu Weitao 8.5 Hu Xuejiao 7.8 Wang Yong 7.4 Liu Shaofei 6.4 Luan Shengde 5.9 Xu Xijian 5.7 Cui Yili 5.3 Zhang Yu 5.3 Zhang Dewan 5.2 Zhang Yale 5.0 Bai Jing 4.8 Liu Yinsi 4.4 Lin Jiaxin 4.3 Li Ping 4.0 Wang Chenglong 2.7 Sun Yukui 2.6 Lu Kaiyi 2.4 Ren Wen 2.2 Du Kaitian 1.3 Li Yuanchao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

git count
by Wu Fengguang 18 Dec '20

18 Dec '20

Shengde is our code star again, thanks Shengde! wfg@crystal ~/todo% bin/git-count 14 Luan Shengde 13 Cao Xueliang 12 Wu Fengguang 10 Liu Yinsi 10 Sun Yukui 10 Wu Zhende 8 Ren Wen 7 Zhang Yu 7 Li Yuanchao 6 Cui Yili 6 Wang Chenglong 6 Lu Kaiyi 5 Liu Shaofei 5 Lu Weitao 5 Wei Jihui 5 Yu Chuan 5 Wang Yong 4 Hu Xuejiao 3 Bai Jing 3 Xu Xijian 3 Zhang Yale 3 Lin Jiaxin 3 Li Ping 2 Xiao Shenwei 0 Tong Qunfeng 0 Du Kaitian 0 Zhang Dewan

1 0

review star
by Wu Fengguang 18 Dec '20

18 Dec '20

Xueliang is our review star, thanks Xueliang! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 121 21 24 24 Cao Xueliang 48 30 57 20 Wu Fengguang 74 16 23 15 Wei Jihui 69 7 20 10 Luan Shengde 40 13 11 7 Liu Shaofei 42 14 20 7 Liu Yinsi 29 4 13 5 Yu Chuan 8 2 6 4 Xu Xijian 36 8 7 4 Lu Kaiyi 26 4 5 4 Xiao Shenwei 36 6 4 3 Zhang Yu 22 6 7 3 Lin Jiaxin 13 4 12 2 Sun Yukui 21 5 12 2 Ren Wen 14 4 2 2 Cui Yili 6 2 1 1 Bai Jing 18 3 2 1 Hu Xuejiao 30 4 4 1 Wang Chenglong 38 1 1 1 Wu Zhende 7 2 0 0 Zhang Yale 15 3 4 0 Li Ping 9 0 0 0 Li Yuanchao 6 2 2 0 Zhang Dewan 10 0 0 0 Wang Yong 25 2 4 0 Lu Weitao

1 0

[PATCH compass-ci] lib/es_query: add query filed from index
by Lu Weitao 18 Dec '20

18 Dec '20

[Why] We need only get specific field instead of entire document. it's like SQL: SELECT DISTINCT column FROM table Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..86c078f 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -61,6 +61,30 @@ class ESQuery return result[0]['_source'] end + + # select doc_field from index + # input: + # eg: suite (@index: jobs) + # output: + # [ + # {"key"=>"build-pkg", "doc_count"=>90841}, + # {"key"=>"cci-depends", "doc_count"=>4636}, + # {"key"=>"cci-makepkg", "doc_count"=>3647}, + # ... + # ] + def query_specific_fields(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 } + } + }, + size: 0 + } + result = @client.search index: @index + '*', body: query + + result ['aggregations']["all_#{field}"]['buckets'] + end end # Range Query Example: -- 2.23.0

1 1

2025

2024

2023

2022

2021

2020

Compass-ci