December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] Redirect stderr only for functions that may use lkp-tests customized make
by Lin Jiaxin 14 Dec '20

14 Dec '20

[Why] we use makepkg 2>&1 so that many errors are not output to the file stderr now. Like these: ==> WARNING: Skipping verification of source file PGP signatures. leptonica-1.79.0.tar.gz ... FAILED ==> ERROR: One or more files did not pass the validity check! And stats/build-pkg can not analyze them so it ignores them. [How] Redirect stderr in functions build(), check(), package() when STDERR_REDIR = y. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++++++--- tests/build-pkg | 4 ++-- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..131ae690 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -1761,11 +1761,19 @@ run_build() { [[ -d /usr/lib/ccache/bin ]] && export PATH="/usr/lib/ccache/bin:$PATH" fi - run_function_safe "build" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "build" 2>&1 + else + run_function_safe "build" + fi } run_check() { - run_function_safe "check" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "check" 2>&1 + else + run_function_safe "check" + fi } run_package() { @@ -1776,7 +1784,11 @@ run_package() { pkgfunc="package_$1" fi - run_function_safe "$pkgfunc" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "$pkgfunc" 2>&1 + else + run_function_safe "$pkgfunc" + fi } build_id() { diff --git a/tests/build-pkg b/tests/build-pkg index c6717d48..aafc120e 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -125,8 +125,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH lkp-tests] Redirect stderr only for functions that may use lkp-tests customized make
by Lin Jiaxin 14 Dec '20

14 Dec '20

[Why] we use makepkg 2>&1 so that many errors are not output to the file stderr now. Like these: ==> WARNING: Skipping verification of source file PGP signatures. leptonica-1.79.0.tar.gz ... FAILED ==> ERROR: One or more files did not pass the validity check! And stats/build-pkg can not analyze them so it ignores them. [How] Redirect stderr in functions build(), check(), packages() when STDERR_REDIR = y. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++++++--- tests/build-pkg | 4 ++-- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..131ae690 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -1761,11 +1761,19 @@ run_build() { [[ -d /usr/lib/ccache/bin ]] && export PATH="/usr/lib/ccache/bin:$PATH" fi - run_function_safe "build" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "build" 2>&1 + else + run_function_safe "build" + fi } run_check() { - run_function_safe "check" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "check" 2>&1 + else + run_function_safe "check" + fi } run_package() { @@ -1776,7 +1784,11 @@ run_package() { pkgfunc="package_$1" fi - run_function_safe "$pkgfunc" + if [[ $STDERR_REDIR == y ]];then + run_function_safe "$pkgfunc" 2>&1 + else + run_function_safe "$pkgfunc" + fi } build_id() { diff --git a/tests/build-pkg b/tests/build-pkg index c6717d48..aafc120e 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -125,8 +125,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH lkp-tests] spec: change the file permission
by Zhang Yu 14 Dec '20

14 Dec '20

The file permission should be consistent with the other file in this directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- spec/local_run_spec.rb | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 spec/local_run_spec.rb diff --git a/spec/local_run_spec.rb b/spec/local_run_spec.rb old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] ssign-account: add config lab yaml file
by Luan Shengde 14 Dec '20

14 Dec '20

[why] key my_uuid is renamed to my_token item key my_token will be stored in file: ~/.config/compass-ci/include/lab/${lab}.yaml to do this change with no sense: for old user(update account): keep my_uuid in the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml my_token has the same value to my_uuid for new user: add my_email, my_name to the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 47 ++++++++++++++++---- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 95bec14..8015a5f 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -110,7 +110,7 @@ class AccountStorage jumper_host, jumper_port = read_jumper_info config_authorized_key(login_name) - config_default_yaml(login_name) + config_yaml(login_name) permit_login_config(login_name) if @data.key?('gen_sshkey') && @data['gen_sshkey'] my_jumper_pubkey = generate_ssh_key(login_name) @@ -127,6 +127,14 @@ class AccountStorage return jumper_account_info end + def config_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + config_default_yaml(login_name, default_yaml) + + lab_yaml = touch_lab_yaml(login_name) + config_lab_yaml(login_name, lab_yaml) + end + def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) @@ -151,26 +159,47 @@ class AccountStorage def touch_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + # before create the file, the directory need to be exists FileUtils.mkdir_p default_yaml_dir unless File.exist? default_yaml_dir default_yaml = File.join(default_yaml_dir, 'account.yaml') + # for we need to read the file before we open it + # need to firstly create the file FileUtils.touch default_yaml unless File.exist? default_yaml default_yaml end - def config_default_yaml(login_name) - default_yaml = touch_default_yaml(login_name) - + def config_default_yaml(login_name, default_yaml) account_yaml = YAML.load_file(default_yaml) || {} - # my_email, my_name, my_uuid is required to config default yaml file + # my_email, my_name is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] - account_yaml['my_uuid'] = @data['my_uuid'] - f = File.new(default_yaml, 'w') - f.puts account_yaml.to_yaml - f.close + File.open(default_yaml, 'w') do |f| + f.puts account_yaml.to_yaml + end + + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def touch_lab_yaml(login_name) + lab_yaml_dir = File.join('/home', login_name, '.config/compass-ci/include/lab') + FileUtils.mkdir_p lab_yaml_dir unless File.exist? lab_yaml_dir + + lab_yaml = File.join(lab_yaml_dir, "#{@data['lab']}.yaml") + FileUtils.touch lab_yaml unless File.exist? lab_yaml + lab_yaml + end + + def config_lab_yaml(login_name, lab_yaml) + lab_yaml_info = YAML.load_file(lab_yaml) || {} + lab_yaml_info['my_token'] = @data['my_token'] + + File.open(lab_yaml, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] assign-account: rename my_uuid to my_token
by Luan Shengde 14 Dec '20

14 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c0753b0..4ed95c0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -24,7 +24,7 @@ get '/assign_account' do end def assign_jumper_account(data) - lacked_info = %w[my_email my_name my_uuid] - data.keys + lacked_info = %w[my_email my_name my_token] - data.keys error_message = "lack of my infos: #{lacked_info.join(', ')}." raise error_message unless lacked_info.empty? -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] answerback-email: rename key my_uuid with my_token
by Luan Shengde 14 Dec '20

14 Dec '20

key my_uuid is renamed to my_token when applying/updating account, will use my_token instead to do this change with no sense for old user(update account) keep key my_uuid and add key my_token for new user just add key my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 25 ++++++++++++++------ 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c08f10e..766e1b3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -21,6 +21,7 @@ names = Set.new %w[ JUMPER_PORT SEND_MAIL_HOST SEND_MAIL_PORT + lab ] defaults = relevant_defaults(names) @@ -29,12 +30,14 @@ JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = defaults['lab'] +TOOL_SERVER = 'z9' my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => nil, + 'my_token' => nil, 'my_login_name' => nil, 'my_ssh_pubkey' => [] } @@ -51,6 +54,8 @@ my_info = { # stdin_info > email_info > my_info_es # when assigning account or update conf for account # if they have the same key, my_info will use the value with higher priority. +# if execute update operations, if there are new key in stdin_info and email_info +# it will use the one with higher priority, and merge it with the the keys in my_info_es # conf_info is used to store keys used to config the account stdin_info = {} email_info = {} @@ -179,6 +184,7 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info + apply_info['lab'] = LAB account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str @@ -200,9 +206,7 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' new_pubkey = new_stdin_pubkey || new_email_pubkey - my_info.update my_info_es unless my_info_es.empty? - my_info.update email_info unless email_info.empty? - my_info.update stdin_info unless stdin_info.empty? + update_my_info(my_info, my_info_es, email_info, stdin_info) return if new_pubkey.nil? return if my_info['my_ssh_pubkey'].include? new_pubkey @@ -210,6 +214,13 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) my_info['my_ssh_pubkey'].insert(0, new_pubkey) end +def update_my_info(my_info, my_info_es, email_info, stdin_info) + my_info['my_token'] = my_info_es['my_uuid'] if my_info['my_token'].nil? + my_info.update my_info_es + my_info.update email_info + my_info.update stdin_info +end + def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] @@ -221,9 +232,9 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == TOOL_SERVER - message = 'please run the tool on z9 server' + message = "please run the tool on #{TOOL_SERVER} server" puts message return false @@ -260,7 +271,7 @@ def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_my_email(my_info) return unless check_email_assigned_account(conf_info, my_info_es) - my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + my_info['my_token'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) return unless check_my_name_exist(my_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] mail-robot: update email message
by Luan Shengde 14 Dec '20

14 Dec '20

update email message with new key: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index a88626c..834e96f 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -27,7 +27,7 @@ def build_apply_account_email(my_info) EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF - my_uuid: #{my_info['my_uuid']} + my_token: #{my_info['my_token']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/defconfig.rb: add administrator account configuration files
by Liu Yinsi 14 Dec '20

14 Dec '20

sbin/build-my-info.rb need to use configuration files /etc/compass-ci/account/*.yaml, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 482dbc8..af7e248 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/account/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/setup-image: set the system time zone
by Wang Chenglong 14 Dec '20

14 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 8e2e88a..787a42c 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -13,6 +13,11 @@ remove_file() rm -rf /.dockerenv } +set_localtime() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + setup_login() { [ -n "$ROOT_PASSWD" ] || return @@ -32,5 +37,6 @@ pack_cgz() install_packages remove_file +set_localtime setup_login pack_cgz -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: bisect email use new content
by Cao Xueliang 14 Dec '20

14 Dec '20

1. Add software name and first error message in subject 2. Add gcc version, error messages and job url in mail body Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 20 +++++++++----------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 189edb8..97ad467 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages] end # first search the good commit in db diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5a208c4..44a5645 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,23 +19,21 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI] #{(a)repo}.git: bisect result" + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" + job_url = "job url: http://183.134.196.212:11300/#{ENV['result_root']}\n" ? ENV['result_root'] : '' body = <<~BODY Hi #{(a)git_commit.author_name}, - Bisect completed for + git url: #{(a)git_commit.url} + git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") - url: #{(a)git_commit.url} - - This is a bisect email from compass-ci. We met some problems when test with new commits. - Would you help to check what happend? - After submitting a job we noticed an error response due to the commit: - - commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") + gcc version: 7.3.0 error_messages: - #{@error_messages} + #{(a)error_messages.join("\n")} - https://gitee.com/openeuler/compass-ci + #{job_url} + Regards, + Compass CI team BODY to = 'caoxl(a)crystal.ci' @hash = { 'to' => to, 'body' => body, 'subject' => subject } -- 2.23.0

2 2