December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 02 Dec '20

02 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..7bf3aa8 --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# usage: +# foo = ErrorMessages.new(build-pkg) +# build-pkg is a file that includes errors generated by make +# error_messages_hash = foo.obtain_error_messages +# error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } +# output: +# src/arg.c:117:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] +# +# src/arg.c:117:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] +# for (nbarg = 0; nbarg < ARGM_NBARGS && ((mask >> (nbarg * ARGT_BITS)) & ARGT_MASK); nbarg++); +# ^ +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'] =~ /error|warning/i + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.146511/job.cgz ' [now values] initrds_uri: - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… - initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz - initrd http://172.168.131.113:3087/job_initrd_tmpfs/crystal.147702/job.cgz Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/scheduler/jobfile_operate.cr | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..7ec3ea1 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -76,8 +76,24 @@ module Jobfile::Operate script_lines = JSON.parse(script_lines) end + def self.handle_job_field(job_content, fields) + fields.each do |field| + case field + when "initrds_uri" + values = job_content[field].to_s.split("\n").map(&.strip).reject!(&.empty?) + values = values.map { |value| JSON::Any.new(value) } + job_content[field] = JSON::Any.new(values) + end + end + + return job_content + end + def self.create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + + need_to_handle = ["initrds_uri"] + tmp_job_content = handle_job_field(job_content.clone, need_to_handle) # put job2sh in an array if job_content.has_key?("job2sh") @@ -100,8 +116,8 @@ module Jobfile::Operate # - this will create a <'#! jobs/pixz.yaml':> in the yaml file # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) + File.open(temp_yaml, "a") do |file| + YAML.dump(tmp_job_content, file) end # generate unbroken job shell content -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.146511/job.cgz ' [now values] initrds_uri: - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… - initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz - initrd http://172.168.131.113:3087/job_initrd_tmpfs/crystal.147702/job.cgz Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/scheduler/jobfile_operate.cr | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..10e93fb 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -76,8 +76,24 @@ module Jobfile::Operate script_lines = JSON.parse(script_lines) end + def self.handle_job_field(job_content, fields) + fields.each do |field| + case field + when "initrds_uri" + values = job_content["initrds_uri"].to_s.split("\n").map(&.strip).reject!(&.empty?) + values = values.map { |value| JSON::Any.new(value) } + job_content["initrds_uri"] = JSON::Any.new(values) + end + end + + return job_content + end + def self.create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + + need_to_handle = ["initrds_uri"] + tmp_job_content = handle_job_field(job_content.clone, need_to_handle) # put job2sh in an array if job_content.has_key?("job2sh") @@ -100,8 +116,8 @@ module Jobfile::Operate # - this will create a <'#! jobs/pixz.yaml':> in the yaml file # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) + File.open(temp_yaml, "a") do |file| + YAML.dump(tmp_job_content, file) end # generate unbroken job shell content -- 2.23.0

3 3

[PATCH v2 compass-ci 3/3] send-internet-mail: update start
by Luan Shengde 02 Dec '20

02 Dec '20

attach MAILDIR to the container store mail files that have been sent - MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..d6e4e25 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,25 +12,28 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) -cmd = %W[ +cmd = %w[ docker run --restart=always --name=send-internet-mail -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

3 4

[PATCH v2 compass-ci] 本地安装compass-ci客户端.md: adjust markdown document format
by Xu Xijian 02 Dec '20

02 Dec '20

1. In markdown documents, text for bash should be contained in block as below: ```bash content ``` 2. Add blank line between title and normal text, or it will not line feed. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- ...205compass-ci\345\256\242\346\210\267\347\253\257.md" | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 43a6bf5..b986626 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -9,20 +9,25 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ## Getting started 1. setup default config + run the following command to add the below setup to default config file +```SHELL mkdir -p ~/.config/compass-ci/defaults/ cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx my_email: xxx my_name: xx EOF - +``` 2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration +```SHELL git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install source ~/.bashrc && source ~/.bash_profile - +``` 3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] send-internet-mail: add function: store email
by Luan Shengde 02 Dec '20

02 Dec '20

add function: store_email store email to MAILDIR/sent file name: ${email_addr}_${time_now} Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) mode change 100644 => 100755 container/send-internet-mail/send-internet-mail.rb diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb old mode 100644 new mode 100755 index 3b1d159..851904c --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -19,3 +19,13 @@ smtp = { } Mail.defaults { delivery_method :smtp, smtp } + +def store_email(mail) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [mail.to[0], time_now].join('_') + file_full_name = File.join(ENV['MAILDIR'], 'sent', file_name) + File.open(file_full_name, 'w') do |f| + f.puts mail + end + FileUtils.chown_R(1144, 1110, file_full_name) +end -- 2.23.0

2 2

[PATCH v3 compass-ci] container/result-webdav: adjust the value of second_dir
by Cui Yili 02 Dec '20

02 Dec '20

[why] 1. when the uri starts with "//", etc., the previous regular expression did not get the second level directory correctly. 2. when the uploaded data package does not meet the format of "xxxx_yyyymmdd.cgz", uploading is also allowed, but no symbolic link. [how] 1. modify the variable second_dir regular expression. 2. make conditional judgments on link_name. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index c880522..1a4dddc 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,17 +54,21 @@ http { path = string.match(uri,"%g*/") upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/.+/(.*)/.+") + second_dir = string.match(uri, "/*%w+/(%w+)/.+") - if (second_dir == "pkg") + if (second_dir == "pkg" or second_dir == "build-pkg") then - link_name = "latest.cgz" + link_name = "latest" else - link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + link_name = string.match(upload_file, "(.*)_%d+%.cgz") + end + + if (link_name ~= '') + then + link_name = link_name..".cgz" + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) end - - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) } } -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/borrow-1h.yaml: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[Why] Now the "pub_key" field is not placed under the "sshd" field in yaml, use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index be22b94a..f2fedd20 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,13 +1,13 @@ suite: borrow testcase: borrow +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> # sleep at the bottom sleep: 1h -- 2.23.0

1 0

[PATCH lkp-tests] jobs/borrow-1h.yaml: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[Why] Now, the "pub_key" field is not placed under the "sshd" field in yaml, use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index be22b94a..38306441 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,13 +1,13 @@ suite: borrow testcase: borrow -sshd: - pub_key: <%= +pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp rescue nil end %> +sshd: # sleep at the bottom sleep: 1h -- 2.23.0

1 1

[PATCH compass-ci] stats_worker.cr: add blank line after next
by Xu Xijian 02 Dec '20

02 Dec '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..4a6ce0f 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -102,6 +102,7 @@ class StatsWorker STDERR.puts e.message next end + next if is_exists new_error_ids << error_id -- 2.23.0

2 2